The U.S. administration identified new regulations for critical infrastructures as a priority in its National Cybersecurity Strategy, including regulations for water utilities. Acting on this priority, U.S. EPA ordered states to include cybersecurity considerations in water utility audits and issued guidelines for such audits, citing dramatic worsening of the cyber threat environment.
Cyber threats
The latest threat report from ICSStrive and Waterfall Security Solutions shows that cyber attacks on operational technology (OT) networks that resulted in physical consequences, such as production shutdowns, damaged equipment or worse, were largely a theoretical problem from 2010 to 2019. In 2020, that changed.
Today, the number of deliberate cyber attacks with physical consequences is more than doubling annually. At this rate, professionals should expect at least 4,500 attacks reported in 2027, impacting more than 15,000 sites.
Water systems are targets
The ICSStrive/Waterfall threat report shows that, within the last year, 2% of attacks causing production outages or physical damage impaired water treatment or distribution systems. Recent examples of outages and near misses in the water sector include global examples, but two U.S. attacks are outlined here.
A stale Microsoft Teams account was used to take over the water treatment system’s human-machine interface (HMI) in Oldsmar, Florida. The attacker told the treatment system to increase Lye injections 100-fold. The sabotage attempt was caught by the plant operator and the correct settings were restored without measurable changes in the quality of water in finished water reservoirs.
The Narragansett Bay Commission (NBC), which is responsible for sewer systems in metro Providence and Blackstone Valley, paid $250,000 to a ransomware group after suffering a cyber attack. The group encrypted and crippled systems, demanding ransom to re-enable them.
Small to medium-sized water systems are prime targets for these types of attacks. These utilities are challenged to defend themselves because they have limited budgets and often have no engineers, no cybersecurity personnel and certainly no industrial cybersecurity practitioners. Cybersecurity programs also are expensive and imperfect. What these utilities need is affordable engineering-grade protection from cyber risks.
Engineering-grade protections
Engineering-grade mitigations operate predictably and reliably in the face of even sophisticated cyber attacks. Such protections are essentially “un-hackable.”
The new National Cyber-Informed Engineering Strategy (CIE) gives the example of physical limits. If the pipe connecting a lye reservoir to a finished water supply is only big enough to deliver up to twice the normal amount of the concentrated additive, then a cyber attack that attempts to increase lye concentrations 100-fold will certainly fail. No matter what rate of lye delivery an attacker tries to set, the pipe can only deliver at the maximum rate that its size allows.
A second example of engineering-grade protection is manual operations. If a cyber attack cripples OT computers, but staff can still operate pumps and filters and other infrastructure manually, then the utility can turn off the compromised computers and continue to treat and distribute water. This is less efficient than doing the job automatically — efficiency is the reason to invest in automation in the first place. However, operating infrastructure manually buys the time that incident response teams might need to repair automation systems without risk to the public or to the environment.
Utilities can design drinking water treatment systems so that untreated water cannot bypass treatment. Under this design, no physical path or piping allows untreated water to move through the system without passing through filters and other purification mechanisms.
A facility can use analog electronics on electric pumps to prevent the pumps from starting unless they are already at an almost complete stop. This prevents attacks that issue stop and restart commands only a fraction of a second apart. Such commands risk connecting pumps to the power grid out of phase with the grid, causing torque on the pumps that is powerful enough to damage them.
Staff can carry out regular manual sampling and analysis of treated wastewater to detect incompletely treated water that might be exiting the treatment system, even if a cyber attack has impaired both the treatment system and the automatic sampling systems.
Network engineering from first principles
A new field of network engineering is emerging as part of this CIE analysis to address the need to keep critical infrastructures running as correctly, continuously and efficiently as is practical. For example, while a pipe can be designed with a physical limit to prevent unsafe levels of lye in drinking water, most water utilities still do not want to see sub-optimal amounts of lye in finished water.
While a manual operation fall-back keeps drinking water safe in the taps during an attack, utilities prefer automation continue unaffected in the face of a ransomware attack to maintain efficiency. While regular sampling prevents long-duration discharges of incompletely-treated wastewater into a watershed, utilities would rather prevent even short-term discharges of such wastewater, in addition to long-term discharges. To achieve these ends, network engineering keeps cyber attacks from entering OT automation systems in the first place.
Three laws of OT cybersecurity
One important principle is the third law of OT cybersecurity: all cyber-sabotage attacks are information, and all information can encode a cyber attack. The only way a control system can change from a normal to a compromised state is if attack information enters the control system. To protect a control system from compromise, control or eliminate all opportunities for incoming information or attacks.
The engineering-grade way to eliminate information-based attacks is to physically control the flow of information. The relevant first principle here is the second law of OT security: all software can be hacked. All non-trivial software has defects or security vulnerabilities, both discovered and undiscovered, hence the need for physical control over incoming information flows.
The simplest example to stop attacks propagating from the Internet and through IT networks into OT networks is the much-maligned air gap. An air gap is a network design where there is physically no way that online information can enter the OT network or its automation system. Offline information — on USB keys or in laptop computers — can be carried into air-gapped automation systems, but no information can flow directly or indirectly from the Internet.
The upside of air-gapped design is engineering-grade prevention of cyber attacks from external networks. The downside is that a lot of automation relies on access to OT data, which means utilities with air-gapped systems cannot benefit from that automation’s efficiencies.
A practical example of engineering-grade protection for OT networks is the unidirectional gateway. The NIST 800-82 standard defines a unidirectional gateway as a device that is a combination of hardware and software. The hardware is only able to send information in one direction. The software makes copies of servers and emulates devices.
If a unidirectional gateway oriented to send information out of the OT network is the sole connection between OT and IT networks or the Internet, then that gateway constitutes engineering-grade protection from online attacks coming from external networks. In turn, the gateway’s replica servers continue to provide access to OT data, enabling modern business automation and its efficiencies.
False economies
The most common complaint against unidirectional gateways or air gaps in small utilities is that remote access through the Internet and into OT networks can save utilities easily time and money. Stakeholders create spreadsheets detailing to the penny the amount of money they can save by enabling remote access to automation systems. Typical savings for even small utilities can be tens of thousands of dollars per year.
However, these “savings” are most often false economies. The spreadsheets ignore the hundreds of thousands of dollars spent on software-based cybersecurity systems to reduce the risk and costs of modern cyber attacks.
Bottom line
In short, on the downside, small water utilities are prime targets for hacktivists and ransomware criminals. Given the rapidly worsening threat environment, utilities need to expect that, within the next couple of years, the EPA and other authorities are going to create additional new regulations, requirements and standards for the cybersecurity of water systems. Small and medium-sized water systems are going to struggle to access the budgets and expertise needed to keep up with these new threats and regulations.
On the upside, water utilities will benefit from practices being incorporated into the new CIE initiative, and more specifically will benefit from following the emerging field and techniques of network engineering.
These engineering techniques represent modest changes to water system, network and automation designs, and the protections they provide are longer-lasting than hackable software-based protections. WW
