Until recently, water utilities regarded resiliency as a matter of defense against severe weather events. Now cybersecurity threats—including cyber terrorism—have taken a place on the list of water utilities’ top concerns.
It’s not enough to be able to keep people from getting into a utility’s system, points out Susan Story, American Water CEO.
“Someone will find a way to get in,” she asserts. “How will you handle it when it happens? We can defeat them 100 million times, but if they get in once . . . we can’t let that one get in. And we’ve got to make sure that when they get in, we know how to respond.”
“Every public water supply and wastewater utility should assess the likelihood and consequences of a supply disruption, identify critical vulnerabilities, and consider alternative power or supply redundancy to mitigate service disruptions lasting up to 72 hours or longer if public health, environmental, or economic impacts are severe,” the American Water Works Association (AWWA) states in a 2014 resolution.
“Careful thought must be given to how much water service—such as minimum daily demand—can be assured, given local circumstances. In addition, every utility should have a robust emergency response plan that includes a public communications plan tailored to its needs and circumstances, for use in case of an electric supply disruption.”
American Water adopted the National Institute of Standards and Technology (NIST) Cyber Security Framework, created as a result of a 2013 executive order from President Obama to improve critical infrastructure.
Created through collaboration between industry and government, the framework consists of standards, guidelines, and practices to promote the protection of critical infrastructure, says Story. “The prioritized, flexible, repeatable, and cost-effective approach of the framework helps owners and operators of critical infrastructure manage cybersecurity-related risk,” she says. “We hold ourselves to the same standard of the electric utilities and the grid. The framework is voluntary for the water sector at this point.”
In the meantime, the AWWA created the Process Control System Security Guidance document to support water utility adoption of the NIST framework, says Story.
The document details 12 steps the water utility industry should take to shore up cybersecurity that addresses governance and risk management; business continuity and disaster recovery; server and workstation hardening; access control; application security; encryption; telecommunications, network security, and architecture; physical security of process control system equipment; service level agreements; operations security; education; and personnel security.
American Water participated as a subject matter expert on the development of that document. That document and NIST standards should be part of every water utility’s blueprint, notes Story.
In a recent presentation to the National Governors Association (NGA)—which has identified cybersecurity as one of its top concerns for every state—Story points out that cybersecurity affects the broader infrastructure sector.
Water utility managers do not have a cybersecurity background, but have had to “quickly learn about this complex issue and spearhead new policies for their companies,” she says.
It’s not unlike ensuring “after hurricanes, ice storms, flooding, and during droughts that people have the critical services they need,” she points out. “These are very serious things people take for granted every day.”
American Water has a significant footprint of customer service in 47 states. The company has regulated operations in 16 states serving 12 million people. American Water also provides water and wastewater services to 12 military installations nationwide. The company also runs 41 different municipal systems for municipalities across the country.
What happens in each state affects the utilities and “we want to be part of the solution that [governors] are working on,” says Story.
Story—who spent 31 years in the electrical industry before moving to the water industry more than three years ago—points out that the physical aspects of utilities and cybersecurity cannot be separated. “Everybody hears about the Internet of Things. It has different definitions. If you are in electric, water, gas, or telecomm, cybersecurity is not just about our systems that have customer information and employee information,” she says. “It’s about the systems that run the grids and ensure water systems are operating. You cannot separate the physical and cybersecurity in the world of infrastructure, especially with utilities.”
A blueprint for cybersecurity is rooted in the Black Sky Hazard, a concept introduced in a National Association of Regulatory Utility Commissioners (NARUC) paper, “Resilience for Black Sky Days: Supplementing Reliability Metrics for Extraordinary and Hazardous Events,” written by Paul N. Stockton.
Stockton is the managing director of Sonecon and an international leader in infrastructure resilience, continuity planning, and installation and personnel security, as well as US national security and foreign policy.
The Electric Infrastructure Security Council began using the Black Sky concept as a framework when working with electric utilities, the US Department of Homeland Security (DHS), the US Department of Energy (DOE), and the US Department of Defense—along with the United Kingdom, Israel, and others—on a series of Black Sky playbooks to support resilience, restoration, and recovery planning.
The group soon realized the most pressing health challenges of such a massive outage would actually be water-related in the potential lack of drinking water, basic sanitation, and fire protection, notes Story.
Whether they come via cyber attacks or through severe weather events, threats are intrinsically tied to the water-energy nexus: what happens to one sector affects both. “What if an electromagnetic pulse shuts down the electric grid and you have a population center of greater than one million people who are out of electricity for at least 25 days? How do we withstand that? How would you evacuate a major urban center with no drinkable water and no sanitation services? There will be concerns about disease,” she points out.
The recently-released Electric Infrastructure Protection (EPRO) Handbook II (Water) was written to address resilience of water and wastewater service following a Black Sky event, says Story, noting that the handbook was also a collaborative effort of American Water and AWWA.
In its executive summary, the handbook makes the point that the US and partner nations remain at risk of blackouts far more severe than those occurring from Superstorm Sandy, Hurricane Katrina, or other previous events. More progress in strengthening electric grid resilience is necessary to build preparedness for Black Sky outages covering multiple US states or regions lasting a month or longer.
Risks come not only from cyber-attacks, but geomagnetic disturbances from severe solar storms and an electromagnetic pulse strike, damaging unprotected, high-voltage transformers and other high-voltage grid components throughout multi-state regions. Cataclysmic earthquakes in seismic zones pose unique threats.
Such power outages disrupt water and wastewater systems that depend on the flow of electricity, affecting water pumps, lifts, treatment systems, and other critical system components, making it impossible in a wide area, long duration blackout to provide emergency drinking water and water for firefighting to millions of affected citizens. Failure of wastewater systems would contaminate surface water, a primary factor in spreading disease.
A growing number of utilities are installing their own emergency power generators or arranging with partners such as the US Army Corps of Engineers (USACE) to do so. Many also are expanding capacity to store generator fuel onsite, and plan for essential treatment chemical delivery.
Utilities also are affiliating with the AWWA’s Water and Wastewater Agency Response Network (WARN) in which utilities help other utilities respond to and recover from emergencies.
Part of a utility’s playbook is establishing minimalist, sustainable service levels in coordination with regulators, emergency managers, and other partners for meeting customer needs in wide area blackouts lasting a month or more, and accounting for limited fuel, treatment chemical resupply, and other logistical problems.
That entails infrastructure investments as necessary, which may require additional funding by utility boards of directors or government officials. Power generators and fuel storage may require regulatory policy development and associated pre- or post-outage waivers of Clean Air Act standards and other regulatory policy changes, the EPRO handbook points out.
Private contractors and government agencies will need to be able to supply replacement generators, as well as fuel and treatment chemicals despite the severe disruption of transportation and communications systems Black Sky hazards will create. Backup power requires resilient fuel sources and reliance on those sources depends on a utility’s location and infrastructure.
Key components of natural gas systems, such as pipelines, compressors, and industrial control systems rely on electric power. Natural gas is not typically stored onsite and must be delivered as it is consumed.
Hydroelectric power plants typically have access to their gravity-fed generation facilities. Coal-fired generators also have such access to fuel with sufficient coal stored onsite to enable generators to produce electricity for the long run. Storage of natural gas in salt caverns or other facilities co-located with power generators may offer viable storage options in some regions if those storage facilities have power to operate.
Dual-fuel generators may offer an effective means to hedge against disruptions in natural gas supplies. These generators can run on natural gas as primary fuel or, if gas supplies are interrupted, can use clean diesel #2 fuel oil or other secondary fuel sources to sustain operations until gas flow is restored, the EPRO handbook points out.
In many US regions, gas transmission compressors—historically fueled with gas taken from the pipelines—are being replaced by variable speed electric-powered units to reduce onsite methane emissions and increase compressor efficiency.
Black Sky outages could interrupt the flow of electricity to these units. Even if compressor stations rely exclusively on gas-powered compressors, these stations typically require electricity for other functions, including control, gas cooling, and telecommunications systems. Most compressor stations have backup power generators.
Colocation of new gas storage and power generation facilities is one example of a coordinated approach. Power generators sited adjacent to storage facilities provide a ready supply of backup fuel until pipeline flows are restored.
Black Sky events are likely to require restoration from assets within the outage region through black start operations that re-energize the grid using specially designated generation resources within a blacked-out area rather than by importing power from outside that zone.
Black start operations depend on black start units and cranking path generators. Black start units are often hydro- or diesel-fueled power generators that can start without support from an outside electric supply.
Once started, these units then provide power to start larger generators along a specially designated cranking path to gradually re-start other generating units. A designated portion of black start resources requires adequate onsite fuel for weeks of operation.
The Federal Emergency Management Agency (FEMA) is partnering with the DOE to develop a new Power Outage Incident Annex to the Response and Recovery Federal Interagency Operations Plans to address the response and recovery to a mass or long-term power outage regardless of the cause.
FEMA, USACE, and other federal agencies also are moving forward in strengthening US emergency generator inventories and fuel delivery capabilities.
Backup communications capabilities are especially important but also at great risk in Black Sky events; hazards could cause highly disruptive damage on communications system components.
The National Security Telecommunications Advisory Council has put forth recommendations to strengthen the resilience of communications systems against emerging threats focusing on an a robust communication system that shores up minimally adequate voice and data communication for critical operations that operates without external power for at least 30 days.
Another playbook also is under development to assist public service commissions in similar planning and coordination efforts, notes Story.
Story says American Water managers learned lessons after natural disaster recovery.
“After Superstorm Sandy in the Northeast, with American Water, we had fuel, but we had no place to dispense the fuel,” she says. “Some of the local areas in New Jersey, New York, and Pennsylvania had space, but needed fuel.
“We worked out a deal to provide them fuel if they gave us space to dispense the fuel. It’s built into our emergency plans. The convergence of physical and cyber don’t necessarily have to have separate plans.”
UV disinfection technology at Sweeney Water Treatment Plant
Story points out that as the water utility industry develops intelligent water systems—which is a corollary to the electric smart grid—”we’ve got to make sure we build cyber controls into every part of that technology, not something bolted on later. The best technology is critical, but it’s got to include physical and cybersecurity controls designed and integrated from the very beginning of the project.”
Story maintains that for every water utility—from the smallest to a large one such as American Water that operates across many states—it’s critical to partner with state and local governments in the effort to protect against cybersecurity so that “what one person knows, the other knows.”
American Water’s state operations partner with environmental organizations, fusion centers, public service commissions, and state emergency planners, she says.
“From a federal standpoint, we are able to help our state operations, because we partner with the Federal Bureau of Investigation, the Department of Homeland Security, and participate on the Water Sector Coordinating Council. We partner with the Centers for Disease Control and the United States Environmental Protection Agency because of water quality and potential water contamination.”
Story says she is proud that at American Water, cyber and physical security is not solely the job of the company’s IT department or operations department.
“Every level of employee, including myself and our board of directors, gets involved with cybersecurity at American Water,” she says.
American Water has placed its research and development, environmental, and IT organizations under one Chief Technology and Innovation Officer, whose role is the integration of all of the technology into operations and systems to—among other goals—ensure the company builds in cyber and physical security into its systems from the beginning.
Operational technology encompasses not only systems for email, but the people running the water treatment at the plants, Story points out.
“It’s connecting all of that together and saying technology is not separate from our business anymore,” she adds. “The cultural shift is in getting people to understand cybersecurity is every single employee’s job.”
Story suggests that utilities engage in an annual exercise to ensure that. In 2015, the company’s state operations participated in an exercise with its information technology and security teams. Following that, the executive leadership team, led by Story, participated in an all-day session in which only she and the head of cybersecurity were cognizant of the scenario. She spent the day observing how American Water’s senior executives would react to a cyber intrusion into the company’s system.
“We then took the exercise to our board of directors and told them we’re going to spend two hours walking them through it so they can see how we handled this,” adds Story.
American Water conducted another exercise—a simulation game style exercise developed by PwC called the “Game of Threats.”
“It’s the most incredible thing I’ve been through because it’s real life,” says Story. “You divide up into two groups. One group is a hacker, the other is the company. Based on what each side decides to do, you have to react in real time how you would deal with an intrusion into your system.”
Through the exercise, Story says she discovered “it’s a lot easier to be a hacker than it is people trying to protect the systems, whether it’s companies, states, or local agencies. It was incredible how difficult it was. But it was a really great exercise to go through. At the end of the day, we have to look at physical and cybersecurity and the integration of the two of those for all of the critical infrastructure you have in all of the states and our companies.”
Getting water utility employees on board is not difficult, Story contends.
“The great thing about being in a water company is that each of our employees understands the criticality of what we do,” she says. “Cybersecurity is not an IT function. The biggest cultural issue in corporations, states, and other organizations by human nature is that the computer guys will take care of it, right? Not anymore. Change that perspective.”
American Water management discovered that by going through exercises such as sending out fake phishing emails to its employees changes the company culture. “Our 6,700 employees across the country are thinking about it now,” she adds.
Story notes that as a utility starts to do exercises, it becomes apparent how employees will react to potential dangers. “We say we’re going to do an exercise and you, Mr. Executive Vice President, are not going to know what it is. We’re going to watch to see how you respond and follow up with best practices to see where we can do better,” she says.
That is critically important for a company’s culture, she says. “This is for everyone to learn. Set the expectations up front. As we do these exercises, it’s to have a safe place where we can say we may not do everything perfectly in the exercise, but if we don’t, let’s support each other and look for best practices and not try to do this façade of going through the motions as opposed to really trying to find out where we have weaknesses. That’s from the bottom of an organization to the highest levels.”
Lambasting an employee for not knowing how to handle the situation should be avoided because while “that sounds like a small thing, I worry that could be the thing that could trip up effective exercises,” says Story.
Still, it helps to have someone on staff who can troubleshoot problems as they arise or help prevent them before they do. While it may be difficult for water utilities to find skilled workers and technical professionals, that’s not the case for finding IT specialists, Story says.
“Millennials want to do something that matters to the world,” she adds. “We’re getting recruits because they want to be part of the water story. They love that they’re going to be doing something that truly makes a difference to the lives of people, communities, and the environment. If you can combine stable pay, training, and retirement benefits along with making a difference in the world, we’re finding recruiting can be quite successful.”
Story issues a four-point call to action to states and governors: “One, please promote communications and teamwork. Make sure your agencies are working with all of the utilities. Make sure the public utilities commissions, electric, gas, and water providers are doing this together and bring in federal partners such as the Department of Homeland Security. It’s important that people are talking and sharing information.”
Ensuring resiliency in assets and infrastructure is her second call to action. “This is not easy,” she concedes. “From the water industry standpoint, we have, in many of our states, the ability to get capital investment in pipes recovered more timely through system infrastructure charges, which is really important. But to get approval to do something for resiliency that we hope never gets used? It’s more difficult.”
The third call: consider private-public partnerships. “Nobody can do this on their own,” points out Story. “We all have to share our best practices, because we’re all trying to do the same thing and we’re after the same objective.”
The fourth call to action is for governors to head up simulation exercises in their states. Recently, Pennsylvania held a day-long Black Sky exercise facilitated by the Electric Infrastructure Security Council, drawing together 130 people representing federal agencies, the military, Homeland Security, state agencies, and all of the utilities.
“If we all work on these together, we can make sure your citizens and our customers can feel more peaceful that we’re ready to handle whatever comes along,” Story told governors during their meeting.
Michael Richardson, who served as a subject matter expert for the AWWA Process Control System Security Guidance document, is the water resources manager for Cape Fear Public Utility Authority in North Carolina, which services 68,000 water and sewer connections. “Our concerns are two-fold in the fact that all of our plants have quite elaborate SCADA systems that allow the operators to operate the plants,” he says. “The SCADA systems are always one of the first thoughts—somebody being able to hack in and to turn pumps and other equipment on and off or take it out of control of the operators. We’re always looking at how to keep that safe.”
A second concern is emails and the business side of the utility’s operation. “In dealing with customer service as we continue on with enhancements to our customer service program, allowing customers more access to their billing data and payment by credit cards, it’s making sure those systems are being as well protected as they can be,” notes Richardson.
“Sometimes it involves third parties that take information and use it as far as the billing system and how those interactions between those third party vendors and the authority are also a concern.” There are the third-party vendors who troubleshoot problems and do updates through the system and “we’re trying to determine if we have proper protection to allow these people in,” adds Richardson.
Cape Fear Public Utility Authority has nearly 300 employees, each one dependent upon the email system, Richardson says. That email may bring with it attempts at phishing and spear-phishing. “There are so many hidden ways,” he notes. “We started looking at this through our IT department and we have a dedicated security emergency management person. Through them, we are initiating some upgrades as well as vendor access. We’re always looking at those types of things as to how to best protect our data as well as our systems.”
In the event that the system is compromised through a human-created or weather-related event, the Cape Fear Public Utility Authority has ample backup power, notes Richardson. “As being part of a team with the state of North Carolina, we were dealing with solar flares and different things,” notes Richardson. “Our system being on the coast and subject to severe weather—primarily hurricanes—means we have a lot of generators in place already for most of our major systems. We run those regularly.
“We’re able to have sufficient storage of materials and fuel as well as established contracts with outside vendors to supply fuel,” he adds. “We’re always looking on an annual basis at our readiness plan for our emergency systems.”
SCADA systems are a high cybersecurity priority for American Water.
Generators are tested on a regular basis throughout the systems. The utility has portable generators for some of its smaller facilities that don’t even require them. The generators are diesel fueled. The utility is considering alternate fuels such as natural gas when its system is upgraded.
“We look at it as something that would be cost-beneficial to go to as opposed to storing fuel onsite, having a supply,” notes Richardson. “We’re also looking into how well the natural gas industry can supply if they have problems.”
Richardson points out that even the smallest utilities can take measures to ensure operational continuity.
In a disaster, “getting the water and sewer back in operation is a key to the community getting back on its feet as soon as possible,” he says. Richardson has served as chairperson of NCWaterWARN, a North Carolina-based network of water utilities helping each other respond to and recover from emergencies.
It is especially essential for smaller systems to be a member, he adds. “Sometimes they say they’re small and don’t have anything to offer. It’s not about offering. It’s about having that connection whenever you get into an emergency situation, whether it’s a big disaster or a local disaster,” he says.
Cheryl Santor is unit manager of the Information Security Services Unit for the Metropolitan Water District of Southern California.
In addition to serving as a subject matter expert for the AWWA document on Process Control System Security Guidance, Santor also attended two of the DHS NIST workshops at universities, joining operations and information technology and security practitioners to create the framework. Discussions covered every aspect of running and managing water and wastewater facilities.
“Oftentimes, attendees were looking for prescriptive means of managing cyber security, wanting a check off list of what to do and how to perform the tasks,” she notes.
Going through the NIST assessment and using the AWWA guidance has provided a self-examination of where the Metropolitan Water District of Southern California stands and “what can further our cyber security posture,” notes Santor.
“The very first assessment task is to do an inventory assessment—to find out every piece of equipment/system/application used in the management and running of water and wastewater,” she says. “It goes back to the old adage, ‘You can’t manage what you don’t know.'”
Many times, companies are unsure of what they have and how it interfaces and connects with other system functions, Santor points out. “This has been a critical issue brought to light in the past few years with interfaces with HVAC systems connecting to business network systems and lack of security measures on databases, among other factors,” she adds. “With critical infrastructure, we need to be more aware of every link in the chain to assure it is not the weakest link.
“The nation depends on the activities in critical infrastructure to count on day-to-day services. It is in our best interest to conduct an assessment not just one time, but minimally each year.”
The Metropolitan Water District’s efforts over the years have been to provide least privilege access to systems and applications, conduct patch management in a timely and current manner, separate its business and SCADA networks with hardened firewall technologies, provide whitelisting where feasible so that only appropriate functions occur, continuously run vulnerability assessment tools to reduce and manage risk, and work with tools and techniques to strengthen the cybersecurity posture, Santor says.
Santor notes that it is critical for every operations and security personnel to seek out professional organizations for more oversight on what can be done for their companies. MSD’s participation in such organizations has given the utility a sense of best practices and standards that can be implemented in its own operation for managing risk.
Richardson points out that a utility’s successful execution of an emergency plan comes down to the support of upper management “to A) allow their people to do what needs to be done and B) to try to find the funds as necessary that enable them to do that.”
It also is important to share information with customers in order to get them on board for the possible rate hikes it may entail to build resiliency. “If they want to ensure the fact that they can brush and flush every day, then they have to understand that rates have to support those things,” he says. “That’s the only way utilities are only going to have the means to do what they need to do.”
