Outsourcing Vulnerability Assessments – Tips for the Smaller Utility District
By Terry Hall
The federal government gave small water systems short shrift when it distributed grants to water utilities to help them prepare their EPA-mandated vulnerability assessments (VAs). Despite the best efforts of organizations like the AWWA and the Rural Water Association, systems serving between 3,300 and 50,000 customers are left to fend for themselves financially as they begin the VA process. And with the June 30 deadline looming, many small-system managers are looking for ways to complete their VAs as quickly and as cost effectively as possible.
Misperceptions about cost, a lack of existing benchmarks and the newness of the requirements created confusion in the marketplace and has led many system operators to believe that the only way to complete their VAs on time and on a typically modest budget is to conduct them in house, without assistance from a security consultant or a design engineering firm. But a closer examination of the VA process, as well the experience that came out of the first round of VAs, will show that working with an outside consultant can ultimately be more cost effective and a smarter long-term investment.
When larger systems conducted their VAs, the exercise was fraught with inefficiencies because the specific process mandated by the EPA was new both for systems and for many of their vendors. However, the experiences of these large- and mid-sized systems can be instructive to smaller systems, as they illustrate some of the pitfalls that small systems should avoid when it comes to cost, technology, and choosing a consultant partner.
The first group of systems that had to complete and submit their VAs were those serving more than 100,000 customers. Those large systems were also eligible for the maximum allowable grant distributed by the EPA: $115,000. And when the large systems issued RFPs for outside consultants to perform their vulnerability assessments, an interesting thing happened: Most of the bids were right around $115,000. When the second group of systems, those serving between 50,000 and 100,000 customers, went out for bid, prices came in all over the place.
The bids that came in from the first two rounds of VAs created the first misconception that small districts need to overcome: That hiring an outside consultant will cost a minimum of tens of thousands of dollars. The fact is that an effective VA for a smaller system can be done for much less when the firms involved capitalize on the right efficiencies. Small system operators should not be dissuaded from seeking outside assistance simply based on what the larger systems have been paying.
A second misconception coming out of the first rounds of VAs was that the EPA required systems to use Sandia methodology to conduct their assessments. While this was never explicitly stated, systems assumed by default they needed to use Sandia to receive the maximum allowable grant money from the federal government. While highly respected and effective, the Sandia methodology can also be time consuming and cost prohibitive for smaller systems. There are a number of well-respected approaches to conducting vulnerability assessments. One common practice is to use elements of the Sandia methodology and combine them with best practices for physical security assessments to create a custom process that is not only effective, but reasonably priced for a small- and medium-sized districts.
Why Outsource?
Water system vulnerability assessments are new to a lot of the players in the industry because they are predicated on events that many system managers never considered. Typically, the only thing a water system and its engineering partners had to guard against was a natural disaster, system fatigue and breakage or petty vandalism. Now, the EPA requires systems to evaluate their ability to withstand malicious attack by an individual or group wishing to cause physical or psychological harm to a system's infrastructure, employees or customers.
The foundation of the assessment is what's known as the Design Basis Threat. A system needs to know exactly what it is that it's guarding against, and assess the capacity of its facilities and personnel to withstand an expansive list of malevolent possibilities. Sometimes, however, it is difficult for a system insider without expertise in security to envision the full spectrum of potential threats, especially when there has been no past history of assessments against which to benchmark, like there might be with a high rise building, chemical plant, or other target.
Understanding the full scope of potential threats is one reason to outsource, but there are other reasons as well. While departments within the system can offer some specialized expertise, such as IT for SCADA and access control, and operations for personnel and engineering, each functional group sees the VA and security improvement process with its own prejudices. They also see the same system, every day, from the inside out.
An outside consultant, however, can wear a black hat for a day and view a system and its vulnerabilities from the outside in — exactly how a potential intruder might.
Further, with Department of Homeland Security High-Value Target Units paying what have effectively amounted to surprise visits on some water systems, a relationship with a security consultant can prove invaluable, as they can help the system manage the visits much like an accountant would during a tax audit process.
Choosing a Consultant
Because the smaller systems have to pay for their VAs out of pocket, many are not getting the same attention from the large design engineering firms or security consultants the big systems use. Their budgets are considerably smaller, and thus have a much smaller pool of qualified vendors bidding for their business.
Note how the size and sloping shape of this enclosure prevent intruders from climbing over or around this secure access point.
At the same time, because water systems serving between 3,300 and 50,000 customers make up the majority of systems across the country, a lot of new so-called security firms are jumping on the VA bandwagon to serve what they see as an emerging and lucrative market. So while these systems can find an outside vendor to perform the assessment, it may be the first and only water VA that this vendor has ever done.
The very nature of the VA process makes it difficult to assess a vendor's qualifications. The finished product, the assessment, is confidential. There can only be two copies, one at the water system and the other under lock and key at the EPA. A vendor that performed the VA is not even allowed to keep notes taken during the assessment process, so it can't show completed work as an example of its proficiency at performing water system VAs.
What system operators can look for, however, is a vendor's experience. In addition to experience conducting VAs specific to water districts of all sizes, look for physical security experience with other facilities such as chemical plants, hazardous material sites, high-rise office buildings or refineries. Vendors with this type of experience can leverage this institutional knowledge to help them complete the assessment thoroughly yet quickly.
This facility, currently undergoing improvements, is a strategic point in a system that serves a large community. The facility is also located along a heavily traveled recreational area. Concertina wire, in addition to being effective, also sends a message to potential intruders that a system is serious about its security.
A consultant that has performed both physical assessments and a number of VAs across a variety of district sizes can quickly spot patterns of vulnerability and extrapolate them throughout the system, such as insufficient locks or barbed wire, easily accessible tanks, disappearing fence lines, etc. Waters system operators should also check references and ask to see project lists.
Avoid using Proprietary "Technology Device Companies"
After the VA process, system operators must look ahead to determine how to best make security improvements required to mitigate the identified vulnerabilities. The improvement process is complex, and requires much more than technology solutions. The risk in selecting a technology device company to perform the VA is that many security devices and systems are neither scalable nor interoperable. A small system, already on a limited budget, cannot afford to risk purchasing an access control system – specified by a device vendor — that will need to be torn out and replaced if the system needs to upgrade.
By the same token, districts with tight budgets should also steer clear of security firms with no experience specific to water systems.
"Blended Expertise"
Many top-flight design-engineering firms are reluctant to work with smaller water systems because the scale and budgets involved make engagements with these systems less profitable. However, choosing a security consultant that has an existing relationship with a design build firm enables smaller districts to work with a first-rate firm at a much lower cost, because the security firm can manage many of the administrative duties associated with the assessment.
Here's an example of a system that has some superficial deterrents to thwart a casual trespasser, but may not be effective against a determined assault. The shape of the barbed wire on the left side of the gate and the broken tension rod suggest the gate has been compromised.
Engineering firms have their own approach for viewing security issues and mitigating risks. Most view the assessment strictly from a hydraulic standpoint: How much redundancy is there in the system, how many cutoff valves there are, how the system floats from one tank to another, as well as where the water is coming from and how it is getting out to the customer. Selecting a security firm with a design engineering firm partnership can allow each organization to allocate its respective strengths and be as cost-effective as possible for the client.
For example, one reason that the earlier rounds of VAs were inefficient was that both water system engineers and security personnel were doing site visits and attending meetings. Some water districts were large enough to afford this, but for small districts this level of redundancy is simply not an option. By choosing a firm with both institutional knowledge and a relationship with a design firm, systems can capitalize on the efficiencies of experience while still getting access to the specific expertise that their unique systems require.
While there a number of hardware and physical similarities across systems, the hydraulic components may vary dramatically. But rather than spending time on site, the security consultant's design engineering partner can perform a hydraulic review simply by looking at the system's plans at their home office, dramatically saving travel time and expense.
Conclusion
Managers of small water districts should not feel compelled to perform their Vulnerability Assessments in house simply on the basis of cost. Because the VA ultimately provides the foundation for the security improvement process, it is important to have the assessment done thoroughly and correctly by firms with the appropriate experience. Doing so can set the stage for sensible, long term cost-effective improvements, rather that patchwork fixes that are more expensive in the long run.
About the Author:
Terry Hall is a Sandia RAM-W certified Senior Consultant with The Steele Foundation, a global enterprise risk management firm headquartered in San Francisco. Hall is expert in conducting security assessments for water companies, and has more than 20 years experience in providing loss prevention control, physical risk assessments, threat and vulnerability analysis, electronic system design and evaluation, systems integration planning, and security operational evaluations.
