After September 11, officials with the City of Boca Raton, FL, were concerned with the potential for cyber threats and knew they needed a solution to protect the plant floor systems and network at the city’s water treatment facilities.
by Richard Horta
After September 11, officials with the City of Boca Raton, FL, were concerned with the potential for cyber threats and knew they needed a solution to protect the plant floor systems and network at the city’s water treatment facilities. That concern led to implementation of a new system that provided the needed security without losing any system functionality.
Boca Raton is a medium-sized utility consisting of both a water and wastewater facility on one campus. The plant control system infrastructure consists of a supervisory control and data acquisition system (SCADA) over Internet Protocol with offsite communications via telemetry systems. Included in the plant network infrastructure is a general corporate network for support staff and management.
The utility experienced a series of cyber security incidents that resulted in plant shutdowns. The problem reached a tipping point when the water plant’s SCADA system locked up, causing the plant to shut down. It took nearly eight hours to re-establish control of the plant. Because no monitoring system for network traffic existed, it was difficult to troubleshoot the source of the problem. The only conclusion plant managers and systems technicians were able to draw was that the network had experienced a data storm of some type.
Not knowing the problem’s root cause, the City of Boca Raton made various network upgrades with little to no luck in eliminating the issue. This “shooting-in-the-dark” approach became costly. Eventually the utility concluded that it needed a solution to better manage traffic and monitor security on the plant floor network because of the following conditions:
- The network consisted of a single backbone for both real-time process systems and general business systems connected directly to the entire city network
- The system had no IP separation between the plant floor process and the general business network
- The network did not have antivirus protection on the plant floor process nodes
- The network had no control or monitoring of traffic types or protocol
The initial proposal was to isolate the process control systems and network from the rest of the network. However, this alternative caused users on the general network to lose access to data on the plant floor network. Access to real-time operations data was vital to the core role of many users in the organization. Another suggestion was to create a demilitarized zone (DMZ) network configuration in which real-time plant floor data would be collected on a data server that would sit between the process network and the general network. The server would allow general users to get required data without direct access to the SCADA system itself.
Unfortunately, this solution was cost-prohibitive. It would require a new server and historian software that was compatible with the current SCADA HMI or Human Machine Interface system. In addition, the new historian software would have required a redesign for data management and reporting. This approach was going to create both a maintenance and a management nightmare that the city was not staffed for or prepared to handle.
During a visit to an industry conference, the City of Boca Raton information technology group was introduced to Plant Data Technologies, a Division of Verano Inc. After discussing its unique requirements with the Verano team, the city started a bid process to determine the vendor that would provide an updated SCADA system for the city. The process determined that the company’s Industrial Defender, a security technology solution designed specifically for the real-time control environment, met its network traffic management and security monitoring needs and was the best bid.
“It impressed us that Verano had a working knowledge of the process systems we use,” said Rich Horta, Senior Analyst/Process Control, for the City of Boca Raton. “This experience and knowledge gave us a secure feeling that they understood our situation.”
The Industrial Defender deployment consisted of installing switches to create a separate backbone for plant floor network; implementing security segmentation for the plant in accordance with best practices; installing Industrial Defender Guard perimeter protection and a security event management console to monitor and manage traffic as well as provide antivirus protection at the plant perimeter; and setting up new policy rules to manage traffic flow, access, and maintain a history of any violations to rule sets.
Deployment of the system was more cost-effective than any of the solutions proposed initially and did not compromise the functionality for general or operations users. In fact, deployment was a nonevent — users noticed no changes to their data access or work processes and experienced no down time to SCADA or other systems. Systems administrators also found Industrial Defender easy to set up, use and manage.
Using the Industrial Defender platform, Verano was able to safely implement a solution that met Boca Raton’s security needs without losing any of its system functionality. Plant staff did not experience a learning curve because no changes were made to the current SCADA system.
“During the installation of Industrial Defender, it was business as usual as far as plant operations were concerned,” said Gabe Destio, Boca Raton’s wastewater treatment plant supervisor. “We did not even know that the system was implemented until we were informed about it weeks later.”
About the Author:
Richard Horta has been in the water and wastewater treatment field for more than 24 years. In the past, he has held positions as a superintendent of wastewater treatment systems in Palm Beach County, and has also worked as a control systems designer and integrator. He has attained his skills through various specialized training programs. He holds a Class A Wastewater certification with the State of Florida and now specializes in the design, implementation, and maintenance of SCADA systems in the water and wastewater field. He is currently a Senior Systems Analyst / Process Control for the City of Boca Raton Utility Services.