By Stephen Rafferty
Every summer on Cape Cod we welcome vacationers and summer residents from far and wide. In the town of Falmouth, just across the sound from Martha’s Vineyard, the population balloons from 25,000 to more than 120,000 in the summer months. That’s great news for the local seasonal economy but it places tremendous demand on the town’s water infrastructure, which consists of nearly 400 miles of water mains, thousands of gate valves and hundreds of hydrants. In the winter months, the system delivers 1.5 million gallons per day on average; during the recent July 4th weekend, the system delivered 9.7 million gallons. To say demand is variable is an understatement.
A New, Modern Plant
In 2015, to keep pace with growing demand and quality standards, we recognized the need to move from our legacy unfiltered surface water supply to a new, state-of-the-art water treatment plant. The cost for the new plant was estimated at $50 million, a significant investment for a town of Falmouth’s size.
Fortunately, our community is very forward-looking. Falmouth is proud to be home to two world-class institutions, the Woods Hole Oceanographic Institution and the Marine Biological Laboratory. With actual Nobel Prize winners working and living here, there was a strong community commitment to building a plant that would meet our needs well into the future.
Reducing Inefficiency
As we moved from one type of operation to another, it required a new way of looking at our infrastructure. Our former system, which included a pumping station built more than a century ago to provide water from a local reservoir to the island steamships, had plenty of pumping capacity. If we had a problem with a pump or had to go to manual control, we simply turned on more pumps.
But as we designed the new treatment plant, building in excess capacity was not a viable option. It would mean millions of dollars of additional cost. You don’t add an extra 6 MGD and $14 million to the construction budget just because you need extra capacity for a few hours on peak summer mornings. We wanted to avoid that trap; our goal was to build a plant with adequate storage and a system that would run at a nice, economical, level rate.
Idle infrastructure was one thing we wanted to avoid; having idle staff was another. I didn’t want people sitting there all night long watching a computer. In addition to the impact this would have on the operational budget, I was mindful of the challenge of recruiting operators qualified to run a plant classified as a level 4 treatment facility.
Automation Was the Key
To achieve efficiency and reliability goals for the plant, automation was a core design priority from day one. I must admit, this pushed me into unknown territory. With nearly 40 years of experience in environmental engineering, I know water systems, not computer systems. Fortunately, I found some great experts to assist me close to home: two of my sons are senior IT professionals at major Boston-area institutions.
Our dinner conversations focused on what computer systems we were planning to use for the new plant and the topic of virtual computing was raised. In a virtual system, my sons explained, a single server hosts multiple “virtual machines” operating independently. This offers far more efficient use of computing resources than having multiple physical servers, reducing capital costs and simplifying management.
Assuming that virtualization was eventually where our control systems would end up, I asked the instrumentation engineers working on the project what it would take to implement virtualization right from the start.
Assessing Fault Tolerance
One question was what type of virtual server to use. The instrumentation engineers pointed out that the garden-variety servers the old pump station was using for our supervisory control and data acquisition (SCADA) system offered about 99 percent availability. This translated to around three instances of unplanned downtime every year. Considering the impact of downtime on residents and the local economy, this level of reliability was not acceptable.
Availability was of particular concern given the automated nature of the plant design. A highly reliable system would help me persuade the Commonwealth of Massachusetts that the plant could be run on an automated basis and not require having an operator sit there around the clock during the peak summer months. Achieving this approval was pivotal to the plant’s success.
The engineers with Rockwell Automation, our automation partner on the project, suggested fault-tolerant ftServers from Stratus Technologies. Built specifically for “edge computing” applications — operating in automated environments in the field where continuous availability is required — these servers are designed to deliver 99.99 percent uptime while offering all the advantages of virtualization.
Managing Systems Remotely
With no IT professionals working at the plant, it was important that our computer systems not require specialized skills on-site. The ftServers are maintained and serviced remotely by Stratus.
The system also enables plant personnel to keep an eye on things remotely. We have an operator on call every day; when they go home, they take a laptop with them. They are able to monitor plant and system operations, receive any critical alarms and make operational changes remotely (with the exception of chemical feed systems due to state regulations). That means faster response to issues requiring attention. Everyone employed at Falmouth’s water treatment plant can get there within 45 minutes. But why waste the better part of an hour making a decision — like taking one pump off-line, bringing a different one on-line and putting it into automatic control — that could take only a few minutes to make remotely?
Major Cost Savings
Since the new plant became operational in the spring of 2017, it has operated flawlessly. Thanks to the automation systems, the plant nearly runs itself — and does so very efficiently.
We estimate that the high level of automation has saved the town around $3 million annually in avoided costs for salaries and benefits. Most importantly, these savings are achieved without compromising plant reliability.
The new system was put to the test in March 2018 when a violent winter storm hit the Cape, bringing down trees and causing widespread power outages. I received a text message that indicated one of the servers was down. The fault-tolerant servers operated as intended and there was no disruption to operation. When we got to the plant, it turned out the cause was human error. During routine testing of one of the uninterrupted power supplies (UPS), someone forgot to turn it back on. Nothing else was impacted.
Streamlining Reporting
The new system also improves the accuracy and efficiency of complying with state reporting requirements. In the past, our plant operators relied on handwritten notes because past data losses had left them wary of relying on computers.
When the new plant came online, operators still wanted to keep paper notes and transfer them to the monthly reporting forms rather than rely on the computer printout. But there were some slight discrepancies between the paper and the computer. After about six months with the new system, the operators came to accept that the computer was always right. They also realized that critical data would be protected from loss, reducing their reliance on paper.
Fulfilling the Mission
Now that we are entering our third year of operation, we are confident that the automated treatment plant was a great investment for the community. It will enable us to deliver on our mission to the public for years to come.
Providing safe, reliable water to the citizens of Falmouth should be invisible. It shouldn’t be something they worry or even think about. Having a computer system as reliable and state-of-the-art as it can be makes our job easier, helping us ensure people get clean, safe water every time they open their taps. WW
About the Author: Stephen Rafferty is the water superintendent for the town of Falmouth, Mass.
