By Larry Pomatto
Comprised of 27 member agencies that provide nearly 60 percent of the water used by nearly 18 million Southern Californians in 240 cities, the Metropolitan Water District of Southern California (MWD) is a regional water utility with a service area covering 5,200 square miles in Southern California.
MWD provides water to communities from Oxnard, Calif., to the Mexican border, including all or parts of Ventura, Los Angeles, Orange, San Bernardino, Riverside, and San Diego counties.
Possibly the world's largest water filtration and distribution organization, MWD uses more than 775 miles of pipeline to distribute water from two sources. Water from Northern California runs into Castaic Lake and Lake Perris, the west and east branch terminal reservoirs of the State Water Project. The 242 mile long Colorado River Aqueduct moves from the California/Arizona border to Lake Mathews, near the San Diego county border. MWD operates five filtration plants: Joseph Jensen Plant; F.E. Weymouth Plant (520 mgd); Robert A. Skinner Plant (520 mgd); Robert B. Diemer Plant, Yorba Linda (710 mgd); and Henry J. Mills Plant, Riverside (326 mgd); five pump stations spread across California's southeastern desert and mountain ranges that lift water a total of 1,616 feet; 15 hydrogeneration facilities with a capacity of 102 megawatts; and two operation control centers.
Originally organized in the 1920s, MWD still occupies sites built at that time, and uses technologies from that date forward. In the late 1980s, MWD went to bid to upgrade its system which included over 150,000 I/O points, and a variety of RTUs at the filtration sites and along the water distribution system. As part of the upgrade requirements, the existing system had to remain functional until the replacement system was completely operational.
The first challenge MWD faced was finding an integrator who could meet information collection needs and provide control at divergent sites without impacting water customers. It was essential that the integrator be able to link old technologies with new solutions. Further, the system needed to be flexible, expandable, and one that MWD staff could maintain and upgrade.
In January 1995, the project was awarded to Systems Integrated (SI) of Orange, Calif. Under the same ownership and management since 1974, the company focuses exclusively on the utility industry and has software and hardware engineers capable of handling a project from design through engineering and into warranty periods.
"We brought our expertise and MWD brought an open mind to the negotiation table," said Susan Corrales-Diaz, Systems Integrated chief executive officer. "We brought working knowledge of a number of technological breakthroughs to the project. We offered MWD an opportunity to move away from the limited industry standards that continue to define SCADA systems, into a new world of information management at a scale far beyond the initial project scope.
"Our expertise, combined with MWD's willingness to accept new technology, allowed both organizations the opportunity to implement industry-proven networked solutions for one of the world's largest and most complex water delivery systems."
Based on the same technology the Internet uses, SI's solution allows every server, workstation, and RTU in MWD's system— all of which are referred to as nodes— to directly communicate with each other. If one node in the network fails, all other nodes continue to function normally.
"The system is designed to be resilient to natural disaster," said Larry Pomatto, chief engineer for Systems Integrated. "California is prone to earthquakes which can do major damage over a wide area. The networked peer-to-peer system provided to MWD allows them to effectively manage the entire system from any facility."
The Realtime Utility Management, Information, and Control (RUMIAC) system supports redundant hardware, software, network, and communications. If a redundant server fails, the backup server takes control. If one workstation at a site fails, other site workstations can access the same information and provide the same control. If an entire treatment plant goes down— or the associated Wide Area Network (WAN)— any other networked treatment plant may take control.
To provide rapid and secure network communications, SI recommended frame circuits. Originally designed by and for the banking industry, frame circuits support secure, high speed, reliable communications. Although available for years, frame circuits were not cost effective until technological advances brought prices down.
The MWD system uses onSITERegistered software, an off-the-shelf software package that is modular in design and easy to upgrade. SI staff provide custom software requirements as upgrades to the software.
"Every time we make a modification for a customer, we add that functionality into the core onSITE, making it a part of the shrink-wrap software," said Tom Waldowski, lead software engineer at Systems Integrated. "The result of this approach allowed MWD to avoid waiting for upgrades."
The software's modules are function-specific. Each module can be assigned to any node or the functionality of the modules can be distributed to all nodes in the system. For example, the Alarm Manager, Event Server, and Message Logger can operate on one node, or can be routed to three different nodes anywhere in the network.
Taking advantage of Onsite's functionality, MWD's communication systems evolved to include microwave, radio, cellular telephone, and satellite links, in addition to the frame circuit network.
The system also features "smart" RTUs with 128 megabytes of memory, 4 gigabytes of non-volatile memory, and from 200 to 500mHZ 64-bit processors.
"Why spend thousands of dollars on a PLC when we could give MWD the same high level of reliability, more programming capability, and significantly more computational power by putting a computer in the RTU?" Pomatto said.
Using computers in the RTUs also made sense given the tight turn-around time. Because the Intelligent RTUs run onSITE, MWD spent less time on implementation and programming, as the staff only needed to learn programming for one product. Using RTU computers also allowed for faster implementation, as the software was the same at all different levels. The RTUs— like any other computer in the system— can perform any functions required within the system. In a matter of minutes, MWD is able to re-configure any computer or RTU to fulfil any role necessary. This means that MWD can add a monitor, keyboard, and mouse to the computer at the RTU and have a fully operational workstation or server.
Before implementation could start, SI had to provide a solution to MWD's requirement that the existing system remain operational during the implementation. In essence, the field RTUs had to be accessible by both the old and new systems.
"Our solution was exactly the same as what any water agency does when flow needs to be diverted— we used a T-connector and valves," Pomatto said. The "T-connector" in this case linked the field RTUs, the existing front-end processor, and the software system.
"During the initial phase, MWD sent data to and received data from the field RTUs using both the old and new systems. Often, MWD monitored and controlled the field RTUs using onSITE during the day and the old system at night. When MWD was satisfied that onSITE met the specifications, they simply turned off the old system and pulled it out."
MWD formed a group of operators and technicians to develop graphic screens. During a six-week session, MWD staff learned to use the software's Graphic Editor, created a library of dynamic symbols, set navigation paths and identified jump points, devised color parameters, and developed a style guide for future builds.
"MWD had the right idea: the people who use the screens should design them," said Josie Armstead, SI's documentation and training coordinator. "We've had other customers use programmers and engineers to build screens. During training— the first time the operators and technicians saw the system— I nearly had a riot on my hands. Objects did not reflect alarm conditions or show that points were blocked. The designers had to revisit and restructure every screen in their system."
The Graphic Editor allows MWD's screen designers to create a generic object— for example, a pump or a turnout— add animation dynamics such as color changes, flashing, and size changes; set variable links to the database; and save the object. Once created, the same object can be copied and used again and again at any facility. When the designers build the final screen, the system resolves the variables, automatically creating the necessary database links. They can also copy any screen in the system, reset the variables, and have a new, fully configured screen for the next site within minutes.
MWD's RUMIAC uses two communication paths: a Local Area Network (LAN), and a Wide Area Network (WAN). In the treatment plants, the workstations, servers, and RTUs communicate using a fiber optic Ethernet LAN. The legacy system used a polled communication approach which was left in place while MWD staff pulled fiber optic cable for site RTUs. Inter-treatment plant communications use a frame circuit WAN, installed by the telephone company in parallel with other activities. The result is a 10/100 megabyte fiber optic Ethernet communication backbone in each treatment plant, and a redundant frame circuit Wide Area Network (WAN) connecting the various treatment plants.
Once the fiber optic backbone was in place in the treatment plants, the RTUs were installed and connected to the Ethernet network. When the Intelligent RTUs were fully configured, system functionality was transferred. The approach was logical and orderly, as it took plant functions and switched them over one at a time to the new system.
About the Author:
Larry Pomatto is Director of Engineering for Systems Integrated. An Electrical Engineer, he has worked in the automation industry for 25 years, helping design and install distributed control systems for water, electric and telephone utilities. Pomatto holds more than a dozen patents on a variety of control designs and equipment.