This post was authored by Graham Nasby, ISA leader and specialist in water and wastewater industrial automation technology, and Matthew Phillips, P.Eng., is a licensed professional engineer.

The need for standardization led the City of Guelph’s Water Services department in Canada to embark on a multi-year program to modernize and enhance its Supervisory Control and Data Acquisition (SCADA) system using a standards-based approach. Starting with a comprehensive SCADA Master Plan, the city developed a set of SCADA standardization documents that were incorporated into its procurement program. The result was a smooth and orderly transition from an aging feature-poor infrastructure to a standardized SCADA system designed to meet present and future needs.

Need for SCADA Standardization

City of Guelph's Clair Rd pumping station

SCADA has become a vitally important tool for the operation, management, and monitoring of public water utilities throughout North America. SCADA now has an ever-present role that includes looking after all automatic control and alarm management, logging of critical process data, and providing operators with remote access to equipment.

However, unlike many other fields of engineering, SCADA is a comparatively newer field with many of its current features only becoming available in the last 10 to 15 years. With SCADA only recently maturing as a technology, most water utilities tend to have a wide range of installed SCADA equipment/networks that can vary considerably in terms of age, feature-set, connectivity, and vendor support. In addition, most systems have been built piecemeal over the years by a wide variety of contractors, vendors, and consultants, each with their own programming approach.

The result: many water utilities now find themselves with complex, varied, and comingled systems that are often difficult to manage in terms of operations, maintenance, and overall system robustness. In 2003, the City of Guelph realized a new approach to SCADA was needed for its waterworks. Their SCADA equipment worked, but it was difficult to maintain in a consistent fashion, and it required a large ever-growing investment in terms of staff training, staff time, and the use of external consultants.

Equipment such as historians, human-machine interfaces (HMIs), operator interface terminal (OITs), programmable logic controllers (PLCs), remote terminal units (RTUs), and SCADA network infrastructure came from a variety of vendors, and each individual piece of equipment had its own custom programming, wiring, and support requirements. Furthermore, the city staff who had painstakingly built the existing system wanted a change so they could harness new SCADA technology and not be hobbled by the complexity that had plagued them in the past. These considerations, as well as the increasing demands for data-logging to meet regulatory requirements, drove the city’s need for a more uniform SCADA system.

The road to standardization

Seeing the need to develop a plan for the future, the city embarked on a comprehensive study of its SCADA needs for the next 20 years. All current and potential stakeholders of the SCADA system were invited to participate. This included operations, maintenance, engineering, and management, as well as the regulatory compliance and procurement departments. The result was a draft SCADA Master Plan in 2003, a standardized SCADA hardware platform, and a draft set of standards documents. The following two years were spent testing out the new standards on a variety of pilot projects.

These ranged from the construction of additional UV disinfection systems to complete control system replacements for several remote wells. Total design time for SCADA and controls integration was cut significantly, and the process of FAT/SAT testing was simplified by the use of standardized control narrative formats. Building on these successes, the city released its first set of official SCADA standards in 2005 and made them a mandatory component of the contract documents for all City of Guelph waterworks capital projects. As the city accumulated more experience with applying its SCADA standards, the lessons learned from each project were continually merged back into the SCADA Master Plan and the standards themselves.

Rather than acting as static contract specifications, the standards were treated as "living documents" that evolved as the city's needs changed. Unlike the former approach of always going for the "latest and greatest," any major shifts in technology were now undertaken only within the long-term outlook of the SCADA Master Plan. In addition to providing the framework for the city's standardization program, the SCADA Master Plan also greatly simplified the process of making strategic automation investments. Instead of rushing to upgrade all of its existing sites at the same time, the master plan prioritized and aligned individual site upgrades in a long-term context.

For example, upgrading critical sites that had older technology that was problematic to maintain was made a high priority, while other older sites that were not yet causing significant issues were put on a more long-term replacement schedule. The SCADA Master Plan continues to be an important tool used by the city for planning and scheduling its automation upgrades.

Developing your standards

The key to developing standards is to weigh the time investment required against the potential benefits. Too many standards documents, or too much detail, can consume staff resources and prevent you from being able to innovate. Too few standards, or standards that lack sufficient detail, can be just as bad as having no standards at all. Looking to its SCADA Master Plan, Guelph decided to develop the following standards documents:

• New system-wide universal tagging standard
• Standardized control panel and field-wiring designs
• SCADA network equipment/addressing guidelines
• List of approved PLC hardware configurations
• Single approved OIT with code templates
• PLC programming standard with code templates
• Standardized suite of SCADA/HMI software
• SCADA/HMI programming standard with code templates
• Standardized format for process control narratives

The city was careful to only develop standards it felt would deliver good return on investment. Each standard was tied to an area in the SCADA Master Plan where the city had identified a need to improve. Some standards were easily formulated by collating specifications from past projects; others were brand new documents and templates that had to be developed from scratch.

Selecting your SCADA platform

One of the most important parts of the standardization is selecting the SCADA hardware, software, and network platform to use. By reducing variation in your system, staff training requirements and the use of external consultants can be significantly reduced. However, it is important to remember standardization does not necessarily mean sole-sourcing everything from one vendor; sometimes a mix of products from various vendors is the best solution. Whether you go with a vertical solution from a single vendor or a collection of products from various vendors will depend on the specific needs of your SCADA system. Factors Guelph considered when selecting its SCADA platform included:

• Feature sets vs. needs
• Connectivity to other systems
• Cost and availability
• Impact on current staff in terms of skill sets and training requirements
• Local vendor support
• Vendor's road map for future support and future product releases
• Availability of local system integrators with the required skill sets
• What worked in the past
• Ease of integrating it with the existing system during the transition period

Case studies

The transition from the city's old SCADA system to the new standardized one took careful planning, creativity, and hard work. Like most public utilities, the city had a limited annual budget to spend on new infrastructure, and these funds had to be split between capacity upgrades and scheduled equipment replacement. The cost-sensitive and political nature of public utilities also meant a large rate surcharge could not be applied to finance a short-term upgrades program. To further complicate matters, most of the city's water facilities were in constant use, so downtime had to be minimized. The result was a roadmap to standardization that was planned out as a series of smaller projects over a 10-year period. Some of the upgrades completed by the city are discussed below.

Core infrastructure upgrade

To start its standardization upgrade program, the city selected two core elements that needed to be addressed immediately-its aging SCADA network and centralized SCADA servers. The challenge with the network was it was vast and would require considerable effort to replace.

Rather than trying to replace everything at once, the city decided to upgrade the network backbone that connected in SCADA servers first and then upgrade the rest of the network over six years. For the new network, the city chose Ethernet because of its wide availability, vendor neutrality, and the availability of a local high-quality fiber-optic network provider. The replacement and reprogramming of the city's core SCADA servers was broken down in a series of smaller projects.

In the first project, the new servers were installed, but the existing HMI, alarming, and data collection applications were only migrated "as is" into the new environment. Then, the detailed process of updating the HMI screens, scripting, tags, and alarms to the city's new HMI programming standard was tackled as a series of smaller projects over the following years.

Facility PLC upgrade program

When Guelph embarked on its standardization program, the city had a wide variety of PLCs in use throughout its waterworks. Using a prioritized approach that started with the older and more exotic PLCs and finishing with the PLCs that only required reprogramming, the city adopted a program where facilities would be upgraded one at a time. At each facility, a phased approach was used because taking it offline could only be done for short periods of time. First, new control panels with the new PLCs and field wiring were installed but not connected.

Second, new programming code was installed on the new PLCs but only run in simulation mode in order to test the new control philosophy. Once testing was complete, equipment in the plant was then switched over to the new pre-wired field wiring and the old field wiring disconnected. After a verification period, the old field wiring and panels were then removed. As of Spring 2011, this facility-by-facility approach is still ongoing.

Well field upgrade

In many ways, using SCADA standardization does not change how capital projects are undertaken. Projects are still envisioned, budgeted, designed, and tendered out as projects. The only major difference is the SCADA standards are now part of the tender documents. A typical project where the SCADA standards were used was the addition of two new bedrock wells to the city's Arkell Springs well field. More than half of the city's water comes from the well field, so it was vital the addition of the new wells and the associated control system upgrade went smoothly.

It turned out to be a perfect example of how the new SCADA standards contributed to lower design costs, simplified software development, and a streamlined commissioning process. "In many ways, the Arkell Springs Upgrade was the same as any large capital project, except that the SCADA portion required very little management compared to other parts of the project," said Vincent Suffoletta, supervisor of Supply for the City of Guelph's Water Services Department. "The SCADA standards resulted in real-time savings for both our in-house and external system integrators when it came to system design, programming, and testing. Our operators also liked the consistent HMI screens, which made their jobs easier. Furthermore, we incorporated the lessons learned from the project back into our standards documents as part of our continuous improvement program."

New pumping station

Realizing automation hardware/software products do not last forever, the city's SCADA Master Plan considers the entire life cycle of the SCADA technology they use. The life-cycle planning includes all components of the SCADA system, from initial installation through ongoing maintenance to eventual end-of-life replacement. In 2009, the construction of a new booster pumping station provided an opportunity to test a "next-generation" PLC platform that had been identified in the city's SCADA Master Plan. Using the new PLC platform was a learning experience that yielded valuable lessons about the new technology.

These lessons were incorporated into the city's SCADA standards immediately after the project was complete. The city now includes this new "next-generation" PLC as one of the options in its standardized PLC hardware and software specifications.

Advice for others

Looking back at its SCADA standardization program so far, the city offers the following advice:

• Before undertaking a standardization program, develop a SCADA Master Plan. The master plan will help focus your vision and serve as a guide.
• Select standardized hardware, software, and network SCADA platform using a life-cycle approach. Short-term cost savings do not always make sense over the long term.
• Write standards documents to suit individual needs. Tailor the level of detail to what is needed to implement a master plan and to enforce consistency in a new SCADA platform. The point of the standards documents is to make them work for you, not the other way around.
• Programming standards for HMIs, OITs, and PLCs must consist of written specifications accompanied by code template files. Most programmers, whether in-house or part of a system integration outfit, prefer programming to reading. Help yourself by helping them. Provide pre-configured "base load" programs, working code examples, and a large library of pre-built widgets, modules, and screen components. If there are not programming resources in-house to do this, consider hiring a system integration firm to help you develop the standards.
• Use the SCADA Master Plan to help develop a strategic timeline for updating the older nonstandard parts of the SCADA system. Rather than rushing to try to upgrade everything all at once, prioritize potential upgrades in terms of the goals of the SCADA system and schedule accordingly.
• Standardizing does not mean stifling innovation. Use the SCADA Master Plan as a "living document," and review it at least annually. When new technology becomes available, look at it in terms of the entire life cycle of the SCADA system and how it can fit into the SCADA Master Plan. If a technology looks promising, incorporate it into the plan and test it out in a pilot project. Then review and evaluate the results. If it turns out to be the direction you want to go, revise standards documents accordingly and incorporate it into your workflow.
• Once SCADA standards are written, use them for every capital project whether it is new construction or upgrades to an existing site. To work effectively, the SCADA standards must be included as part of the contract documents for every project undertaken.

Steps from here

The City of Guelph's Water Services department sees SCADA standardization as a continuing part of their operations, maintenance, and capital-improvements workflow. Two years ago, the installation of a new version of their HMI software was identified as a priority in their SCADA Master Plan. A project to complete the upgrade will be proceeding in the summer of 2011. To guide the roll out, the city is updating its HMI standards document, so the new HMI functionality can be effectively harnessed for the upgrade and all new projects moving forward. The process of SCADA standardization never ends, but neither do its many benefits.

About the Author
Graham Nasby, P.Eng., FS Eng., PMP, CAP is an industry-recognized leader in the water/wastewater community for his efforts with DCS/SCADA systems, standards development, raising cybersecurity awareness, and alarm management. Through his work with the International Society of Automation (ISA) and IEC, he has coauthored international standards in alarm management, cyber security, and HMI (human machine interface) design. He has also worked with the ISA, American Waterworks Association, Water Environment Federation, and other industry groups to author numerous articles on SCADA best practices, led the annual ISA Water/Wastewater and Automatic Controls Symposium, and contributed to other industry events.

Connect with Graham

About the Author
Matt Phillips, P.Eng., is a licensed professional engineer who has worked in the field of SCADA and software systems for more than 10 years. As the City of Guelph’s Water Security Coordinator (SCADA), he manages the planning, construction, operation, and maintenance of the SCADA infrastructure for the city’s Water Services department.

Connect with Matt

A version of this article also was published at InTech magazine.