This post was written by Chad Harper, senior director of technology for MAVERICK Technologies, a Rockwell Automation company.
Process manufacturing plants are famous, some would say notorious, for their tendency to keep automation systems and associated networks up and running for decades. Some estimates suggest that easily half of the automation systems controlling plants in North America have parts that are at least 20 years old, and 30-year old systems are not rare. Those systems do not look like they did when they were initially installed.
Hard drives, monitors, and keyboards that receive constant use simply do not survive over decades, and that old equipment is not available. If you ask your local computer store for an IBM XT motherboard and a 20MB hard drive, you will get some puzzled looks. So unless your automation systems are brand new, they will be multigenerational due to obsolescence and devices wearing out.
In some cases, old subsystems and components may be replaced with newer equipment, such as with the human-machine interface (HMI) subsystem, that has higher functionality. That kind of upgrade will probably include new PCs, with operating systems capable of supporting functionality that was not available when the original HMI was installed. With the long operating history of many automation systems, you may be experiencing problems due to system providers going out of business or being acquired by other companies.
If you think back to the 1980s or even 1990s, the landscape of companies providing control systems was different than it is now. If you are working with an automation system from one of those more or less extinct companies, you are probably very aware of this reality. Getting the multigenerational and multivendor components of an automation system to work together can be difficult. But that approach may be the only option when there are not any practical alternatives, and end users must be prepared to deal with the inherent challenges.
At some point, an old automation system that has not been upgraded will become a serious threat to production. As printed circuit boards and network devices get older, individual components begin to fail, and systems fault more often. These cause unscheduled shutdowns and outages that are especially disruptive when replacement parts are not available. Companies still specialize in recycling parts for these old systems, and there is always eBay, but over time supplies get tighter and tighter. Individual components, especially the chips, are long out of production and cannot be replaced.
The automation vendor may try to create some sort of functional replacement, but redesigning an old board with new components is expensive, which will be reflected in the price. If you know your plant is going to be shut down or go through a major redesign for some specific period of time in the near future, you can limp along with the old system until that date. But that is not an appropriate long-term strategy, and eventually you will have to upgrade or migrate. For purposes of this discussion, in an upgrade you add newer elements but stick for the most part with the same automation system vendor and platform. In a migration you make a major platform change, typically involving a different vendor, but sometimes sticking with the same company.
Replacing some part of an automation system with something newer typically follows one of two patterns. In one, companies use multigenerational systems, which some automation vendors have done a good job creating. This approach typically happens when the original company is still in business and has created upgrade strategies for users to add new parts to an old platform, gradually bringing up the whole system on a modular basis and without any major disruptions. This is not easy for the vendor to do and requires a conscious effort to make it happen.
It is really a good way to update your automation system, but it is not possible in every case. Even if you are remaining with the original vendor, it is still important to be aware of the support dates for each piece of the automation system. Often, certain controllers or nodes lose support before others, so you have to plan the timing of the upgrades. In the other pattern, new elements from a different vendor can be “bolted on” to an existing automation system.
This is a difficult thing to pull off in the real world. Such a system can be buggy, and it will not have the greatest vendor support. Based on our company’s experience working with many different types of automation platforms, these bolt-on solutions are rarely successful over the long term. We view bolt-on solutions as temporary Band-Aids, either to add a few years to the older system, or as a part of a larger phased migration plan that will change the bolt-on components to an integral part of an entirely new automation system.
Of course, bolt-on solutions may be necessary for discontinued legacy systems, but instead we often see them as the result of a vendor’s aggressive cross-platform competitive attack. In our experience, one supplier trying to dislodge another often uses a bolt-on solution as a means to get in the plant and establish a position for getting deeper into the system. Unfortunately, we have seen too many situations where the effectiveness of these solutions was overpromised in the sales and marketing process. So if you find yourself needing to make an upgrade, but you are not ready for a “rip and replace,” you may have to follow the path of a cross-platform, multigenerational upgrade or migration to your automation system.
There will be complications, but they can be managed if you develop a comprehensive project plan ahead of time. It is really a good way to update your automation system, but it is not possible in every case. Even if you are remaining with the original vendor, it is still important to be aware of the support dates for each piece of the automation system. Often, certain controllers or nodes lose support before others, so you have to plan the timing of the upgrades. In the other pattern, new elements from a different vendor can be “bolted on” to an existing automation system. This is a difficult thing to pull off in the real world. Such a system can be buggy, and it will not have the greatest vendor support.
Based on our company’s experience working with many different types of automation platforms, these bolt-on solutions are rarely successful over the long term. We view bolt-on solutions as temporary Band-Aids, either to add a few years to the older system, or as a part of a larger phased migration plan that will change the bolt-on components to an integral part of an entirely new automation system. Of course, bolt-on solutions may be necessary for discontinued legacy systems, but instead we often see them as the result of a vendor’s aggressive cross-platform competitive attack.
In our experience, one supplier trying to dislodge another often uses a bolt-on solution as a means to get in the plant and establish a position for getting deeper into the system. Unfortunately, we have seen too many situations where the effectiveness of these solutions was overpromised in the sales and marketing process. So if you find yourself needing to make an upgrade, but you are not ready for a “rip and replace,” you may have to follow the path of a cross-platform, multigenerational upgrade or migration to your automation system. There will be complications, but they can be managed if you develop a comprehensive project plan ahead of time.
Why do people upgrade or migrate? Today, the biggest drivers are life-cycle issues. Adding functionality is usually a distant second, unfortunately. We show customers where there can be improvements, but adoption of those recommendations during a migration project is pretty rare. The justifications for adding improvements do not often make it into the business-case argument in a compelling manner, or there may be other corporate financial issues that preclude all but the essential expenditures.
Years ago (prerecession), companies might have gone to management asking to begin migrating before a complete collapse of the old system. They would point out how the new capabilities they were getting would improve production and cut operating costs. That sort of thing does not happen as much today. Companies will rarely even discuss an upgrade or migration until the old system is coughing up blood.
If you have a plant that you want to run reliably and safely for years to come, you have to think of any solution you consider as permanent and act appropriately. That is one reason why bolt-on solutions disappoint, because they often become permanent. Once word gets out that you are considering automation improvements, vendors will come with sales pitches, simulations, and lots of promises. An integrator can help sort through the hype and determine what is actually possible with unbiased assessments of what products and migration paths actually work.
Cross-platform, multigenerational projects are not easy, and you will need all the help you can get. All vendors can show you good projects, but each one has also had disasters, which they of course do not mention. On the other hand, an independent system integrator can speak freely with you about both the successes and failures of a particular vendor. As you consider launching a project, there are a few basic questions:
Some suppliers can and have oversold the capabilities of bolt-on approaches, so sometimes an integrator has to come in and be the voice of reason. That can mean dumping cold water on some of the sales pitches, a necessary if often unpleasant task.
Do not underestimate the value of having equipment that is still being manufactured. You might have forgotten how nice it is to be able to buy spare parts off the shelf. With that in mind, you need to think about what you are buying in the context of a 15- to 20-year lifespan and associated total cost-of-ownership issues. Does the platform have a guaranteed support date? Does the vendor have a solid record of supporting its platforms for as long as planned?
Is this a newer model automation system, or does the vendor have another automation system already in development that will soon replace the current offering in both support and focus? New equipment will bring new functionality. Things will not be exactly as they were, nor should they be. There will typically be many opportunities for improvements when obsolete components are replaced with newer offerings. We often bring in operators early in a project to educate them about how a new system can be used to improve operations. This typically includes better graphics, improved techniques for interacting with the HMIs, and simpler alarm management.
We explain the concepts of high-performance graphics, supported by findings from the Abnormal Situations Management Consortium, and show what they should expect to see when the new automation system is running. While it is OK to keep your basic control strategies, I/O, and field devices constant for the most part, you do not want to replicate all of the old functionality in the new automation system.
Trying to make a new system behave like an old one ignores many useful improvements, can create a maintenance nightmare, and usually requires going to great lengths just to make everything work. Even so, we sometimes have to preach to our customers and convince them to use the built-in capabilities in the new automation system. We need to make a convincing argument about why each benefit is better for their specific situation, show them where the technology is going, and explain why deviating from a vendor’s preferred path can be very expensive.
Before you begin to explore available solutions or listen to the first vendor presentation, you have to dig deep in the planning process by answering these and other questions:
Companies frequently get in trouble right out of the gate, because they do not evaluate their controllers and networks adequately. New systems invariably create more network traffic and require more processing power, and not all existing systems can handle the extra load. Here is an example of what can happen if a proper up-front evaluation has not been completed:
Realizing that might be a possible outcome, another company decides to have both HMI systems work in parallel, so that the old one can serve as a temporary backup. However, both crash because having both systems requires even more network resources. There is no substitute for proper planning and testing up front, as a relatively small amount of time and money spent on these tasks can prevent disastrous results down the road.
There is no universal best platform or solution, because each one has its particular strengths for specific types of applications. There is also no single leading automation system or migration path, as many factors lead to the best choice for each project. There is much at stake in such projects, because the loss of a week or even a couple days of production can be worth far more than the cost of the project. In fact, downtime costs often make a true rip-and-replace approach infeasible due to the economics of lost production, forcing a phased upgrade or migration.
Every situation is different, and what works in one facility may not work for you. Success comes from a combination of a deep understanding of the process and control strategy, the shape your controllers and networks are in now, and what functionality you need to add. You may need to work with an integrator or other unbiased advisor who can give straight answers for your specific situation. All this needs to be done very early in your planning process, so that the projects can be scoped, scheduled, and budgeted correctly.
About the Author
Chad Harper currently serves as the senior director of technology for MAVERICK Technologies, managing the subject matter experts for various control platforms. Chad has a background in advanced process control, DCS migrations, and operations management.
A version of this article also was published at InTech magazine.