The following technical discussion is part of an occasional series showcasing the ISA Mentor Program, authored by Greg McMillan, industry consultant, author of numerous process control books, 2010 ISA Life Achievement Award recipient and retired Senior Fellow from Solutia Inc. (now Eastman Chemical). Greg will be posting questions and responses from the ISA Mentor Program, with contributions from program participants.

Here we see how a general and open-ended question can lead to a very insightful, useful, and at times, humorous discussion on a critical phase of migration projects where automation is transferred from the old to a new system with many safety and performance implications.

The question was posed by Marsha Wisely, a protégée. The answer is provided by Hunter Vegas, ISA Mentor Program co-founder, who has the most extensive practical experience of the program resources in automation project design and execution.

Marsha Wisely’s initial question

I am looking for general information on the different types of cutovers and what the impact of each type may be. If this topic is too big, I would appreciate even some quality documentation on the topic such that I can read up and maybe come back with more specific questions.

Hunter Vegas’ initial answers

It IS a pretty open-ended question, and the topic is rather huge. Books by practitioners are more useful because they reflect actual plant experience. Greg adds that academics who have partnered with industry have expanded our understanding and principles through theory supporting practice. You get good at cutovers by doing them – lots of them – and learning (sometimes the hard way) what works and what does not. Older folks in automation have lots of gray hair, or none at all, and plant startups are most likely to blame.

There are several universal constants that you must know about startups: a) Instrumentation and automation are almost always the last thing to install and check out because you can’t hook up instruments on pipes that don’t exist, and you can’t talk to instruments if there are no wires. Invariably the civil, mechanical, and electrical installations all get delayed on a cutover but the start date does not slide so instrumentation is ALWAYS critical path. You start off with three weeks to install your equipment and that usually gets whittled down to three hours.

“You mean you aren’t DONE yet?!?!?!” b) You can’t run the plant without instrumentation so if your manager keeps hounding you about when you’ll be done, tell him that you absolutely promise that you’ll be done before he starts up. c) Regardless of who is on what shift, the night shift is almost always half as productive as the day shift. d) Good, Fast, and Cheap (pick any two). e) There is no such thing as too much coffee on a startup.

Cutover types

The different “cutover types” isn’t nearly cut and dried or simple. I’ll start by talking about a control system retrofit as opposed to a new plant or a major plant expansion. Usually a retrofit has a couple of different types: 1)  Software upgrade – same instruments, same hardware (or maybe a few upgraded computers), and a new revision of software. 2)  Hardware upgrade – same field instruments, new hardware and software. (Maybe same vendor, maybe not) 3)  Total revamp – upgrade/replace a lot of field instruments and the entire control system.

All of those can be done in various ways that have different impacts on the production plant. Nearly all COULD be done without shutting down the plant. It would be expensive, and it would take a while, but it can certainly be done. I’ve converted three or four plants from pneumatic controllers on panelboards to full-blown distributed control systems (DCS) and never shut them down. Ultimately, the economics of a plant production drives the decision. If they are making a million dollars of product a day, they can afford to pay an extra $200,000 to cutover on the fly. Similarly, if they are barely running eight hours a day, a two- or three-day outage doesn’t matter. Often the cutover is a mix – you do as much work as possible prior to shutting down and then you come down and finish the rest.  

How do you cutover?

That totally depends on the plant, the process, and the timing. When I am looking at a retrofit project, I start by talking to the plant and understanding a few things: 1)  Their budget 2)  Their schedule 3)  The financial justification for the project 4)  What their “pain points” are Based on that, I then start going over the project – looking at panels, mulling my options, determining what I have to do, and how I might go about it. I also investigate what options I have and what is required to change over the system. Is the I/O compatible?

Are there interconnect options and do they work? (Just because a company SAYS they offer interconnect options, does NOT mean they actually work as advertised!) Eventually, I pull it all together and hatch a plan, figure what it will cost, and present it to the client. If they can afford it and like it, we are off to the races. If they don’t, then we tweak it accordingly. Maybe they can afford to take a longer outage and save on installation costs.

Maybe they are willing to pay more to bring the plant on line faster. Maybe there is a way to jury rig things to get the bulk of the benefit now and do the rest of the cutover later when they have more time. Well, that’s a start… let’s see what questions this generates.

Marsha Wisely’s Subsequent Questions

Thanks for the information! When you have only seen one of something, it’s hard to know what the norm is or what the other options are. Also, I really like your universal constants section! After reading your comments, I started thinking about how the information applied to my experience. Below I have a summary of my experience to give you some background on me, and it inspired some questions as well.

• Type of project: brownfield/retrofit
• My role was strictly software-oriented
• Moving from a combination of a programmable logic controller (PLC) and legacy system (Provox) to a modern DCS platform (DeltaV)
• The idea was for it to be a software-only upgrade, temporarily
  • Profibus DP served as the interface between the PLC and modern DCS platform
  • A specialized interface (DeltaV Controller Interface for Provox I/O) was used to communicate with the legacy system’s I/O
• To your point on the interconnect options: both solutions above were tested prior to startup, which saved us some headaches I’m sure
• The long-term goal was a full conversion to electronic marshalling
• The customer also had some existing control modules on the modern DCS platform they wanted to integrate once we were onsite.

The cutover and startup both took a good deal longer than previously anticipated. Loop checkouts were taking longer than anticipated, the plant discussed how to improve, but management never prioritized timeliness over safety and quality of work, which I respect and appreciate (I know that isn’t always the case). There was some unexpected equipment issues, namely valves were backwards (fail open when they should be fail closed and vice versa). Startup was also slower due to some equipment issues – air lines to pneumatic valves got leaks and equipment that was working before startup needed to be repaired before being brought back up.

This leads to some frustrations and finger pointing at the software – “The valves aren’t working; they worked before. It must be the software.” Definitely a good lesson in patience. In those cases, we basically did an additional loop check, which quickly found the issue.

Shutdowns are in the hands of operations and can be pretty hard on equipment. This leads to the following additional questions.

1) If you do a hardware and software cutover, are those types of equipment issues more likely to get caught? And are these types of equipment configuration errors common?  

2) Are there any common issues caused by shutting down equipment that are good to check before starting back up?

3) In your notes, you mention asking the customer about their “pain points.” Could you provide some examples?

4) My next couple projects are for new plants – one of which I will be leading. Do you have any advice specific to new plants? It looks like your “HOW to cutover section” covers a lot, but are there any additional challenges associated with new plants?

Thanks again for your help! I greatly appreciate your insight into the field since I haven’t had much field experience. My next couple projects, in addition to being new plants, also have a larger equipment and instrumentation component to my assignment, and I am looking forward to that exposure.

Hunter Vegas’ subsequent answers

Let me try to answer your questions:

1) If you do a hardware and software cutover, are those types of equipment issues more likely to get caught? And are these types of equipment configuration errors common?

If you are doing a hardware/software cutover you have to be fanatical about the details. Picking up failure modes of valves, ranges, square root versus linear, interlocks, etc. is pretty much a given. Only our most experienced people generally do the decoding of the existing system because it is so crucial to success. If you toss it to some inexperienced engineers, you will pay a wicked price on startup and rework.

2) Are there any common issues caused by shutting down equipment that are good to check before starting back up?

One thing I learned very early on was, if it all possible, obtain historical data of the running plant before you shut down and have it available for access during the start up. It is a very common trick for plant personnel to get the automation company to fix instrumentation that hasn’t worked for years. If you can point to the fact that the transmitter has been flat-lined since 2006, it is a pretty easy argument to say that fixing that transmitter is out of scope.

3) In your notes, you mention asking the customer about their “pain points.” Could you provide some examples?

Pain points are what keeps them up at night. Are they struggling with quality? Production? Throughput? Reliability? Does something break and it takes the techs 2 days to figure it out? Is there a particular area in the plant that is always in manual because the controls never work? Is the messaging awful so the batch stops and nobody knows why? Etc. Often I can fix a lot of those things fairly easy and make them very happy. Or I can offer solutions that increase the work scope some but has very big payback.

4) My next couple projects are for new plants – one of which I will be leading. Do you have any advice specific to new plants? It looks like your “HOW to cutover section” covers a lot, but are there any additional challenges associated with new plants?

New plants can be very painful and difficult for a whole new set of reasons. Specifically: a) How good is the engineering contractor? The large Architectural & Engineering (A&E) firms can be awful or wonderful; it all depends on what team you get. If you get the “A” team, things will be in pretty good shape. Unfortunately, they might bait you with the “A” team but swap you the “C” team later in the project or you end up with the “C” team due to turnover.

Either way, the engineering is just wrong. Wrong pipe, wrong materials, wrong instruments, wrong drawings, wrong…wrong…wrong. And it’s your job to make it work using BIFF principles (Big Improvements for Free) because the money has already been spent. b) Was anything reviewed by someone other than the engineering contractor? The best option is for the plant personnel to review and approve everything; however, the plant often lacks either the expertise or the time to do that and the engineering contractors are infamous for dropping 1,000 pages on your desk and demanding you approve it overnight or “the project will be delayed and it was your fault.”

The second-best option is for a third party to at least look things over and catch the worst stuff. If nobody looks it over, then you better have the “A” team or it will be bad. c) Does anyone even know how this is supposed to work? Is the plant a copy of an existing plant and the process is well understood (and ideally you have people from that plant helping you) OR is this Serial #1 and the only people who have a vague clue are some lab chemists who had a tiny pilot reactor running somewhere? Obviously, Serial #1 is going to be tricky because nobody knows what they don’t know and nobody has the answer. d) What is the schedule/budget like? Was it stupid aggressive to start?

If so, it will be tough, as nothing ever goes to plan on a new plant and both the budget and schedule are bound to suffer. I have had no-bid projects that I knew were badly estimated because failure was virtually assured, and I was much happier having my competitor get the black eye than me! Now don’t get me wrong – large projects are executed successfully all the time and with the right team things can go very smoothly.

But if the client goes with an unproven, low-bid entity, they often regret the decision. I once was part of a large project that was more than half-way completed but things were going so badly that the client fired the A&E firm mid project and went with a new one. They lost $10 million in engineering but probably saved $20 to $30 million in extra startup costs because the first firm was doing so poorly.

Additional Mentor Program Resources

See the ISA book 101 Tips for a Successful Automation Career that grew out of this Mentor Program to gain concise and practical advice. See the InTech magazine feature article Enabling new automation engineers for candid comments from some of the original program participants. See the Control Talk column How to effectively get engineering knowledge with the ISA Mentor Program protégée Keneisha Williams on the challenges faced by young engineers today, and the column How to succeed at career and project migration with protégé Bill Thomas on how to make the most out of yourself and your project. Providing discussion and answers besides Greg McMillan and co-founder of the program Hunter Vegas (project engineering manager at Wunderlich-Malec) are resources Mark Darby (principal consultant at CMiD Solutions), Brian Hrankowsky (consultant engineer at a major pharmaceutical company), Michel Ruel (executive director, engineering practice at BBA Inc.), Leah Ruder (director of global project engineering at the Midwest Engineering Center of Emerson Automation Solutions), Nick Sands (ISA Fellow and Manufacturing Technology Fellow at DuPont), Bart Propst (process control leader for the Ascend Performance Materials Chocolate Bayou plant), Angela Valdes (automation manager of the Toronto office for SNC-Lavalin), and Daniel Warren (senior instrumentation/electrical specialist at D.M.W. Instrumentation Consulting Services, Ltd.).

About the Author
Gregory K. McMillan, CAP, is a retired Senior Fellow from Solutia/Monsanto where he worked in engineering technology on process control improvement. Greg was also an affiliate professor for Washington University in Saint Louis. Greg is an ISA Fellow and received the ISA Kermit Fischer Environmental Award for pH control in 1991, the Control magazine Engineer of the Year award for the process industry in 1994, was inducted into the Control magazine Process Automation Hall of Fame in 2001, was honored by InTech magazine in 2003 as one of the most influential innovators in automation, and received the ISA Life Achievement Award in 2010. Greg is the author of numerous books on process control, including Advances in Reactor Measurement and Control and Essentials of Modern Measurements and Final Elements in the Process Industry. Greg has been the monthly "Control Talk" columnist for Control magazine since 2002. Presently, Greg is a part time modeling and control consultant in Technology for Process Simulation for Emerson Automation Solutions specializing in the use of the virtual plant for exploring new opportunities. He spends most of his time writing, teaching and leading the ISA Mentor Program he founded in 2011.

Connect with Greg: