The following discussion is part of an occasional series showcasing the ISA Mentor Program, authored by Greg McMillan, industry consultant, author of numerous process control books, 2010 ISA Life Achievement Award recipient, and retired Senior Fellow from Solutia, Inc. (now Eastman Chemical). Greg will be posting questions and responses from the ISA Mentor Program, with contributions from program participants.

Mike Laspisa has spent 37-plus years working in the instrumentation and control (I&C) discipline, including 32 years as a lead I&C engineer or manufacturing plant staff I&C engineer. Mike’s primary motivation is to advance the automation profession by sharing knowledge gained from plant experience as seen in Control Talk columns, “Instrument specification: Where we are and where we should be,” “I&C Construction scope,” and “Instrument Index Insights.”

Mike Laspisa’s Question:

What are some of the problems and solutions for dealing with the many issues involved in packaged equipment?

Mike Laspisa’s Answer

This area has historically been a source of many problems. The equipment packages are typically owned by the Process Mechanical group. I/E discipline input is requested in various forms (either attachment or as a stand-alone guideline spec.) during the Equipment Package specification preparation phase. However, their input is rarely requested during the bid technical analysis and vendor recommendation stages. Vendor drawing approval is more often than not the first chance to see what was actually purchased as part of the “package.” Verification of specification requirement compliance and project device tagging structures is usually difficult to do at this stage.

Packaged System initial design and drawing review topics/questions that can impact both engineering and construction on most projects: 

1. Vendor information does not come in a timely manner. It can take weeks for receipt, review, and re-submittal under the best circumstances. Sometimes, the I/E comments are not considered significant in the early stages of an order, or the Vendor will not start the I/E design until manufacturing is pretty far along.
2. Does the package instrumentation, control devices, and control system (including the interface with the plant control system) meet the equipment specification expectations/requirements (i.e. Manufacturer or equal, type or series, control voltage level, and control system: hard IO vs. Data communication interface, fusing/segregation/spares, etc.)? There have been many times where the plant standards, or even preferred vendors, were ignored to save money by accepting OEM devices well after the specification phase.
3. Sometimes the package PLC is still looking for a hardwired permissive/interlock even though Ethernet data passes were all that was intended. Sometimes the standard “Ready,” “Run,” and “Alarm” hardwired I/O status signal requirements are not provided by some simple package designs (i.e. Air compressor, RO Water, etc.). The tag structure for data passes to a DCS needs to be identified and standardized, especially if multiple packaged skids are involved.
4. Is either processor, I/O, power supply, and/or communication redundancy required? Has it been provided with the package?
5. I/O Voltage mismatches and isolated vs. grouped point designs are common issues that need to be resolved before the Contractor is landing wires. Changes may be required on either end depending on the timing of the information flow.
6. What is being furnished loose for Contractor installation? What field wiring is required? What wire/cable/support needs to be provided by the Contractor? This impacts the Contractor Scopes of Work, location plans, and cable schedules.
7. Is the Plant Control System specified interface with this package accurate or does it need to be changed? The communication interface methods have improved over the years beyond hard wired I/O. Both DCS configuration and PLC programming software now include recommended tags and rungs/routines for passing information and monitoring the integrity of the data pass.

Once identified, these issues can be difficult to get resolved. Multiple parties (Vendor, Engineer, Client) usually need to be involved because of the potential impact on project cost and schedule.

Luis Navas’s Answer

Sometimes the end user or engineering, procurement, and construction (EPC) company does not specify in a proper way the requirements to integrate in a “seamless” manner the skid package to the rest of the plant. Therefore, the package vendor performs the job as usual. Other times, there are good requirements, however the package vendor does not fulfill them at 100%. Most of the time, things not done previously are resolved during a very exciting commissioning stage.

I have had several experiences related to lack of naming convention, inefficient communication mapping, or even worse: Seam solutions. Most of the time processor, I/O, power supply, and/or communication redundancy is not specified and not even considered. Sometimes there are grays in the contract that result in no clear boundaries and no clear scope.

Greg McMillan’s Answer

Hunter and I discussed our experiences and provided key alerts and extensive guidance in the Control Talk column, “8 rules to avoid skidding accidents.” The packaged equipment supplier goals are often at odds with the practitioner (project E&I professional and end user). The primary goal of winning the bid results in the use of cheaper measurements and valves, local controllers, and pressure and temperature gauges and switches rather than ensuring the most accurate and reliable instrumentation that provide signals to and from the control room, enabling complete control and monitoring.

Also, the need to make the package compact often leads to insufficient straight runs for flowmeters and distances for mixing of streams for electrodes and thermowells. The packaged equipment supplier expertise and understanding of the implications in terms of performance and maintenance is often missing. The practitioner needs to be proactive in essentially designing the automation system including the installation details. The practitioner may need to be able to justify the increase in packaged equipment price and delivery date to plant management (no easy job).

Hunter Vegas’s Answer

As you mentioned, you and I provided a lot of good suggestions for dealing with this issue on the “skidding” article in Control Talk.

 To piggyback on some of Mike’s comments, here are some potential major software issues: 

• The Vendor provides wrong data or wrong data addresses. When the unit arrives on site, communications do not work or transferred data is incorrect or garbled.
• The Vender fails to “pack” the data and just provides a wide range of disparate, non-sequential data addresses. Some plant control system communication methods limit the number of data sets per port and those data sets may be 100 items each. If the required data is spread across a wide range of addresses, it may not be possible to obtain the data you need (Particularly true with Modbus TCP and/or Modbus Serial).
• It is important to “coordinate” the skid controls with the plant controls. Here are some examples of problems that need to be considered:
  • If the plant controller wishes to start a pump, does it set a run bit, set a start bit and the skid PLC resets it, does it pulse the start bit true for XX seconds and then reset it? If communications are lost, does the plant system need to read up all of the command bits to see their state or just re-write all of the bits as it was before?
  • How are a loss of communications detected?
  • What happens (on either side) should a loss of communications occur? (Skid shut down, skid keeps running with last values, etc.)
  • How are unintentional “zero writes” avoided? (Scan is lost and a zero gets written down for a setpoint momentarily?)
  • How are communications limited to keep a communication configuration error from overwriting the wrong areas in the controllers?
  • Working out data structures (signed integer, unsigned integer, real, double real, etc.) so the data is correctly interpreted.
  • If the skid has a local HMI, how do you determine if the local HMI or the plant control system is in control?
  • If both the local HMI and plant control system is in control, the HMI commands must be carefully coordinated to avoid competing/conflicting commands from the two systems.

When all is said and done, I would strongly suggest that all of the communications be thoroughly tested at the manufacture’s site before the skid is shipped. Bring a spare plant controller, set up the communication network, and prove the data transfer is correct and all of the commands work as expected. Do not let the vendor ship the unit until it has passed these tests.

Daniel Warren’s Answer

Being that I have had the fun of working at a number of sites that are “skid” built, there is no end to the possibilities one can run into. At this point, I see that a number of concerns about communication, tags, data sets and programming capabilities have to a good degree been captured.

However, I have a number of pet peeves with skid builders, especially when a detail set of specifications, standards, and other associated documents detailing the clients preferred direction are all but ignored.

One of the major concerns I have run into has been that the skid builders try going for a cheaper alternative or solution. Well, nothing worse than having devices mounted on the skid you are purchasing that don’t follow your client’s preferences. And unless it says that alternatives are acceptable, trying to get those items corrected usually ends up as an additional cost. Be it transmitters, valves, cabling, sensors, etc.

To add to this, even if you have to accept what you are given, sometimes getting a clear specification on what was installed may not happen.

I have also run into issues of material compatibility. I have seen it done on many applications (including code issues) that the skid builder substitutes with what they call their “floor standard,” having totally disregarded the client’s specification for said application.

Then there is the “practice;” that is, the how or why did they install that device in that location. Too often I have seen devices, especially instrumentation, shoehorned in such a manner that to extract the device (if maintenance was required) would be with a blowtorch, pry bars and a couple of handy gorillas to get said device out.

And the list goes on.

So, if the E/I team is not part FAT or any other skid selection criteria, but there will be a requirement for E/I devices to be part of the skids, then I place several requests before the construction acceptance and commissioning team, especially if they don’t want any hiccups during commissioning.

What I have asked for at some sites is that a receiving and inspection area be setup to perform a QA and deficiency list prior to accepting the skid. This gets sent to management to resolve. Sometimes the skid builder has to correct the found discrepancies. At other teams, we have had to accept management’s decision and work with what we get stuck with. I just make sure to document everything.

Here is a list of things I usually go through when it comes to “skid” provided equipment.

PRE List:

• What is this skid being provided for—its purpose?
• Is the client providing a specification, and does it include acceptable materials and components?
• Was the fabricator issued a set of the clients’ standards, even a guideline?
• Will there be a FAT or any form of QA for the skid?
• Was an inspection done upon receipt to check for any damages?
• Has a deficiency list already been started? (If not, start one)
• Do the instrument and electrical devices meet the requirements (spec.)?
• Has documentation been supplied along with the equipment (i.e., data sheets, wiring diagrams, maintenance manuals, spare parts lists, setup and programming information, etc.)

RECEIVING List:

• Do I have a list of the devices and their specifications?
• Does the documentation and the device match?
• Has it been installed correctly? (Per standards, code, and mfg. recommendations?)
• Do I have maintenance access for future concerns?
• Were the instruments tied to any process or impulses lines and are they still installed correctly?
• Are the JB’s and tubing manifolds installed correctly for signal, power, and air tie-ins?
• Is there sufficient support for the devices, cabling, and tubing?

PRE-COM list for turnover and buy:

• Does the device power up properly?
• Does it communicate with the DCS/PLC system properly?
• Is the device calibrated (some devices may come pre-calibrated)? (To be determined prior)
• Do the devices appear to function correctly?
• Does the valve properly stroke?
• Does the motor have the correct rotation?
• Does it meet the control narrative requirements? (This may involve other loops or devices and interlocks)

As you can see, when it comes to a skid, does the device communicate? That is only one of the issues to contend with. And yes, I have run into issues with scaling factors, range ability, interlocks, setpoints, voltages, etc. Just makes it part of the additional challenge that we get to work with.

Jose David De Sousa Drumond’s Answer

Quite a lot has been covered already by Mike´s, Greg´s, Hunter´s, and Daniel´s answers.

What I will add to complement the picture will come from the end-user perspective:

• Make sure that any Packaged Equipment´s local control system does not follow a “Black-Box” approach and that it can be fully integrated into the existing Plant systems.
• Regardless of whether or not preferred equipment and OEM´s have been used, the local packaged equipment control system shall be designed so that the totality of the control algorithms/loops can be functionally integrated into the main Plant or main Facility DCS.
• For example, all of the tuning parameters, Set-points, selectors, etc., inside each of the functional blocks (PID or otherwise) performing local control strategies are correctly mapped.
• Unfortunately, the above is frequently ignored and this ends up hindering the future ability to deploy Advanced Process Control, Real Time Optimization, and Plantwide Control Strategies.
• Not doing the right mapping and integration from the beginning will increase the time and the costs required for future APC and RTO initiatives.
• Having this clearly stated in your internal technical regulations and standards, as well as that in the project design basis helps in ensuring better integration.
• All of the above will help as well with future overall Operational Excellence, Proactive Maintenance, and Digitalization initiatives.
• A supporting motivator is that at the end this contributes to lower the Total Costs of Ownership.

As it is always the case, these skids are not made of products from a single OEM; you shall define a detailed list of what your Packaged Equipment vendor/contractor shall provide as part of the Packaged-Equipment´s “Asset Register.” The idea is that the register extends to not only “hardware,” but also electronic assets and attributes of each piece of configurable device included in the Scope of Supply of the Skid under consideration.

This should include at least:

• Maker, model, and Firmware version of any configurable device (transmitters, positioners, logic solvers, dedicated electronic controllers, comms. and networking devices, etc.).
• The above will allow you and your team to verify with each OEMs whether there is any advisory (safety, security, end-of-life, etc.) issued that applies to what is going to be shipped to your site. It is always faster and cheaper to replace, upgrade, or update at the vendor/contractor site.
• This can be also extended to the software assets that come with the skid, if it is the case that the Packaged Equipment comes equipped with local HMIs Panels and/or any other Industrial IT/OT equipment, in which case also the register should include the version of OS, runtime utilities, etc., for the same purpose of verifying active advisories from OEMs.

Pat Dixon’s Answer

In addition to comments by others, the problems I have encountered have to do with maintaining these skids. Often, the PLC on the packaged equipment is a different brand/model than the standard adopted by the facility. This can be addressed by specifying acceptable PLCs, but sometimes a vendor may not be able to accommodate this. From the perspective of the vendor, it is challenging to take logic they have developed in a PLC and port it to a different PLC. They would like to standardize to reduce their cost and the complexity of projects.

Today we have OPC-UA and MQTT that make data access to these PLC more standardized. However, if someone needs to look at the logic inside a PLC, you need the engineering tool that supports this. Rarely will anyone change the logic other than the vendors of the equipment because of warranty issues. However, you might need to look at the logic to see what is locking you out.

For example, on a recent project a facility had an old thermal oxidizer that must be running in order to make production. The PLC on this thermal oxidizer was an old model from a brand that differed from the rest of the facility. When the facility could not start the thermal oxidizer, there was no way to look at the logic to figure out what was wrong.

Nick Sand’s Answer

Some of the issues with packaged systems are the inconsistent HMI and alarm system. These are often overlooked until the system is installed. Surprise!

The HMI is often a panel screen, with different symbology and organization, and sometimes a different language. To overcome this, plan for extra training time and keep procedures with screen shots local to the system. Of course, a secondary view can usually be created in the primary control system. There can still be confusion when the two views do not match.

The packaged system alarm functionality is often limited, like a single alarm priority for example. Sometimes only a common alarm is available, like on many heat-trace panels where it stays in alarm.

Ideally the packaged system could support key aspects of the HMI philosophy per ANSI/ISA-101.01 and alarm philosophy per ANSI/ISA-18.2. There is additional guidance in ISA-TR-18.2.7 -2017 Alarm Management When Utilizing Packaged Systems.