Learning Outcomes

• Explain the scenario that often occurs during the construction and commissioning of large hydrocarbon processing facilities, where critical plant subsystems such as utilities and safety systems will be handed over to operations teams long before the rest of the plant is commissioned.
• Operators face a big challenge in monitoring and controlling these early subsystems due to high nuisance alarms generated by non-commissioned equipment and ongoing commissioning activities.
• This article discusses three alarm management techniques that can be implemented in distributed control systems (DCS) to suppress alarms from non-commissioned systems at operator workstations (OWS).
• Implementing these techniques will ensure situational awareness and consequently reduce spillage, machinery damage, personal injury and other incidents caused by operator failure to respond.
  

Introduction

Commissioning campaigns for new hydrocarbon facilities can be complex and demanding. Large, complex systems must be integrated and tested while maintaining worker safety and operator situational awareness.

The commissioning sequence is driven by a schedule that prioritizes the early integration of utility and safety systems.

Utility systems such as instrument air, hydraulic power units (HPU), power distribution boards and power generation must be live before other systems can be brought online.

In addition, fire detection systems must be operational in equipment rooms and areas where rotating equipment is located.

During the commissioning campaign, there will be a drive to complete these critical systems first and hand them over to operators. Operators will then be responsible for monitoring the health of utility systems, operating these systems and monitoring the fire systems.

However, commissioned plant units and the activities of the commissioning teams generate a tremendous volume of alarms, which can severely compromise operator's ability to monitor the live systems and equipment.

Operator failure to respond can result in equipment damage, spillages and other hazard events. This is compounded by the increased likelihood of distraction due to ongoing work during the commissioning campaign.

To address this issue, alarm management techniques can be executed during the commissioning phase to suppress alarms of non-commissioned equipment from operator stations.

These techniques are adapted continuously in response to the progress of the commissioning execution. When new systems are handed over, the associated alarms will become available simultaneously.

This strategy aims to achieve good alarm performance in accordance with the recommendations specified by ISA-18.2/IEC 62682 and maximize operator situational awareness in a challenging environment.

Commissioning Handover to Operations

Once the engineering design has reached a mature enough stage with all process units well-defined and tagged, the commissioning authority will start to divide the plant into subsystems, normally based on functionality or specific plant units.

For each subsystem, the commissioning authority will compile a comprehensive dossier of all check sheets and tests to validate each system's function before it can be handed over to the end user.

Construction and commissioning are planned in a carefully sequenced manner, considering the interdependencies between each subsystem and the constraints of both workforce and material availability.

Subsystems will enter the commissioning workflow after first achieving a state of mechanical completion, which indicates the handover from construction to commissioning.

These first subsystems in the commissioning schedule will be needed as a prerequisite for energizing industrial control system (ICS) panels and other critical equipment, which cannot be powered up without coverage of the fire system. 

• Power distribution and PMS: Needed to energize other systems and ICS
• Main ICS (DCS/SCADA): Needed for instrument loop checking
• HVAC: Needed to keep ICS servers and other panels cooled
• Fire and smoke detection in panel roots: At least smoke detection will be needed to monitor ICS panel rooms
• Inst/Air or HPU: Needed for commissioning activities
• Others such as water makers/sewage may be needed for remote/offshore locations to provide a habitation for workers

Once these utility and safety systems are commissioned to an agreed state of readiness, they will be handed over to an operations team.

The handover to operations following commissioning is crucial, as the subsystem integrity and health must be continuously monitored to ensure they remain available to support dependent subsystems.

Whether or not the handover to operations constitutes a contractual milestone or transfer of ownership is irrelevant; some entity must be responsible for operating the systems.

Commissioning Schedule

Nuisance Alarms Due to Un-Commissioned Equipment

 The problem for operators who take ownership of these early sub-systems is that 80% of the plant will not be commissioned while the ICS is fully energized.

 At this stage, it may be reasonable to expect thousands of stale alarms to appear on the alarm list. In addition, the number of chattering alarms may be above 100 per hour.

The possibility of nuisance alarms during commissioning may be due to several factors, including but not limited to:

• Field loops that are open circuit as the loop is not commissioned
• Ongoing loop testing by instrument commissioning teams causing chattering alarms
• Package PLCs will be powered down, causing communication alarms 
• Process alarms due to plant shutdown conditions, such as empty tanks causing low-level alarms
• Spurious alarms caused by channel faults or loop ground faults
• Diagnostics alarms from the ICS if the full SAT/SIT has not been completed — channel alarms, power supply failures
• Low-pressure alarms from HVAC pressurized rooms due to excessive traffic through doors
• Temporary arrangements in power distribution during the commissioning phase causing alarms from the power management system
• Insufficient dynamic alarm suppression (missed during the engineering phase)
• Indication status signals which have been incorrectly assigned as alarms during the engineering phase, such as drive status or feedback from package, which are not true alarms as per IEC 62682 definition
• Missing definition files for HART, EtherNet/IP devices casing communication failure alarms
• Alarm states that have been incorrectly configured with inverts and not discovered during FAT
• Insufficient alarm hysteresis in ICS modules is causing bad actors to chatter
• Incorrect analog alarm setpoint and default alarm settings in ICS are left enabled

This list of possibilities goes on. Under these conditions, the operator will quickly become exhausted from attempting to continually filter and parse the enormous stream of alarms on the ICS.

Unfortunately, with so many alarms, operators will undoubtedly be forced to mute the ICS alarm sounder, increasing the potential for missing important alarms related to live equipment.

Operator overloading due to the high volume of nuisance alarms will eventually result in accidents, such as equipment damage or spillage.

Defining Nuisance Alarms

Nuisance alarms are defined as those alarms that satisfy any of the following conditions:

• Do not indicate a true malfunction or fault
• There is no operator action available to clear them

In the context of a newly established production facility that is transitioning between commissioning and operation phases, nuisance alarms could be considered as:

• Any alarm emitted from equipment that is not live and handed over to operations
• Equipment that is handed over but with residual faults (exceptions captured under a punch)
• Missing instruments and equipment on live systems captured under a material punch
• Process alarms in vessels due to abnormal operating cases (not in production)

Different Types of Nuisance Alarms

ISA-18.2 and IEC 62682 ICS standards provide recommended performance metrics for nuisance alarms.

End users and facility owners will normally maintain their own requirements for alarm system performance and may impose these on engineering contractors.

Some end users will specify less stringent performance during the commissioning phase, with the caveat that the contractor is actively taking measures to improve.

Alarm system performance generally becomes more important and receives more attention as the project approaches the final handover and startup of the process.

End users will be reluctant to take ownership of a facility from a contractor if there is a fundamental issue with the alarm performance preventing them from safely operating the plant.

Challenges for Alarm Management During Commissioning

Unfortunately, directing time and resources toward improving alarm performance during a busy commissioning campaign can be challenging.

The relentless drive to push systems through the project milestones and achieve a healthy rundown of commissioning activities will generally take precedence, and it may be difficult to convince management to absorb any resources or time in alarm management.

• Alarm management not being a priority — commissioning managers may often say, “We are in commissioning, so you have to expect some alarms.” Unfortunately, they are only interested in getting the commissioning done and will be unlikely to divert resources to something that is not directly accelerating the commissioning campaign. 
  
• Inter-disciplinary coordination — Reviewing, investigating and clearing alarms on a large plant will require input and coordination from many other disciplines. For example, it may be necessary to converse with the process team to understand why a low pump discharge pressure alarm is constantly running below the setpoint and clarify if a setpoint change is needed.

Coordinating between discipline and taking time from individuals already focused on commissioning and testing can be challenging. It may be difficult to get engagement if you do not have sufficient authority. 

Adaptive Alarm Management Strategies

Despite the challenges as described above, it is possible to very quickly make a significant improvement to the alarm during the commissioning phase.

The techniques are explained using functions within the Emerson DeltaV system but should be replicable in any modern ICS with an ISA-88 compliant database structure, where the application software is structured using areas and modules.

The alarm management strategies described here have been used on several hydrocarbon projects with excellent results in delivering compliant alarm system performance during an environment with simultaneous operations and commissioning.

The strategy involves three approaches, each with increasing levels of indenture, starting with plant area-wide suppression, followed by module-level suppression and finally, individual alarm shelving.

These strategies should be actively deployed to achieve a perfect alignment of alarm suppression with the current state of plant handover so that operators receive only alarms from handed-over systems.

• Plant Area Suppression: Suppress entire plant areas by unassigning the plant areas from OWS alarms and events.

• Module Area Suppression: Use to suppress individual modules by relocating to a temporary alarm area that is not assigned to OWS.

• Alarm Shelving: Use to address any residual nuisance alarms that emerge.

Alarm Management During Commissioning

Plant Area Suppression

Emerson DeltaV uses buckets called “plant areas.” These are logical, software-based divisions of the control system that normally correspond to physical areas of the plant. Plant areas contain the modules that form the application logic used to control the plant. The configuration of areas and modules is viewed within the DeltaV Explorer application.

Areas and modules are part of the ISA-88 process model pattern, which defines a hierarchical classification, including areas, process cells and units. Emerson DeltaV is developed around the ISA-88 process model and DCS products from other major automation suppliers.

Plant areas are assigned to a workstation node's “alarms and events” subsystem. Any module or device that has alarms configured will report these alarms to any workstation for which the associated area is assigned. 

Plant area suppression involves considering exactly which areas the operator needs to monitor for the current phase of the project and adapting the current area assignment accordingly. Note that by default, in DeltaV, a new workstation configured will have all areas assigned.

This function makes it possible to suppress alarms from certain areas for specific workstations selectively. 

It is good practice to make a matrix-type list indicating all areas and workstations and then identify the ones intended to be unassigned or assigned. This list can be developed with the input and coordination of the commissioning authority to establish the current commissioning status and timeline for handing over systems.

Over the course of the commissioning campaign, regular reviews are advisable to ensure the current area assignment configuration is tracking the actual state of completion.

If sub-systems become ready for use, then the associated area and alarms should be assigned to the operator's workstation so operators have visibility of any associated alarms.

Here is an example of a basic matrix that indicates the current assignment of areas for operator workstations (OWS) and engineering workstations (EWS) used by commissioning personnel. 

A table such as this can be used to track and record the current configuration throughout the commissioning campaign; helping to align and optimize OWS area assignment with the actual state of plant handover.

It should be recognized that such changes to the ICS application logic and will need to be controlled by careful management of change (MOC).

Module Level Suppression

Module-level suppression refers to the practice of transferring modules to temporary areas that are not assigned to any operator workstation. This will suppress all alarms from the modules contained within the temporary area.

This is a good strategy to implement in the following cases:

• The DeltaV plant area covers multiple commissioning sub-systems

• If a subsystem in handed over, but there are residual faults or missing equipment that have been accepted as exceptions
  
  
• In cases where it is acceptable to suppress the entire module and all associated alarms

This method makes good use of modern DCS's modular nature, where modules can be easily dragged or bulk-edited to other areas.  

It is advisable to create a single temp area alias for each area using a suffix to avoid confusion when returning modules to their original area. As an example, the temporary area for Inst_Air would be Inst_Air_temp.

It is also advisable to record any associated punch number or applicable reference within the DeltaV version control for good traceability.

The effectiveness of this strategy will also depend heavily on the ability to execute good tracking, recording and management of change (MOC).

If the fault of the associated equipment is resolved, then there must be a mechanism to trigger the DCS service engineer to transfer the module back to its original area.

One way to achieve this is to record a comment within the associated punch database that the “DCS engineer must re-instate equipment alarms before punch closure.”

Alarm Level Suppression 

After implementing alarm suppression at the area and module level, the plant's alarm performance metrics should approach nominal values, and operations stations should only report true alarms.

However, nuisance alarms may continue to emerge due to sensor malfunctions, process changes and severe weather conditions. Shelving is a good approach if corrective action can be taken by maintenance teams within a shift.

Shelving is a method of suppressing the alarm that can normally be done directly by operators from the HMI without any intervention from the ICS technician. Shelved alarms have a preconfigured time duration that, when elapsed, will cause the alarm to automatically un-suppress.

Some end users will permit the operator to define the timeout period, while other facilities will preconfigure it based on the type of alarm. In DeltaV, the shelved alarm lists is available via a dedicated alarm page and can be easily extracted to Excel.

In addition, DeltaV allows operators to record a reason for shelving the alarm via a preconfigured dropdown list; this function is also part of compliance with ISA-18.2/IEC 62682.

Operation teams should review shelved alarms weekly with the same level of oversight as overrides, bypasses and forces in the ICS.

Alarm shelving is often considered a desirable approach because operators can implement it without requiring ICS download and MOC workflow. If chattering alarms suddenly appear, alarm shelving can quickly reduce operator loading.

It's important that clear guidelines and workflows should be in place to help operators determine:

• Exactly which types of alarms are allowed to be shelved
• The maximum number of alarms that can be shelved at any given time
• Procedures for recording and managing shelved alarms

Operators should be trained to understand the correct situation when shelving is necessary, and shelved alarms should be reviewed during shift handover.

Conclusion 

During extended commissioning campaigns, where subsystems are brought online and handed over sequentially, the alarm management strategy must be continuously adapted to the current state of system handover.

The author challenges the commonly held excuse that high alarms are "normal" during commissioning. If plant operators are overloaded with nuisance alarms during commissioning, machinery damage, spillages and other incidents will result from the operator's failure to respond.

To avoid overloading operators with nuisance alarms, operator stations should report alarms only from units, equipment, and devices that have been formally handed over and are considered in use by the commissioning authority.

This is achieved through three progressive measures with an increasing level of indenture. The first measure involves aligning the alarm and event area assignment of operator workstations with the status of commissioning completion. Only plant areas of subsystems considered live and handed-over should be assigned to an operator's workstation's alarms and events.

The second strategy at the module level involves moving modules to temporary areas in the explorer database. Modules related to devices or equipment that do not need to be monitored can be relocated to temporary areas so that alarms generated by this equipment will not report to an operator's workstation.

The third strategy is the lowest level of indenture dealing with individual alarms. It involves the operator shelving nuisance alarms. This can be used as final tuning to deal with new nuisance alarms as they emerge.

Finally, change management and tracking are critical. The designated authority should ensure the current configuration of operator workstation area assignment and temporary areas are continuously tracked and reviewed.

• Plant areas must be assigned to operator stations immediately when the associated subsystem is handed over.
• Modules that have been relocated to temporary Areas must be reinstated to their original location and the temporary area deleted from DeltaV Explorer.
• Shelved alarm must be managed, reviewed and un-shelved when corrective action has cleared the alarm.

By implementing the alarm suppression strategies mentioned above, along with effective oversight and interdisciplinary coordination, commissioning teams can achieve excellent alarm system performance.

This can enhance operator awareness during extended commissioning campaigns, resulting in improved safety outcomes.

The ISA18 Standards Committee is actively working on the ISA-18 standards and seeking input from anyone interested in alarm management. To learn more, email ISA Standards at standards@isa.org.