This post was authored by Dan McGrath, P.E., associate director at Cognizant, and previously solutions manager for Panduit

The benefits of converging industrial and information networks with a validated secure architecture based on standard Internet Protocol (IP) technologies are well established. The benefits include greater connectivity and integration across plants, easier data sharing across the enterprise, and better visibility into real-time operations.

Bringing together information technology (IT) and operations technology (OT) networks is complex because of the gray area between IT and OT roles and responsibilities within the company. IT and OT professionals must have a common understanding of a host of techniques and technologies to overcome this complexity and establish a converged infrastructure that is secure and manageable by all critical stakeholders.

The techniques and technologies used in network design can be simplified by leveraging the ISA/IEC-62443 Zones and Conduits Model developed by the ISA99 committee. The three design areas are:

• cell or area zone
• production site operations
• enterprise zone integration

Gateway physical segmentation example: two NICs for network segmentation.

Designing for the cell or area zone

Several considerations must be made to ensure the network infrastructure addresses your data, security, and availability requirements at the cell or area network level. Machines and process skids are seeing high growth in number and criticality of IP connections. One of those considerations should be logical segmentation, which is the process of dividing end points into subnets and virtual local area networks (VLANs). A key recommendation for industrial networks is to create smaller layer 2 networks to improve performance with the maximum of 200 devices within a zone or VLAN.

VLAN spanning multiple switches

Segmentation allows for smaller layer 2 domains, which helps constrain broadcast and multicast traffic. It also helps manage the network’s real-time communication properties and supports the network’s traffic-flow requirements. With segmentation, manufacturers and industrial operators can meet their security requirements by limiting remote expert or original equipment manufacturer network access to only specified machines.

Organizations often accomplish physical segmentation within the cell/area zone network by using separate cabling and switches. This common approach in Ethernet networks can become a hindrance to network performance if not properly planned. For example, physically separating input/output (I/O) and human-machine interface traffic without connecting the I/O traffic to an interconnected layer 3 switch can limit overall connectivity and even cause delays. Networks should, at a minimum, be connected to a layer 2 or layer 3 switch, rather than a controller, to interconnect.

VLANs are a very effective way to execute segmentation: specifically, for segmenting different traffic types—industrial and nonindustrial—as well as creating smaller layer 2 networks. Establish VLANs in a one-to-one relationship with subnets to make routing easier and more straightforward. Devices on a single VLAN are typically assigned IP addresses from the same subnet, and they do not require a layer 3 switch or router to communicate among each other within the VLAN. Using a layer 3 switch or managed switches with layer 2/3 functionality allows communication between VLANs. A management VLAN should be established for management across multiple cell/area zones.

Additionally, using structured cabling for interswitch links and more critical runs in your cell/area zone is a best practice network design approach. Point-to-point cabling is the norm for connecting end-point devices in close proximity. However, more critical connections can benefit from using industry standards designed to ensure a testable, scalable infrastructure. A structured cabling approach has better organization and testability with patching fields, permanent links, and patch cords that are validated as a high-performance system for rising data rates and high availability. Structured cabling built on TIA-1005 or ENXXX standards have the bandwidth and noise immunity for challenging cell or area zone deployments to ensure uptime.

Designing for production site operations

The Internet of Things (IoT) has created an explosion of smart IP–enabled devices that were not traditionally connected to the network. This has created an opportunity to deploy a more flexible architecture with mobile access to data and connectivity within the production environment. Wireless network technology is one of the key enablers for realizing the value of IoT. Wireless technology offers new capabilities, such as “bring your own device” and wireless security cameras, to protect assets and lower installation and operational costs.

Wireless local area networks are significantly different from wired LANs, and they should be designed for your security, reliability, bandwidth, and throughput requirements. For example, Wi-Fi Protected Access 2 with Advanced Encryption Standard encryption is the only mechanism recommended for control and automation wireless applications, and it should be used in combination with other security methods, such as preshared key authentication.

Wireless channel packet rates should be limited to 2,200 packets per second to help avoid packet delays, and they should be reduced in areas that experience interference and radio-frequency issues. Also, avoid the more heavily used 2.4-GHz band for industrial control applications because you may encounter interference. The 5-GHz band provides dedicated bandwidth and less interference. Conducting a site survey will also help you identify and curtail other potential interference within a production environment.

Virtualization is another key network technology in the connected industrial enterprise. Virtualization breaks the previously unbreakable link between hardware and software, so you can keep industrial applications running beyond the life cycle of their hardware. Virtualization also abandons the traditional one-to-one approach, meaning multiple applications and operating systems can run from a single server.

Important considerations when designing a virtual environment include your applications’ RAM, CPU, and disk I/O requirements, as well as how many applications will be deployed in the virtual environment and the kind of network switching. Take into account your current needs, but also design for growth in the next five to 10 years.

You can engineer and build a virtualized system from scratch, but this approach can be costly and time consuming. You will need to purchase equipment from multiple vendors and spend weeks assembling, installing, and testing the system. An alternative approach is to purchase a bundled virtualization system with preassembled, tested, and validated hardware. Bundled systems also often come with support services, including on-site configuration and integration, which can shave days or even weeks off the deployment process compared to the build-from-scratch approach.

Keep in mind, however, that bundled virtualization systems are produced for a range of industries. It is important to select a system that is custom built for the demands of the industrial environment.

Enterprise integration

A truly converged IT and OT network architecture achieves seamless integration both horizontally, across multiple sites, and vertically, from the shop floor to the top floor. Doing this requires an understanding of the key techniques that bridge the gap between historically separate networks.

Using fully standard IP networks throughout the architecture, such as EtherNet/IP for automation communication, eases future application design. Only an IP-centric network infrastructure can help you better use the Internet of Things—which is the proliferation of connected industrial and commercial devices—because it is a unified digital communications fabric on which these IP-enabled devices can talk to each other. A fully standard IP network means your network design is more future proof and can deploy new services or applications without specialized gateways or communication stacks that create integration challenges.

Cloud computing also helps integrate the enterprise. Moving data centers to the cloud simplifies implementation and maintenance, reduces costs, and improves an organization’s agility. However, the cloud still requires a robust physical infrastructure. Factoring in important design considerations—such as deciding what kind of cloud you will use and identifying your bandwidth and cabling requirements—will help you make the most of your cloud investment.

Enterprise integration also cannot occur without considering security. Given the expansion of the network fabric, increased access to sensitive data, and the breadth of cybersecurity threats, designing in multiple layers of protection should be considered a security best practice. This “defense-in-depth” approach builds a system of safeguards for both digital and physical security, putting multiple security measures in place to counter each individual threat.

Digital security measures include device authentication according to the 802.1X standard, policies that ensure the most recent antivirus updates and security patches are installed, monitoring key network statistics and log files for signs of intrusion, configuring network switches to manage traffic and access, and tightly controlling software used for remote access.

Physical security includes pass codes and access cards to limit personnel access to rooms and connected machines. USB block outs help prevent unauthorized network access, the removal of sensitive information, and the threat of spreading a virus. Lock-in/block-out devices can prevent unwanted plugging and unplugging for copper and fiber connections.

Lastly, an industrial demilitarized zone can be a crucial barrier of protection between the enterprise and industrial zones. It serves as a choke point for all traffic between zones and can help better manage access, such as with authentication enforcement or by monitoring traffic for known threats.

About the Author
Dan McGrath, P.E., is associate director at Cognizant. He previously was solutions manager for Panduit, helps lead Industrial IP Advantage, which is a coalition launched by Panduit, Cisco, and Rockwell Automation promoting standard, unmodified Ethernet and Internet protocols for industrial applications. McGrath has more than 25 years of experience in automation and industrial networking for global manufacturing operations. He holds a B.S. degree in electrical engineering and has attained professional engineer and ASQ certified quality engineer certifications.

Connect with Dan

A version of this article also was published at InTech magazine.