While industrial software compatibility issues still exist today, OS (operating system) suppliers, system integrators, engineers and application software vendors have been using common tools and protocols that make it easier to minimize these issues. Nevertheless, at least seven areas where industrial software applications can face compatibility problems include (in no particular order): (1) OS issues, (2) hardware incompatibility, (3) dependency on legacy systems, (4) interoperability between/among software solutions, (5) version incompatibility, (6) security software conflicts and (7) updates/patches—whether application or OS.

When Legacy Hardware and Software Refuse To Die

No doubt about it, keeping legacy hardware operating on modern software systems presents compatibility issues. I know this firsthand. Of two film/photo scanners I own, the older one supports large format film, which I still occasionally scan. Since the scanner has no driver support later than Windows XP, I run the scanner via a Windows XP virtual machine (VM) within Windows 10. Thus, the scanner has a new lease on life—and I haven’t spent $700 or more to replace it.

My wanting to keep old hardware running is by no means unique to those in the manufacturing world. “Wholesale replacements of legacy systems are rare and expensive, and thus integrating with legacy systems remains an important consideration for adopting new technologies in a non-siloed way,” says Joseph Dolivo, CTO, 4IR Solutions, a Control System Integrators Association (CSIA) member. Legacy systems, however, present security risks, and it’s crucial that they are mitigated in a safe way without adversely affecting production or the safety of new assets that are deployed, adds Dolivo.

“As American manufacturing infrastructure continues to age and SMEs (subject matter experts) retire at an overwhelming pace, companies find themselves making the difficult decision to replace their legacy systems, rather than update them, due to the difficulties of guaranteeing compatibility and tracking available stable revisions across the gamut of their manufacturing operations,” says Bill Paczkowski, Emerson senior product manager for PACEdge.

Legacy systems also lead to issues with interoperability among software solutions, adds Paczkowski. “Although communications are a bit more democratized now with standard protocols, many legacy hardware and software solutions depend on proprietary software and hardware to operate together, making update control and management that much more difficult.”

Legacy systems may use outdated technologies, proprietary interfaces, or obsolete operating systems that are no longer supported by modern hardware or software, says Paige Minier, manager OT infrastructure, Gray Solutions, a CSIA member. “Integrating these legacy systems with newer technologies or upgrading them can be complex, time-consuming and expensive. Compatibility issues may arise due to differences in data formats, communication protocols or security vulnerabilities.”

As if connecting old and new systems weren’t hard enough, many manufacturers don’t even have a system for updating what they have. “Despite the significant advantages of updating legacy automation, an alarming two-thirds of organizations lack effective OT patch management procedures,” says Matthew Masarik, Control Software at Rockwell Automation business manager. “Legacy systems operate in isolation and are costly to run. However, the threat of obsolescence may seem less daunting than upgrading. While some executives may shy away from updating or patching legacy systems due to the risk of them breaking, the impact of a security breach or using outdated technologies can have far worse consequences that harm the business and its customers.”

Security Creates Additional Incompatibilities

Security and legacy systems often are at loggerheads. “Manufacturing environments are increasingly vulnerable to cybersecurity threats, and implementing robust security measures is crucial,” says Minier. “However, compatibility issues can arise when security software—such as antivirus programs, firewalls or intrusion detection systems—conflicts with other software applications or operating systems in the manufacturing environment. Moreover, applying updates and patches to ensure security often requires careful testing and validation to avoid disrupting critical manufacturing operations.”

Security software conflicts are an issue because, as many controls engineers and technicians are aware, most automation development environments will not properly function unless the local user is running it in an administrative capacity, says Paczkowski. “This heightened level of privilege goes against much of the work IT teams are trying to accomplish in protecting the companies’ assets. A few years ago, at another company, I remember having to get IT to shut down the security suite installed on my company-provided laptop each time I needed to edit a PLC program.”

“Security has become a critical reason for end users to spend time and money on their plant floors to reduce risks and upgrade the software and hardware much more often than in the past,” says Kyle Reissner, ICONICS director, product management engineering. “From a software vendor standpoint, this makes the ability to update the software via patches/upgrades with as minimal disruption as possible a key area of investment. Now, it’ll never be as easy as updating apps on your phone, but it can and must get easier given the increased security risks end users face.”

Thorny Compatibility Issues and Ways to Minimize Them

Seemingly impossible situations make compatibility issues thorny not only for food processors but also for system integrators and industrial automation software vendors. “Pulling data out of proprietary systems when the vendor is either unwilling or unable to assist (or no longer exists) is a challenge,” says 4IR Solutions’ Dolivo. “We’ve addressed this in a handful of ways, including reverse-engineering proprietary protocols, deploying other hardware-software systems alongside the proprietary systems to collect data in parallel or writing the systems out of scope completely.” Dolivo suggests that larger manufacturers with “sway” pressure their suppliers into supporting open protocols and providing access to the source code of the systems they purchase.

Interoperability between/among software solutions supplied by many different vendors and of varying vintages remains a challenge, says Emerson’s Paczkowski. “Manufacturing customers have an established infrastructure, which they usually need to preserve to the greatest extent possible due to the cost, complexity and risk of making significant changes. They are also limited based on resource availability, training and other challenges. Implementing new solutions requires seamless integration between the systems, preferably using solutions designed specifically for this purpose.” Custom integration is possible, but increases cost, complexity and risk.

Not only can compatibility issues exist among different makes of equipment and software, they can exist within the same supplier’s system—and as we already mentioned, keeping that equipment updated to the latest version is a real challenge.

“Version incompatibility within large, distributed solutions that rely on common core services seem to provide some of the bigger challenges,” says Bryan Wilson, manager, digital transformation, Gray Solutions. “These solutions tend to be a mix of SCADA along with historian and other information systems. In many cases, upgrades to portions of the solution require updates to other core services. This means the systems need to be taken offline, leading to more planned downtime during maintenance opportunities. When we come across these scenarios, we help customers evaluate whether to upgrade all components of the solution during the planned downtime. The goal is to minimize the frequency and duration of the planned downtime for the customer while still providing the benefits of software updates.”

“Compatibility issues that seem to be the thorniest to deal with are the dependencies on legacy software/hardware systems in outdated facilities,” adds Gray’s Minier. Some systems are so old that they no longer have any experienced site personnel in that system. “When we try to gather information, we often get the ‘It just runs’ answer with few specifics. This gap in system knowledge presents some significant problems to manufacturers, especially if those systems experience critical failure before information gathering and implementation of new solutions take place. The longer that legacy system upgrades are delayed, the more complex they become, which directly translates to the cost of those upgrades.”

Lifecycle Issues In Tools and Platforms Add To the Incompatibility Mix

From a product development and lifecycle perspective, the speed at which platform companies are introducing and deprecating toolkits/frameworks that are required to build programs, services and apps on operating systems is now at “cloud speed,” says ICONICS’ Reissner. “This creates compatibility and security issues when you don’t keep up. In some cases, they are moving from the previous ten-year lifecycles down to two years. For instance, Microsoft’s .NET Core is end of life (after being available for ten+ years) and the .NET 6 lifecycle is only two years. Now these time frames are OK for cloud-based products that are consistently being updated and have customers who are fine with a level of downtime for their services. However, in industrial projects, this just isn’t the reality.”

Systems are deployed for years, sometimes decades, and the expectation is that the products will be supported for those timeframes, adds Reissner. In addition, upgrades, especially on validated production systems (of which more food systems are becoming more pharma-level validated), need to be planned and re-validated—so keeping up with this is a challenge both from a software vendor and end user perspective. “Our best advice to ease solutions to these challenges is to take care to avoid vendor lock-in by deploying software that embraces the use of open industry standards,” says Reissner. This strategy helps to ensure the greatest level of compatibility and interoperability with other applications running on the plant floor—and even business systems such as ERP, MES, CMMS and other supply chain platforms.

End Users Try to Slow the Frenetic Pace of Technology

As software compatibility issues are becoming more and more of a problem for many end users, some have resorted to the use of multiple virtual machines, each with a different OS to run all the different software components necessary to run and manage the factory, says Paczkowski. The best way to get around this is to work closely with a company that chooses to use lean evergreen programming schemes such as web-based development environments utilizing universal communication drivers such as MQTT and OPC UA.

But even today’s “evergreen” open source and communications technologies could be tomorrow’s deprecated tools. Think back to serial and parallel buses, PATA hard drives, OS/2, deterministic MAP (Manufacturing Automation Protocol) networks, PASCAL, CoffeeScript, Micro Channel Architecture, protocol converters, and the list goes on.

Rockwell’s Masarik suggests a way to cope with managing compatibility issues, including communications protocols, security architectures, OS, etc. That is having a strategy in place. “When enterprises lack a thorough software compatibility strategy, they often find themselves being reactive to ‘keep things running’ rather than proactive and tackling these challenges before they arise,” says Masarik. This eventually creates a tangled web of temporary solutions which becomes untenable, jeopardizing profitability. What is more effective is a strategy that allows decisions to be made in regard to security posture, standardization and technical requirements.

What does this strategy look like? Masarik says, “Having goals like all systems are expected to be able to participate in our data layer and MES, or that all systems comply with a desired security architecture. Those milestones direct your decisions on compatibility and modernization. Without them, you’re left with a reaction focused posture which rarely positions operations for the future.”

“Microsoft offers LTSC (long-term servicing channel) releases of Windows, and there are various “de-bloating” tools and scripts that can certainly help streamline Windows deployments,” says 4IR Solutions’ Dolivo. “Wherever possible, however, we strongly encourage our end users and vendors to move to Linux. Leading vendors like Inductive Automation already do this, and others are working on it. Even Microsoft offers their own distribution of Linux for companies who want to remain in the Microsoft ecosystem.”

For those not familiar with Windows 10 LTSC, it is designed for Windows 10 devices and use cases where the key requirement is that functionality and features don’t change over time. For example, these include medical systems (such as those used for MRI and CAT scans), industrial process controllers and air traffic control devices; in many cases these OS applications are used in embedded systems.

“Most of our software vendors have teams dedicated to testing compatibility with Microsoft updates,” says Gray’s Wilson. “Prior to applying an update to the OS, we determine if that update has any known compatibility issues utilizing the testing and knowledge provided by the vendor. In other scenarios of highest criticality, we will build a test environment identical to the production environment and apply the updates there first to check for compatibility issues. With some environments running on older hardware and software where building a test environment is not possible, we mitigate update issues by first creating a solid rollback plan that typically includes full system images when possible. Creating a full backup of the system before updates are applied is a critical step.”

ICONICS’ Reissner suggests the best way to test OS and/or application updates is offline when scheduling is best suited for production and the engineering staff. “In my past, I’ve seen this situation for a furnace control system that was prone to issues stemming from various types of updates. This [testing] scenario included not only software updates but also sensor changes, PLC logic changes, network upgrades, alarming, process tweaks, etc. But the way the machine components were controlled, using this offline system always resulted in finding system issues we didn’t think of. Catching these issues and putting fixes in prior to propagating such changes to the live system saved a ton of grief and money.”

Some industrial control vendors have switched to Linux for industrial controllers. For example, Siemens offers the SIMATIC Industrial OS for its industrial process controllers. The OS is based on Debian’s Linux long-term support kernel. The OS can be trimmed down in size and supports real-time operation through the “PREEMPT RT patch.” Emerson has also gone the Linux route for some applications.

“Emerson’s PACEdge product exists in a dedicated Ubuntu Linux environment, shielding it from scary OS updates,” says Paczkowski. “Even if a local browser is affected by the latest OS update, the PACEdge web-based programming environment can also be accessed through the use of a monitor, keyboard and mouse directly connected to the edge device. Linux hardware is an increasingly dominant force on the plant floor because other operating systems continue to become bulkier, overburdened, and frankly, less secure.”

To Virtualize or Not: A Few Thoughts

While virtualization solved my scanner problem, is it a good idea in the plant? Well, it depends. “Isolating legacy OS-based systems is critically important, as they put not only themselves but also any newer systems that integrate with them at risk,” says Dolivo. “Virtualization is a good technique for doing this and ideally is a stop-gap measure until a more modern alternative is available. Custom integration, particularly where proprietary interfaces or protocols are involved, can be a costly endeavor, but so is replacing hardware.”

“On one hand, virtualization has dramatically expanded users’ ability to manage disparate environments including multiple OS generations,” says Rockwell’s Masarik. “On the other hand, it doesn’t change the limitations that those legacy systems impose on operations. So, we’re finding ways to accommodate outdated systems, but those systems soon find that they are technologically behind their contemporaries, which have been purpose built with ideas like data availability and analytic integrations in mind. In my experience, even with virtualization you’re going to continually hit limits on what you’re able to extract from those legacy systems.”

“From an integrator standpoint, we are often interfacing with outdated legacy systems as part of equipment refreshes or plant upgrades that require us to run older OS in virtualized environments,” says Gray’s Minier. “For manufacturers, running critical operations on older outdated OSes can present some significant problems concerning security and performance—and the longer those upgrades are put off, the more costly they can become. The option to install new hardware/software versus design custom connectivity solutions can vary and is subject to many factors. Purchasing new hardware/software is definitely more of a long-term solution but it is not always possible due to limiting factors such as cost.”

 “I’d be a proponent of engineering a solution that is more supportable for the long term, which means going through the pain of new hardware,” says ICONICS’ Reissner. While the VM solution will solve the problem, it makes the solution more complex and, over time, unsupportable. “Plus, it’s just kicking the can down the road for someone else to deal with upgrading it in a more easily maintainable way,” he adds.