In this article, I ask system integrators and automation suppliers how they work with users to deliver automation updates safely and efficiently. We also look at the importance of backups in updating automation software and equipment.

Automation software update delivery

System integrators (SIs), machine builders and automation suppliers provide updates to food and beverage processors who range from those with their own engineering staff—to companies with maybe just a plant engineer—to a facility with no really qualified tech person on staff. This means while some processors can certainly handle updates with no assistance, others will need some help—and this means suppliers have to be ready to provide not only remote assistance, but also potentially send someone to the client’s plant to supervise the update process.

Providing updates is generally dependent on the technical capabilities and willingness of each client’s technical resources, says Dan Malyszko, director of Denver Operations, Malisko Engineering, a Control System Integrators Association (CSIA) Certified Member. “On one end of the spectrum we have clients who have a very capable technical group—so we’ll notify them of the update to be installed and they execute the update themselves.

“On the other end of the spectrum we have clients with very limited or no technical group so we first notify their operations manager that an update to their systems is recommended/needed, then schedule a time and day where our technical resource(s) can perform the update(s) either on-site or remotely via VPN access while someone at the plant monitors the systems on the plant floor for abnormal operation once the update is completed.”

“In all cases we first have a discussion with the plant to cover the extent of the updates, the plan for installation and the potential risks along with the corresponding risk mitigation and recovery plan, if needed,” adds Malyszko.

Being on site can reduce errors

Being onsite has its advantages, should something go wrong with an update. “Generally speaking, manufacturers will inform us of updates and the preference is to actually be onsite in combination with any production operators or managers,” says Sean Creager, senior electrical engineer for Huffman Engineering, a Certified CSIA member.

“From a safety standpoint, working together and ensuring the environment around the area is safe and secure during any updates is the best option,” adds Creager. “If that is not a possibility we have been able to remote in for updates, but we would definitely want to make sure an onsite operator is available during the updates for both safety and practical purposes. On a production line there are elements to updates that require a physical presence with the equipment.”

Interstates, a CSIA Certified Member, is another SI that can provide whatever help a processor needs. According to David Smit, systems analyst, Interstates can provide remote software upgrades and can fit any necessary requirements that the processor prefers. The SI can also send teams to a site to perform upgrades when technical staff is limited at a facility.

When necessary, Allpax, maker of retorts and other process equipment, will perform the work remotely, but with the assistance of the local group that manages the IT/OT system for the user, according to Jonathan Watkins, VP of technology.

Remote updating process provides choices

With remote capabilities, the update process is much easier. For example, Jerry Leuthold, senior project manager at Bachelor Controls, Inc, a CSIA Certified Member, suggests that his engineers can VPN into a manufacturer’s system to perform the updates or let the processor apply the updates themselves.

Updates—unless they are safety updates—are left to the end user, says Terry Meister, controls engineering manager for Automated Systems Group, Applied Manufacturing Technologies (AMT), a CSIA member. Most software updates on PLCs are for added features and simplicity, also to ensure operation with newer hardware versions. With this being the case, most of AMT’s customers want the same software revision on all the systems in the plant if possible. This allows them to standardize the software on laptops and for all their maintenance people and technicians. “When an update is desired at a plant level, they will update the controls specification and roll that out to any new projects,” says Meister.

Most of the time, the plant will handle the updates, says Meister. If a customer does not have the resources to update the software and has reason to, AMT will have a technician go the plant and perform the work. Having someone who is not a PLC programmer try to update the software is taking a risk. When done wrong, it has the potential to “brick” the processor.

From an application software perspective, food manufacturers want to minimize disruptions to their operations due to update downtime, says Antony Bourne, senior vice president – Industries, IFS. But there are also new functionality and security benefits that they can realize by updating to newer software.

“Here’s where cloud brings huge benefits and is one of the reasons IFS launched IFS Cloud,” says Bourne. It provides a single solution that delivers a world class-leading application across service management, enterprise resource planning and enterprise asset management. In a nutshell, IFS Cloud marks the start of a new, twice-yearly feature update cadence for an ’evergreen’ customer experience. This gives customers the option to always be on the latest version of our software without the disruptions that come with full-scale upgrades. It also provides manufacturers greater visibility, predictability, control and flexibility in planning their own business development and adopting new capabilities from IFS.

Do you remember these—or are you still using them? (from the author’s collection of art). It’s July 1999, and Windows NT is giving way to Windows 2000. In 1999, many software updates still arrived by floppy disk or CD-ROM, as most connections to the internet were dial-up or DSL.  Not all these companies exist today, and many have been merged into larger companies. Starting (upper left, and moving clockwise): ASAP’s ASIC-300 machine automation software; Siemens SIMATIC WinAC for Windows NT/2000, supporting OPC; Intellution’s FIX Web server HMI; Nematron’s Paragon SCADA for Windows 95/NT and IBM OS/2; Citect for Windows connects via MS ActiveX with ABB’s Advabatch, GE Fanuc CIMPLICITY on Windows CE; Total Control’s FactoryClient 2000 for Windows CE; UDSATA’s Xfactory; (and center): Foxboro’s I/A Series DCS with OPC for Windows and UNIX. Source: Wayne Labs.

Making the controls update process safe and reliable

With more than 500 control system manufacturers and tens of thousands of companies using its programming products, CODESYS’s system of programming tools is the basis of a comprehensive suite that covers the entire software side of automation including industrial internet of things (IIoT), communication, soft PLC, motion, drives, CNC, robotics, redundancy and visualization.

CODESYS, a CSIA member, is a software platform for industrial automation technology. The core of the platform is the IEC-61131-3 programming tool “CODESYS Development System.” It offers engineers practice-oriented, integrated solutions for the convenient configuration of automation applications.

When asked what procedures are followed to make the update process as safe and reliable for its users, Chris Schulze, CODESYS Corporation VP sales listed four steps in migrating from PLC to industrial PCs:

Step 1: Change the I/O to remote bus I/O

Step 2: Change the PLC to an IPC-based solution, write the program ready for the future.

Step 3: Exchange the PLC with the IPC.

Step 4: If necessary change the I/O step by step.

Website connections ease the update process

I recently updated the firmware in my Canon EOS digital camera, Thanks to an easy-to-use Canon support website, it was quick and easy to find the latest firmware update for my particular model, download and install it. The update widens support for Bluetooth connected devices.

Likewise, food and beverage processors are most likely to visit machine builder or automation supplier websites to update firmware in a piece of equipment. But how can a user be assured that an update won’t shut down a control system or turn a PLC into a “brick,” meaning that the device’s onboard memory may have been corrupted, and the controller rendered useless without a hardware repair. Secure portals are one method of making an update both reliable and safe.

“When upgrades are released, Yokogawa posts these upgrade files to our secure customer portal,” says Gerald Hardesty, product marketing manager. This customer portal is password protected, and access is only given to registered Yokogawa customers, so hackers cannot access the upgrade files.

Bosch Rexroth — Electric Drives and Controls ctrlX Device Portal requires a secure registration process to connect the device to the cloud-based portal, says Allen Tubbs, product manager. This establishes a secure, authenticated connection between the device and the Device Portal. The Device Portal tracks the last known connection time and maintains a digital twin of the control so that the state of the controller is always known, regardless of connection status.

As an instrumentation and controls supplier, Emerson’s “plan, select, test, and release” strategy for both cybersecurity updates and feature packs is specifically designed to make the update process safer and easier, says Sesh Natarajan, DeltaV Product Director. Pre-selection of updates and features helps plants identify the most applicable updates that will help improve operations and security, while extensive pre-testing minimizes risk of disruption.

Huffman Engineering follows a proven method of risk assessment, developing a plan, reviewing with leadership, executing the updates according to the agreed plan and checking with onsite operators to make sure they understand the changes made, says Creager. “We also always prefer planned updates so everyone is aware of the dates, and it can be done in the most efficient manner with scheduled downtime.”

If PLC software/firmware updates are required, the respective links to the Siemens Industry Online Support (SIOS) entries are provided, which then contain specific manuals, files and/or procedures for installing the update(s), says Luis Narvaez, product marketing manager, basic automation & industrial security. Siemens also allows users to ensure that the downloaded files are not changed/modified during the download process by implementing a 256-bit secure hashing algorithm (256-SHA). Further information and instructions are available here.

Once the respective update file(s) have been downloaded and verified by the users, they can safely perform the updates of SIMATIC components using various methods, however Siemens mostly recommends using tools like the SIMATIC Automation Tool, which allows users to perform common commissioning, maintenance and service tasks such as setting network parameters, updating firmware and backing up/restoring programs/data from supported devices. More information and a link to download the tool can be accessed via a special website.

Application upgrades—making sure they’re also safe

ADISRA, provider of HMI and SCADA software, has a multiple step process it takes when working with its customers, says Bruno Armond Crepaldi, chief technology officer. ADISRA uses “sandboxes” or virtual machines to run trial upgrades in order to identify any code or database conflicts. “We try to simulate the various customer environments to ensure that our software will work seamlessly for our customers,” adds Crepaldi.

ADISRA performs various testing from functional, non-functional, technical testing, quality assurance, data integrity, security, performance, browser, and device testing to name a few. “Throughout this testing process, we are capturing and tracking issues and prioritizing them based on severity,” says Crepaldi.

ADISRA also performs configuration testing. This testing involves turning on or off features in the underlying architecture to capture the ideal configuration to best accommodate its users. “Finally, all this is performed before we perform user acceptance testing,” says Crepaldi.

ADISRA SmartView provides modern visualization and deployment options for HMI/SCADA functionality. Source: ADISRA

Backwards compatibility is part of Inductive Automation’s DNA, says Travis Cox, co-director of sales engineering. We are all committed to ensuring upgrades are safe and easy to perform, as they are necessary to keep the customer up to date. Developers are always cognizant of how a change or addition will affect backwards compatibility. Ultimately, testing is the most important process that we have. Inductive Automation has a very robust development and quality assurance program. The QA team runs a series of automated and manual tests before a version is released. This includes installing and upgrading on different operating systems. The QA team has long-running systems where regular OS updates are applied to identify issues.

The most important process is our release methodology, says Cox. “We have adopted a ‘release train’ style of scheduled releases.” This popular concept is quite simple: releases go out on a regularly timed schedule with whatever fixes and improvements are ready at the time. Think of each release (e.g. 8.0.1, 8.0.2, etc.) as a train leaving the station. The train leaves the station on a predetermined schedule, with whoever made it to the station in time to get onboard.

In software terms, the way this is done is that each fix or improvement is developed and tested in isolation. When each change is finished and verified, it can be merged into the “main branch” of the software. The big advantage is that the testing cycle of the release, itself, is dramatically reduced because the testing was done earlier, and in isolation. “Using this strategy, we provide nightly beta builds as fixes or improvements are made with new production releases every five weeks,” says Cox. “The release train style will allow us to not only deliver releases on a much more rapid and predictable schedule but also to be dramatically more agile in how we respond to issues, as reprioritizing of changes will have much less of a cascading impact.”

Historically IFS has made large core releases every two to three years with quarterly updates containing a combination of fixes and new features, says IFS’s Bourne. “With IFS Cloud, we now have twice-yearly releases of new functionality, each supported through monthly service updates that only contain fixes.”

In many cases, manufacturers operating under particularly rigid regulatory requirements (e.g. FDA), have requested flexibility regarding the timing of update adoption in their production environment. “With a clear focus on our customers, we’ve chosen to offer the flexibility required for these customers, (via a defined time window where we provide access to the update), to determine when the time is right for them to push a new version to their environments,” says Bourne. Having this flexibility is essential in many regulated industries where any change to IT systems is likely to trigger the requirement for a new third-party audit before rolling it out into production.

As a result, IFS users can:

• Avoid the major/very large core releases that represent big upgrade steps in favor of continuous updates in smaller steps
• Clearly separate delivery of fixes from new features so that fixes can be applied regularly and frequently with zero/minimal impact to business and without requiring customers to apply every new feature release
• Receive smaller, but more frequent, steps and service updates that can be applied with minimum impact on processes and systems
• And, along with the IFS Lifecycle Experience enjoy a more ‘evergreen’ software experience and greater visibility, predictability and control, with total transparency in terms of what new features will be available when.

Don’t forget to backup before updating

Maybe not the prettiest sight nor the easiest to use... I've been using Clonezilla  to image OS hard disks--both Windows and Linux--for years. It gets the job done, it's open source and free, and I've never had an image I couldn't restore--whether a rotating disc or solid state drive. Source: Clonezilla.

Before I do any updates or new software installations, I always make sure I have a recent OS backup (no more than a week old), and if not, I back up my system disk by making a fresh image of it; I also keep a few recent backups as well. My system disk has no important data on it, and data I do have on it is transient in nature. All important data is on my Ubuntu Linux server, which is backed up to three different storage systems on a daily basis.

“We always recommend to users that they back up before updating,” says Tony Baker, chief product safety and security officer at Rockwell Automation. “The time taken is a small price to pay in the case of any complications. Consistently backing up your software is even more imperative in the age of ransomware. Without recent backups, a potential ransomware attack could be incredibly costly to an IT and OT infrastructure.”

For processors who need help in setting up a backup system, a system integrator can help with the task. Malisko Engineering helps processors establish a plan, policy, or SOP (standard operating procedure) for periodic backup; in many cases “automated” backup of the critical system components. The document(s) for “backup” cover the step-by-step methods for Windows-based equipment, PLC’s and PAC’s as well as the configuration profiles of smart instrumentation. The document(s) for “backup” generally include a section specific to performing a “backup” prior to “updates”.

“It is the standard recommended practice for users to backup their system before performing updates,” says ADISRA’s Crepaldi. It is important to be able to “undo” anything and return to the previous norm should the need arise. Backups should be part of any normal corporate process. It is not just backing up before updates, but you want to have previous backups available should there be an unforeseen problem with hardware failure or malware infections.

“In today’s changing world with malware and other cybersecurity issues, system backups should be done on a regular basis to prevent any data loss if an issue should arise,” says Crepaldi.

For PCs, a restore point is always created before updates are implemented, says Allpax’s Watkins. For PLCs or instrumentation, some processors use Rockwell’s Asset Centre or MDT software, which can automate the task.

AMT always recommends backing up a system before updating, says Meister. This is important on PCs, but a must on PLCs. If anything happens to the PLC when it is updating (say you are downloading new firmware), you will want a backup. We never count on our backups or customers’ backups as being the latest and greatest. Always back up before updating, Meister advises.

Controllers should be backed up before updating, especially if there is no software management tool like Rockwell’s or MDT’s AutoSave in place, says Bosch-Rexroth’s Tubbs. With a proper software management tool in place, all changes and updates are documented and recorded such that the last update is always available if something goes wrong with the current update. This type of tool saves the user the time needed to create a backup every time an update is needed and it also tracks changes in the controller so that there is no need to back-up the controller, just in case someone made an unknown change. It is a much better practice to have a software management tool in place to track these changes and provide a known-good working set, rather than trying to archive before a change is made. The latter does not account for a controller failure, another event that would require a software restore point.

Siemens has a variety of tools available for users to use for backup, says Narvaez. For industrial PCs (IPCs) Image & Partition Creator is a good tool to backup PC images and files. For SIMATIC devices (PLCs, HMI, I/O) SIMATIC Automation Tool (SAT) is also an appropriate tool to use. The process for backing up components varies depending on the product(s) used. Siemens recommends users check with the associated product manuals before upgrading to ensure understanding of the backup/update process.

 Besides the need for the sporadic update task, regular backup of files, programs and data is always recommended as just good “industrial hygiene.” This is not only good practice for the sake of program/Windows update processes, but also should be included as part of an overall disaster recovery response plan (e.g. in the event that recovery of systems or data is required). Various software and tools can also help with the restore operations as well (e.g. SIMATIC image & partition creator, SAT, TIA Portal, etc.)

Ignition’s backup process is very simple, and everything backs up to a single file, says Cox. The restoration process is simple as well and only takes minutes. During the upgrade process, Ignition automatically backs up the configuration in case the processor forgot. In the unlikely event that a problem is found, the organization can roll back quickly and get operations running while they figure out how to handle the problem.

Inductive Automation has worked hard to make the backup process easy and provide automatic backups, so customers can easily understand the process. It’s important that OT and IT work together to develop a robust backup and disaster recovery procedure. That way it doesn’t require a single person to be responsible when issues arise, says Cox.

“Users need to make sure they have current backups,” says CRB Associate Steve Pflantz, P.E., P. eng. “A current backup means that you have done it at least in the last year, or better yet monthly. Current also means that you backup to capture any changes that affect the operation of your system so you can restore it back with those changes.” Consider operating data that the systems collect as well. If that is important, make sure that gets backed up as frequently as needed.

Having the backup is obviously strategic, but processors also need a plan and a method to restore anything that gets messed up, adds Pflantz. “Say you are the victim of a ransomware attack. If you have a god backup of your control system, in theory, you could physically isolate you control system (which is easier if you design your system to be properly segregated with limited and properly controllable interfaces), clear all the memory and hard drives, and reload all the software. With a good plan, you could be back in business without paying the ransom.”

“Make sure your backups are on something that is physically disconnected from your main system, otherwise the backup gets locked up, too,” says Pflantz. Bottom line, have a disaster recovery plan. And keep good backups of all the right stuff!

Dealing with Windows updates

No doubt, Windows updates come along, and generally, they don’t cause a lot of trouble with your desktop computer—as most of the time the updates are dealing with system modules, bugs and security issues. Over the years, however, Microsoft has issued major feature updates every six months, but that seems to be changing to once a year. These updates can change the way you do things, and they might even cause other issues with your system.

Windows-based controllers and workstations on the plant floor are another issue, and most control system vendors and system integrators deal very carefully with these Windows updates. To learn more about these issues, visit “Managing and Surviving Software Updates,” FE, August 13, 2021.

For more information:

ADISRA, www.adisra.com

Allpax, www.allpax.com

Automated Systems Group, Applied Manufacturing Technologies, www.appliedmfg.com

Bachelor Controls, Inc, https://www.bachelorcontrols.com/

Bosch Rexroth – Electric Drives and Controls, www.boschrexroth-us.com

CODESYS, www.codesys.us

CRB, www.crb.com

Emerson, www.emerson.com

Endress+Hauser USA, us.endress.com

Huffman Engineering, www.huffmaneng.com

IFS, www.ifs.com

Inductive Automation, https://inductiveautomation.com

Interstates, www.interstates.com

Malisko Engineering, https://malisko.com

Rockwell Automation, www.rockwellautomation.com

Siemens, www.usa.siemens.com/simatic

Yokogawa Corporation of America, Yokogawa.com/us