Automation's Achilles' Heel
More than a few food plant engineers shrug with indifference when the issue of controls-system security is raised. "Getting line three back into production is my priority," mutter these overworked and understaffed souls. "I've got enough to do without tackling IT issues."
Ironically, the problems on line three and the lack of adequate controls-network security might be interrelated. While food and beverage plants are unlikely targets of cyber-terrorists, there are more than enough worms and viruses infiltrating the plant floor's information highway to cause unscheduled downtime. Random maliciousness from virtual vandals is the root cause of a growing number of business disruptions, and the frequency is trending up, experts warn. Security breaches could supplant mechanical failures as a leading cause of downtime in the coming years.
"Food and beverage companies haven't had the awakening yet" to the dangers of malicious interference with plant controls, adds Michael Bush, manager of Rockwell Automation's software security division in Mayfield Heights, OH. Nonetheless, "there is a concern in some quarters that food and beverage might be the next target for tampering."
Corporate losses from computer crimes are actually declining, the Computer Security Institute's CSI/FBI Computer Crime and Security Survey suggests, but the reductions are occurring in general business. Viruses account for more than half the losses, with another third attributed to denial of service, and both are issues in manufacturing. The problem, many believe, is that controls engineers with little or no training in network security are responsible for safeguarding the plant floor, while IT professionals with a solid understanding of appropriate security safeguards steer clear of the engineers' domain. A collaborative approach is needed, and it's beginning to happen, notes Pamela Mars, batch product manager with Honeywell Process Solutions in Phoenix. "With security becoming a greater and greater issue, we are seeing IT and operations coming together to address it."
Financial gain through controls-system tampering poses an ominous threat. Scott Wooldridge, former engineering manager and now sales vice president with Citect Inc., Alpharetta, GA, recently assisted a Chicago manufacturer who had lost control over part of its SCADA system to an outside vendor. The vendor used his access to generate unnecessary maintenance job orders. The scheme went undetected for almost two years, Wooldridge says.
The consensus is that outside attacks have surpassed careless internal mistakes as the leading cause of data network problems. Firewalls that are not properly configured, poorly maintained passwords and access controls and dial-up modems on machinery are among the issues that can knock a plant off line or lead to complete loss of control of a network. Another common and careless breech is the operator HMI loaded with Internet Explorer, "one of the most highly exploitable pieces of software out there," according to George Japak, vice president of International Computer Security Association (ICSA) Labs. One minute, an operator is playing poker on line; the next, a teenager in Uppsala, Sweden, is turning the CIP cycle on during filling operations. Concerns about worms and viruses have led some firms to ban outsiders' laptops altogether. If a service technician needs to hook in, companies are requiring him to transfer any needed programs to a loaner laptop before tapping into the network.
Better-armed GoliathsMulti-plant companies with recipe-management programs and complex ERP systems face the greatest exposure. Disruptions such as the SQL Slammer worm that shut down 360,000 systems worldwide, including ATM networks and air travel systems in January 2003, dramatized the business-disruption risks to major corporations and helped end turf wars that thwarted cooperation between IT and engineering to safeguard systems.
"When viruses and worms like Nimda and Slammer hit 18 months ago, many businesses were perplexed," notes Rashesh Mody, chief technology officer for manufacturing software maker Wonderware Inc., Lake Forest, CA. "Today, people are better informed in terms of security." Mody estimates 70 to 80% of food companies with ERP systems have instituted policies, procedures and tools to better safeguard their systems.
ICSA is "the digital Underwriter's Laboratory," explains Japak. If a new virus is found to penetrate a vendor's firewall, the Mechanicsburg, PA, firm immediately withdraws certification until the problem is fixed. Many Fortune 500 firms specify certification in their firewall RFPs, Japak says, while smaller companies "take an outside consultant's word on virus protection on blind faith, and what they run depends on what that consultant re-sells."
Global companies like Hershey require remote network access and shared Internet connections between plants. They also are cognizant of the Sarbanes-Oxley Act's mandate that they secure their network controls, and they have responded. But single-plant operators also are at risk, and these processors may now be more vulnerable than the biggest firms. "The risk to the beverage company with narrow margins and a single plant is larger," Rockwell's Bush believes. "Does he have enough profitability to survive three days of downtime and not go out of business?"
Security by obscurityIn the days of proprietary batch controls and non-networked plants, security was a minor issue. The era of open architecture and enterprise-wide data exchange has lowered the cost of automation but also heightened exposure to risk. Windows-based operating systems are ubiquitous and a favored target of malicious code. One defense strategy is to stop paying royalties to Bill Gates and opt for an alternative operating system.
"Security by obscurity is a risk management approach that can be a very good strategy," Bush suggests. "It isn't risk free, but it makes you less of a target, and that's not a bad thing."
Wonderware's Mody flatly rejects the alternative O/S defense-not surprisingly, inasmuch as Wonderware introduced the world's first Windows-based industrial HMI in 1987. Mody is not alone in suggesting controls architects risk tossing the baby out with the bathwater by opting for non-Windows O/S, though he goes further. "They sound like free operating systems, but there are more virus attacks on Linux and Unix than on Windows," Mody insists.
His assertion will be greeted with skepticism in some circles. Many IT professionals dread Patch Tuesday, the first Tuesday of the month when Microsoft releases the antidotes to the latest code-busting viruses aimed at Windows. As the number of patches pile up-15 were issued in March alone,-systems administrators approach Patch Tuesday with dread. Still, Windows alternatives also have to be supported with security systems. "Linux is starting to get a lot more traction, and you will see it become a larger target as it becomes more popular," Japak says.
Top management wants ready access to plant floor data, Universal Dynamics' Kay points out, and that means controls engineers "are stuck with operating systems like Microsoft and Linux. As soon as you opt for a closed operating system, security improves but the price of your system is going to quadruple," he warns. Open systems can be hardened sufficiently, but common sense must balance thriftiness: don't purchase network nodes at Best Buy and think they're comparable to industrial hardware, says Kay.
The good news about automation security is that the best safeguards are not necessarily expensive, Citect's Wooldridge says. The bad news is that they take time to identify and implement. Redundant defenses and a disciplined approach to maintenance can thwart random attacks and sabotage by disgruntled employees. Perform a vulnerability assessment, draft an accurate network diagram, and develop a formal security policy detailing the policies and procedures, he advises.
It's not the advice the controls manager trying to determine why a particular piece of equipment periodically goes off line wants to hear. On the other hand, it might be the solution that's needed.
For more information:
Scott Wooldridge, Citect Inc.,
Pamela Mars, Honeywell Process Solutions,
George Japak, International Computer Security Association,
Michael Bush, Rockwell Automation,
Rashesh Mody, Wonderware,
John Kay, Universal Dynamics,