Many industrial control systems used in the food industry are quite old, highly customized, and were not designed with security in place or with the ability to be made secure through updating. Yet, more of these legacy systems are being connected to workstations on corporate networks, thus offering an entryway for hackers to other vital systems, according to the Food Protection and Defense Institute (FPDI) at University of Minnesota. Given the vulnerabilities identified in the food industry and the increasing threats to ICSs, it is critical that food companies have the tools, techniques and knowledge to protect themselves.
We reached out to John T. Hoffman, senior research fellow with the FDPI, and Mackenize Morris, senior industrial consultant at Dragos Inc. and ISA Global Cybersecurity Alliance member for more on this subject, as well as a few security tips to keep your data intact.
FE: Being one year from the JBS cyberattack, what would you say were lessons learned?
Hoffman: Let me list the top issues in three categories:
- Have a detailed inventory of the IT and OT systems within all your facilities.
- Have detailed information on-hand about all your networks, OT and IT.
- Have isolated backups for all your data and frequent server images.
- Have active, real-time intrusion monitoring on all networks and all gateways into your systems and email.
- Ensure all OS, applications, firewalls, anti-malware and anti-virus software, intrusion detection, routers and hardware across all networks, OT and IT are secure and current.
- Properly maintain and update your networks, hardware, applications and OS.
- Implement applications that monitor and record/report user ID that access all critical systems and files (IP, financial, etc.).
- Isolate and segment your critical systems from the Internet, particularly your OT systems.
- Isolate and secure your back-up location.
- Never allow external devices to be connected on any of your network systems.
- Ensure that all connectivity with customers and suppliers is via secure VPN.
- Ensure that all suppliers and customers comply with your security standards.
- Use physical devices and 2-3 factor access for all network entry.
- Limit network access on need-to-know/access basis only.
- Update or replace all legacy cyber or network-controlled devices, applications and OS.
- Conduct recurrent cyber-hygiene training for all employees, at every level in the firm.
- Update training and operational SOPs continuously.
- Engage third party cybersecurity firms to assist in both training and system monitoring.
- Conduct weekly cyber risk reviews.
- Conduct cyber event assessments as soon as practical after each event, even if seemingly minor.
- Have and train a plan for how you will eject an attacker from your system once detected.
- What you may do tomorrow will not prevent what may happen today.
- Have a plan for IT and OT system security and response to any intrusion.
“The more convenient access is to you, the more convenient access is for the cyber attacker.”— John T. Hoffman, senior research fellow with the FDPI
The industry isn’t waiting around either. Companies are actively looking to improve their cybersecurity posture to prevent ransomware events that might interrupt operations. A few common themes have emerged among recent ransomware events. Common themes of the delineation between organizations’ traditional enterprise environments and their operational networks include:
- Weak boundaries between OT and information technology (IT).
- Poorly understood interactions between systems in OT.
- Poorly understood interactions between systems of systems between enterprise IT and OT.
- Remote access schemas put in place to serve work-from-home pandemic needs.
Understanding the ingress and egress of traffic through the boundary of the OT network is the principal requirement that has been highlighted by recent attacks.
“With HVAC being frequently identified as a crown jewel system, a compromise would bring the production process down and cause loss of warehoused products.”— Mackenize Morris, senior industrial consultant at Dragos Inc.
FE: What other items do companies overlook when putting a security system in place?
Hoffman: Most of the recommendations listed above are not considered or they are put off due to time available, costs, training and/or convenience. Many firm managers want instant access to all data on all operations. That is neither realistic nor secure. The more convenient access is to you, the more convenient access is for the cyber attacker.
It is often very difficult to convince the firm’s board of directors or the financial committee to spend significant resources against a low probability, even if potentially high consequence cyber event.
It also worth noting that most medium-sized firms in the sector assume the target of such attacks will be the big firms with deep pockets. The truth is that cyber attackers target small- to medium-sized firms as the gateway to the larger firms, then they demand a ransom or steal the IP of both.
Most firms do not have current and detailed IT device and network component inventories. This is often due to the industry consolidation across the sector. The result is that many sector networks operate on outdated OS or have outdated, insecure devices, drivers and application code. These create easy targets for cyber crooks.
Morris: OT has been an afterthought for a long time, but that is changing, and companies are now looking to invest in the security of their OT networks and continuity of operations. Historically, organizations have placed much of their investment on perimeter defenses and overlooked cybersecurity controls inside the OT network. We recommend five critical controls that companies implement as the bedrock of an effective OT cybersecurity program.
- An ICS-specific incident response plan: Create a dedicated plan that includes the right points of contact, such as which employees have which skills inside which plant, as well as thought-out next steps for specific scenarios at specific locations. Consider tabletop simulation exercises to test and improve response plans.
- A defensible architecture: OT security strategies often start with hardening the environment—removing extraneous OT network access points, maintaining strong policy control at IT/OT interface points, and mitigating high-risk vulnerabilities. Perhaps even more important than a secure architecture are the people and processes to maintain it. The resources and technical skills required to adapt to new vulnerabilities and threats should not be underestimated.
- OT visibility: Our recent research indicates that 90% of organizations have extremely limited to no visibility into their OT environments. A successful OT security posture maintains an inventory of assets, maps vulnerabilities against those assets (and mitigation plans), and actively monitors traffic for potential threats.
- Multi-factor authentication (MFA): MFA is a rare case of a classic IT control that can be appropriately applied to OT. Implement MFA across your remote user-to-system connections to greatly increase security for a relatively small investment.
- Key vulnerability management: Knowing your vulnerabilities—and having a plan to manage them – is a critical component of a defensible architecture. Over 1200 OT-specific vulnerabilities were released last year, the majority of them with incomplete or erroneous information. While patching an IT system like a worker’s laptop is relatively easy, shutting down a plant has huge costs.
FE: Why is cybersecurity critical to food and beverage operations?
Hoffman: Today, nearly all food and agriculture production depend upon IT and OT systems, from precision agriculture operations on the farm to packing houses to food manufacturing, to warehousing, transportation, as well as food retail and food service. When these systems fail or are brought down in cyber events, there is an immediate impact within the supply chains they are a part of. Worse, every cyber attack result in cost increases for the firm impacted and their customers, including consumers. The JBS ransomware attack is an example of how fast this occurs.
Morris: The food and beverage industries are like other critical infrastructure verticals in that cost reduction and increased efficiency driven by smart devices, interconnectivity, automation and analytics have greatly expanded the companies' attack surface. Perhaps more so than other verticals, food and beverage use purpose-built machinery for processing, cooking, chilling, packaging, etc. This specialized machinery is procured from around the globe. Owners and operators leverage support contracts to aid in troubleshooting and maintenance as the machinery requires specialized skills that the local plants often do not have. This requirement means that vendors remote into their supported machines leveraging remote technology. We have found numerous insecure remote access solutions during their engagements in this vertical, ranging from always-on VPN connections to the vendor’s network to using TeamViewer and VNC to access OT systems.
Similarly, we have identified several food and beverage sites leveraging vendor-supported and maintained heating ventilation and air conditioning (HVAC) systems. With HVAC being frequently identified as a crown jewel system, a compromise would bring the production process down and cause loss of warehoused products. Often these systems are also connected back to the general OT network, which could enable further compromise of the production system.
FE: Do you think data is safe in the cloud?
Hoffman: Data is as safe in the cloud as the cloud itself is safe. Therefore, it is imperative that firms do careful research and appropriate due diligence when selecting and contracting with a cloud services provider. But there are steps the firm itself can take to build in security of their cloud-based data. These range from data encryption to communication systems employed, to name just a few.
Morris: Cloud is sometimes a scary word in the operational technology world. The answer is that it depends. Cloud implementations that are done correctly are secure, and implementations that are done poorly are vulnerable. Cloud in OT is already here, many vendors and OEMs have pushed cloud hosting, or cloud solutions are part of their service packages.
The most important part of cloud integrations is controlling access and authentication to the data. Additionally, monitoring and controlling the connection of local OT assets and the cloud resources is an important part of understanding the ingress and egress behaviors of the OT network.
FE: Can you share best practices to combat cyber attacks?
Hoffman: In addition to what I listed in the first question:
- Have a plan. Train the plan. Exercise the plan. Continuously update the plan.
- Give every employee cyber-hygiene training. Employees are most often the weakest link in a firm’s cyber-defense system.
- Have specifically assigned cybersecurity responsibilities to key staff at all levels of the firm.
- Create a company cybersecurity team to advise management, that plans and executes training and exercises, and that leads any cyber-attack response and recovery.
- Locate and meet your local, state and federal cyber crime officials before there is a cyber event.
- Find an attorney with experience in cyber crime events within the food and agriculture sector.
- Find and contract on-call services from a cybersecurity firm with experience in dealing with and recovering from a cyber attack on food and agriculture sector firms.
- Consider engaging third-party, real-time intrusion detection services.
Morris: We detailed the five critical controls for OT environments earlier. In addition to those, there are common best practices for OT cybersecurity:
- OT aware passive security monitoring for threat detection capabilities without impacting operational performance.
- Map data flows between networks and trust zones to better understand the interdependencies of the OT environment and critical assets.
- Conduct regular assessments of the cybersecurity posture and maturity of the OT environment to help shape the roadmap forward.
- For mature organizations, perform an assumed breach penetration test of the IT/OT boundary to validate that security controls are in place and effective.
There are a lot of best practices and security solutions. Recommendations can often be overprescribed, so it’s important to evaluate the current maturity of an organization’s cybersecurity to calibrate an implementation plan. Smaller investments like creating an ICS-specific incident response plan and creating an OT asset inventory go a long way.