We all know that remote control, aka “Remote Desktop Protocol” (RDP) in the IT world, can save precious time and footsteps when a tech needs to make changes to a server—especially in these times of COVID-19 keeping people operating out of their homes. We also know that remote ports on OT skids and controls equipment can help engineers with both maintaining and tweaking equipment to keep it running at its best. That’s why in many cases these remote ports on OT equipment are either cellular (private) to guard against casual connections—or some food processors still leave the LAN cable unplugged from the machine port unless it’s needed for maintenance.

However, with RDP falling into common use in the IT world, and with so many industrial controllers using Windows or even UNIX/Linux-based operating systems (OSs), employing RDP without safeguards on either Windows or Linux platforms can potentially open a path into a control system for a hacker or would-be extortionist—kinda like the inverse of Pandora’s box.

Human engineering is one way to connect maliciously with computers and controllers. For example, one of the most common ways for hackers to get into home systems is to present either a fake web-based message—or send the user a SPAM email—saying that the user’s computer is operating poorly, and they (the “Windows Service Department”) can help the homeowner with “fixing” the machine. The next step is to get the user on the phone, convince him or her that the computer needs attention and talk the user into downloading a RDP program where the “service technician” (aka criminal) will connect with the user’s computer to “repair” it when there are really no issues with the machine.

Problem is, once the hacker has control of a user’s desktop via the newly installed RDP software—which by the way was ignored by the user’s antivirus system because the user OK’d the installation—the hacker can now do whatever is necessary to get the user’s money…or passwords or bank account information. A user can pull the plug, but most likely it will be too late. The RDP program will run on startup, and the user’s antivirus program will continue to ignore the malicious RDP task, which now runs hidden in the background, leaving the computer exposed to the hacker(s).

Similar things can happen to OT equipment

Similar things can happen to process control systems. Even if they’re presumably protected, a hacker could enter from a connected IT system—or simply enter through unprotected and left-open RDP ports on HMIs or connected Windows-based controllers.

A food processor could face several problems caused by a hacker entering a control system. For example, the hacker could alter a kill-step temperature/time value(s), causing a potential problem for a food or beverage product to be contaminated with bacteria—hopefully not a pathogenic variety. Another possible scenario: a clean-in-place (CIP) system is altered such that strong alkalis or acids remain in pipes carrying food, soup or dairy. Best case scenario, food has to be junked and tastes terrible; worst-case scenario, bacteria-laden food gets into circulation because of unclean piping.

Water company spots RDP takeover before damage can happen

Unfortunately, a local Florida water utility located in Oldsmar was hacked into using RDP (TeamViewer, a readily available desktop sharing program) to take over an operator’s screen, with the hacker attempting to make changes to the acid/alkali pH balance to the fresh water going out to customers. Fortunately, the operator had spotted the problem before anything could happen.

While monitoring the system around 8 a.m. in the morning, the operator had noticed his cursor moving around on the screen but didn’t think much about it because his supervisor often would log into the system to check operations. However, later in the afternoon, the operator observed the screen as someone took control of the mouse and directed it to the software that controls the water treatment system. The hacker worked inside the program, for a few minutes, and increased the level of sodium hydroxide (NaOH) from 100 ppm to 11,100 ppm, which would have changed the water’s pH level. As soon as the attacker left the system, the operator immediately changed the NaOH concentration back to 100 ppm. [1]

While there was no danger imposed to water quality—as the process control system is incapable of making such a big change in a short period of time with the equipment following through on the requested change—the situation demonstrates there are bad actors in cyberspace, and the water company made changes to the system to prevent the hacker from re-entering the controls.

Once a hacker connects via RDP to any computing system, unless operators take immediate action, the hacker can use this connection to log in at any time in the future, extort money, install ransomware on the computer, and/or sell the connection login/password and other data on the dark web, making an unprotected system available to anyone who wants to purchase the information.

According to DNV-GL, some methods to protect against unwanted RDP attacks include making sure all security patches are installed on computers and controllers, restricting login attempts to three and locking the account, closing RDP port 3389 on computers, routers and controllers when not in use and making sure any public cloud-based systems are not using RDP at all. For more on this subject, visit the DNV-GL web page on RDP. [2]

If you’re not sure if you have an open RDP port on 3389—or other potential port that can be attacked, e.g., including Microsoft network protocols, a good way to check is Steve Gibson’s Website program, “ShieldsUp!” open-port discovery tool. There is nothing to download, and you can run it from any computer OS and find out which ports are open on your system and exposed to the world-wide internet. [3]

Looking more closely at the OT issues

Marty Edwards, VP of OT Security, Tenable

The Oldsmar Water company prompted me to ask a few more questions relating to cybersecurity in the OT world. I connected with Marty Edwards, VP of OT at Tenable, an industrial OT security company. Edwards also has past experience with the U.S. Department of Homeland Security in cybersecurity issues.

FE: For such a critical application as in controlling drinking water pH, there should be other cybersecurity protections. What are they? Would some form of two- or multifactor (ID) sign-on be appropriate?
Marty Edwards: There are a number of technical solutions that could be used to improve the cybersecurity of these systems but, without knowing additional details about the specific installation, I would only be speculating as to their efficacy. Multi-factor authentication certainly is one of the technical controls that would seem appropriate here.

FE: This was a lucky catch in that someone was actually awake and monitoring the screen. What if an operator was “making the rounds” of the facility and wasn’t paying attention? Since this would have been a long process to contaminate the water to dangerous levels, there was plenty of time to catch this change. What system could be put in place to provide an ample audible or email warning that someone (hacker) had broken into the system?
Edwards: It is essential to maintain visibility into all of the systems and devices that comprise a control system operating critical infrastructure. Logging of all connections into the system and alerting based on policy violations or anomalous behavior certainly can help pinpoint intrusions before they are able to cause any harm. Monitoring your devices for unauthorized configuration changes can also assist in reverting back to normal operations as quickly as possible.

FE: Could a change management system catch this and send a warning? Or, do we need some more high-tech solution such as an AI-driven network monitoring scheme?
Edwards: Change management could be used to catch unauthorized changes and provide a warning or alert. Basic cybersecurity hygiene and fundamentals apply here and by doing the basics right we can reduce the risk of many of these attacks that go after the “low hanging fruit.”

FE: This points the way for hackers to gain access to other unprotected control systems. What advice would you give to control system operators?
Edwards: Control system operators must invest in the people, processes and technologies in order to maintain visibility into their basic cybersecurity posture. Knowing what devices are on your networks, how they are configured, who is making changes—and when—to the system will become extremely important during a forensics investigation.

FE: In the food/beverage/nutraceutical industry, an attack could be far more reaching and devastating. As I described earlier, for example, controls are altered to prevent a thorough pasteurization process; ingredient additions are altered; or a clean-in-place (CIP) system is altered such that strong alkalis or acids remain in pipes carrying food, soup or dairy. Best case scenario, food has to be junked; worst-case scenario, bacteria-laden food gets into circulation. How do we protect these sensitive systems from outside manipulation? Grant “read only” access”?
Edwards: Read-only access is certainly something that can be considered. I would also suggest that there are non-digital hard-wired ‘safety controls’ in place for final inspection of product, such as laboratory testing or other offline procedures. Evaluating the risk that is introduced into a process by implementing something like remote access is a critical business step that is often overlooked by companies. They see the cost savings side of the equation but lack the cybersecurity experience necessary to ask the right “what if” questions. My advice is to bring in the experts to help you evaluate those risks and put the right controls in place to bring the risk to an acceptable level for the business.

FE: There are a lot of 20-30 year old data acquisition/SCADA nodes in plants today. I’ll bet there’s still some Windows 2003 or earlier servers/nodes out there (maybe even DOS)—not counting unprotected PLCs and DCSs. What’s your advice on these? Make sure their data is “read-only” with no access to PLCs and DCSs? I’m a strong fan of following ICS-CERT, but a lot of this ancient equipment is probably no longer patchable, is it?
Edwards: Different components within the control system tend to age at different rates. There could be components that are decades-old and some that are no longer being maintained or patched by the vendor. That being said, there are also components of the system that more closely resemble commercial off-the-shelf IT products. As such, it is critical that organizations know the devices on their networks and take measures to protect and harden the devices that they can. This includes implementing a defense-in-depth strategy to protect the assets that are the most critical to their individual business.

About Marty Edwards
Marty Edwards serves as vice president of OT Security at Tenable, where he works with government and industry leaders throughout the world to broaden understanding and implementation of people, process and technology solutions to reduce their overall cyber risk. A 30-year industry veteran, Edwards has received numerous awards recognizing his achievements. Prior to joining Tenable, Edwards served as the global director of education at the International Society of Automation (ISA), as well as the longest‐serving director of the U.S. Department of Homeland Security’s Industrial Control Systems Cyber Emergency Response Team (ICS‐CERT).

For more information about Tenable, visit its website.

References:

[1] “Someone tried to poison Oldsmar’s water supply during hack, sheriff says,” Tampa Bay Times, 9 FEB 2021, Website.

[2] “Hackers are exploiting Remote Desktop Protocol (RDP): 14 steps you can take to protect your systems,” DNV-GL; Website accessed 17 FEB 2021.

[3] “ShieldsUp!,” Steve Gibson, Website accessed 17 FEB 2021.