Beware of the Self-Wiggling Mouse – Cyber Vulnerabilities in the Water Industry
by Charlie Maclean-Bristol
One of the big new stories from the last couple of weeks has been the hacking of the water treatment plant in Oldsmar, Florida on the 5th February. The town, with a population of 14,000, had a hacking attempt which tried to increase the amount of sodium hydroxide in the water by 100 times. The hack was first discovered by an operator who saw his mouse move of its own accord, which he presumed was his supervisor. He then saw the attempt to increase the sodium hydroxide content and was able to stop the attacker before the composition of the towns water was altered. This week I thought I would talk about the vulnerability of water systems to cyber-attacks and some ideas of what, as business continuity professionals, we can do about this.
The interview for my first job when coming out of the army in 1995, was for the role of Emergency Planning Manager for a water company in the UK. During the interview they talked about telemetry. When I asked what that was, they explained it was a system that controllers at the centre of the company used to monitor and control all aspects of their water network. Having got the job, one of my first tasks was to get a tour of the control room and understand how the network was controlled. In the company there were four controllers who monitored the four quarters of the company area. At this time, they looked after water and wastewater, which is the polite word for sewage! If something broke down or went outside a set of parameters, then they would hear an alarm. They first tried to sort it remotely, which they could often do, but if not then they would prioritise the issue. If it was urgent, a man in a van went out to the site to sort out the problem, if it could wait, then it was scheduled as part of more routine maintenance. There was a similar system for big plants; they had a control room which monitored the plant and if an operator heard an alarm they would dispatch one of the site’s maintenance people to go out fix and the problem.
As the company was technologically advanced for its time, there was remote access to its network by managers and employees, so they had access to alarms and the status of the system. They could log on to the telemetry system and see its status on a laptop, from their van or home. Although they could see the status of the whole system company-wide, they could only make changes and accept alarms within their own patch. There was also a read-only version which I had on my laptop, for people who might want to see the status of the system but not make changes. The system was fantastic operationally, as managers, operators and duty personnel had access to lots of information and alarms and were able to decide whether an issue needed sorting straight away or could wait.
These were the infancy days of the internet, and external hacking of the system was not really considered an issue. The biggest security threat was someone stealing a laptop and then using it to cause chaos within the system. The threat was that the person who had the laptop could cause annoyance by randomly manipulating the system, but an attack by someone who had knowledge of the system, especially from outside the UK was not even considered.
Cyber Vulnerabilities and Risks
Roll forward 25 years and the threat landscape is very different. All the operational advantages of control command of a telemetry system remain, but they now constitute a massive security risk. As the telemetry system can be reached via the internet, the threat is not from a stolen laptop, but a hacker from anywhere in the world gaining access to the system. They may have the industry knowledge to manipulate it in a way that causes a large impact on the users of the water and wastewater.
Although this latest hack got a lot of publicity, there has been a long history of a wide variety of attackers hacking telemetry systems to attack the company which uses them. These have varied from:
- Stuxnet, where Americans and Israelis used malware to infect Iranian nuclear centrifuges, which were then programmed to “spinning themselves apart” while telling the telemetry system all was normal.
- April 2020 – Suspected Iranian hackers tried to alter the level of chlorine in a municipal water plant in central Israel.
- Twenty years ago, “a former contractor of Maroochy Water Services, a water plant in Australia, used a laptop, two-way radio and specialised equipment to release almost 1m litres of untreated sewage from pumping stations into local parks and rivers. “Marine life died, the creek water turned black, and the stench was unbearable for residents.”
- In 2016, a suspected Russian attack disrupted Ukraine’s electricity grid and cut power to about a fifth of Kiev in the midst of winter.
- There are also been numerous other attempts on water treatments plants, which have been unsuccessful in causing any harm.
In the UK’s Risk Register, two of the risks which have the combined likelihood and impact are “attacks on infrastructure” and “cyber-attacks.” I think we need to pay more attention to the risks outlined in this document, as most people ignored the risk with the highest likelihood and impact, a Global Pandemic!
What can you do to protect your organisation from cyber vulnerabilities and cyber attacks?
- If you have a telemetry system or machinery, whether it’s building management systems, manufacturing control systems or telemetry, have someone review their cyber security and check that they have appropriate firewalls and access to them is appropriately controlled.
- As part of your business continuity manual workarounds, check if your machinery can be operated manually. Norsk Hydro, a multinational aluminum producer suffered a ransomware attack in March 2019. All their production equipment had process controllers which were affected. They had to ask their retired production engineers and operators to come back and operate the plan, as they were the only ones who knew how to operate them manually.
- Review your vulnerability to water, wastewater and electricity loss. If there were widespread issues or systems failed, how would your organisation deal with them? If the attacks really wanted to cause issues, they could take days to recover and get them up and running again.
- Exercise some of these issues. They may not be the typical threats considered in exercises, but as per the Oldsmar attack, they are possible. This might be an opportunity to use some of the short exercises from Business Continuity Exercises: Quick Exercises to Validate Your Plan to explore some of the issues associated with them, rather than using the scenario of a larger exercise.
The water company I worked for had 5 million water and 8 million wastewater customers, so they had the money to spend on protecting themselves. In the USA, there are 70,000 separate water authorities and so they are unlikely to have the same level of funding to secure their systems and have plans in place for dealing with an incident. As this “loss of infrastructure” risk is on the UK’s risk register, then you should be doing something to protect your organisation.
Charlie Maclean-Bristol is the author of the new book, Business Continuity Exercises: Quick Exercises to Validate Your Plan
