For the love of the wee man
Why on earth are critical facilities like this on the internet at all? Previous incidents involved nuclear power and steel mills. Just stop - it's possible.
The sheriff of a small city in Florida warned on Monday that hackers had tried to poison its water. Pinellas County Sheriff Bob Gualtieri said Oldsmar's water treatment system, which serves roughly 15,000 people, was broken into by someone, via the internet, who had hoped to flood the supply with levels of sodium hydroxide …
Because ThoseInCharge want to show their friends pretty real-time graphs on their iFads.
Really. That is the only reason.
We tugged on their capes, and were shrugged off. We tapped 'em on the shoulder & were elbowed away. We pulled on their sleeves, and were thrust aside. Some even kissed their boots, and were trodden upon. Our message was always "Please, PLEASE, **PLEASE**!! don't allow the connection of SCADA to publicly available networking systems!"
But did they listen? No. They did not. The idiots.
On the bright side, those of us with a clue are making a pretty penny in our retirement, cleaning up the resulting mess :-)
Because ThoseInCharge want to show their friends pretty real-time graphs on their iFads.
'iFads' - heh.
Though a proper design would only allow remote control if you passed through multiple firewalls through multiple air-gapped [except for that one firewall] systems. I hope this is a wakeup call for SCADA in general.
Here's what might work, for emergencies:
a) ssh into a jail running on a FreeBSD box that's attached to teh intarwebs
b) ssh from that jail into the host box, which has access to the private network
c) ssh (via the private network) into another box that is multi-homed (but does not route) into the nearly air-gapped network
d) ssh into a box on the nearly-air-gapped network that has a command line interface (to perform somewhat cryptic commands using a custom interface) so you can "fix things" remotely.
A bit cumbersome, but for emergency use only. So, in this example, 4 ssh logins are required to get through, the first being a jailed system (FreeBSD jails have completely separate security contexts, and limit what root can do). Sane IT people would make the logins and pass phrases all different.
And so on.
(but anything significantly less secure than that, BAD idea)
The problem, at any rate, is NOT the pretty charts for "iFad"s. The problem is allowing COMMAND AND CONTROL via the same interface you use for the pretty charts.
That's a good question.
The answer is rather complicated, however... the short answer is that there should be an air gap, however if you wanted to have the ability to have a central control center and then remote access becomes an requirement. (Reduction in man power) Not to mention sensor data should be tracked back and stored.
This is an edge computing example.
The issue is more about the controls.
There should have been a limit as to how high a setting can go. So if someone fat fingers 100 instead of 10 where 10 is the max, it should reject the input and flag an alert.
From a design perspective... lots of issues jump out.
Whoever wrote this should be flogged.
There's a lot more I could say on this... but it would probably boar most of you...
Our small town of 1000 people solves this by having the water system totally air-gapped, off-line and controlled by a dedicated 24x7 team of on site engineers operating from an earthquake and forest fire proof bunker surrounded by anti-terrorist fences and armed security. Of course it does mean our water bill is about $1M/year
Actually I think there is no equipment connected to the internet because there is no equipment, I think Bill goes and tips a sack of chlorine into the tank on thursdays if he isn't busy.
We could do with a computer to detect the occasional dead beaver stuck in the inlet. Unfortunately Googling 'beaver stuffed into my pipe' got the last council fired.
"stuffed beaver"? Let me help you with that google that for you...
The particular issue with water delivery is that it tends to collect in difficult to get at places, like aquifers, rivers and valleys, and be needed in other places, like towns and cities. In order to measure the flow rate of an isolated highland stream or river, a remote sensor is used, often with a solar panel and satellite link to send the data to the central management system. That will usually happen over the cheapest form of communications available - the Internet.
Where there are multiple sites managing water and sewerage systems, a communications network is inevitable, and the good old Internet is there. The issue is not why it was connected to the Internet, but why the connections were not secured using encrypted VPN technology.
Why on earth are critical facilities like this on the internet at all?
Right, the critical control systems should be on private networks that cannot be reached from the internet.
And to make sure personnel can be alerted when there's a problem with the system, there should be a SolarWinds instances that can access the internet as well as the private, critical, sensitive network for proper monitoring and alerting.
Problem solved.
the hacker increased sodium hydroxide from 100 parts per million to 11,100 parts per million and then left
Equally as concerning as the internet exposure is that the plant-control software accepted a dangerously high input level. Is there no range checking for safe levels?
Not entirely convinced that "their system works", given that the reason they detected it this time was that a local user happened to be watching at the time.
How long would the overdose have to keep running before they could no longer dilute it again without closing the plant temporarily?
If they hadn't watched it happen, would they have actually noticed within that time period?
On the other hand, is the dosing tank large enough to cause a significant problem if totally emptied out?
Possibly - there was an incident at the Camelford water treatment plant in the UK that accidentally poisoned a load of people when aluminium sulphate was unloaded into the wrong storage vessel. In that case, there would certainly have been enough in the (intended) tank to cause a problem.
https://www.bbc.co.uk/news/uk-england-cornwall-44727036
Ah the case where the South West Water Authority largely escaped liability by being privatised into South West Water plc with the same management?
The delightful Michael Howard who was in charge of the privatisation said that a prosecution of SWWA would "be totally unhelpful to privatisation... and render the whole of the water industry unattractive to the City".
An ex girlfriend long ago had a summer job at a water plant after she graduated from college (with a degree in Chemistry) before entering grad school.
Basically her job was to watch the control panel for any alarms and directly test the water leaving the plant and heading for the water tower for stuff like pH once an hour. Her plant (in a fairly small town probably comparable to the one in this story) would have caught this easily unless they've automated away the hourly direct testing and the system doing that testing would be accessible remotely.
So the local user had opened Team Viewer? For why if they didn't expect remote control to take over?
Or TV is on all the time in which case what was the on-site user looking at if they weren't expecting anything. If they were doing something then the naughty remote person would have quietly tiptoed away until after the end of shift.
It's a long time since I used TV, but doesn't it assume all the current user's privileges? Good for lots of purposes, but surely in a case like this you want a VPN with logs and properly restricted user rights.
If the naughty remote person was a serious hacker won't they have fiddled with the system to give them TV starting say on Monday nights at 9:30? or some other back door? The tone of the statement is that eagle-eyed staff spotted a trivial attempt so 'nothing to see here'. Good for the head's-up, well done for admitting the security failure, well done for having safety nets, but clearly there's other stuff going on and I guess no attempt at catching the NRP.
surely in a case like this you want a VPN with logs and properly restricted user rights
Teamviewer: can be installed, configured (badly) and run by someone with half a brain.
VPN, proper .. user rights: needs some technical skill, and was probably beyond the capability of the plant-owner's nephew.
TV runs over the Internet and there was a known 0-day that got patched a while back that all devices registered in the cloud portal could have had their access details compromised...
We still use it with the local management tool. The local clients are configured, so we can't gain access to the machines without the user of the machine allowing TV to connect or the machine is locked, in which case you need a valid local/domain username and password to get into the machine.
For remote user support, you need something like this. But our production facilities are on a separate network segment and can't be accessed from the Internet and TeamViewer doesn't work on them. Those machines still need a physical visit.
Could be a "work from home because Covid" deal. I'm almost surprised I haven't heard more stories of companies getting compromised because they implemented something stupid and insecure in their rush to quickly deploy a system to let people work from home.
I mean it's not really an excuse, but if that's what happened it doesn't surprise me at all.
Honestly the sludge mixer being capable of speeds hundreds of times over normal makes as much sense to me as the sodium hydroxide concentration setting going hundreds of times above normal.
More seriously if a mixer could be stopped and started quickly maybe it's possible to move the sludge at it's resonant frequency and achieve the desired shitstorm that way?
You would assume the dosing was written for varying volumes of water. It isn't unreasonable to assume that there could be a system elsewhere which has 25 times the volume of water and requires 4 times the dose/litre due to the PH.
The limitation would surely be on the dosing tank. Although you'd also assume there would be PH (and other?) sensors near the outlet.
Sure levels will be different spending on the water volume, but the software listed levels in terms of ppm not volume.
The problem with the software seems to be that it allows levels of 11,100 ppm. Is there really water out there so acidic it could ever need that much? Perhaps so, but it would always be getting a lot of lye to get to a desired pH level.
It sure seems like the software should refuse to allow anything like a 100x change. Maybe allow a small divergence like from 100 ppm to 200 ppm, but require special access or password for changes beyond that - or better yet using some type of hardware interlock to unblock larger changes.
I think the probable result, if it hadn't been caught would be the entire stock of NaOH being emptied into the supply, and swiftly running out. Because it's nasty corrosive stuff, they're not going to be having a huge mountain of the stuff, and they'll probably only have enough for a week or month at a time. The mechanism for pumping it into the water supply probably wouldn't have been capable of pumping 10,000 at times the normal rate, and I'd be amazed if there aren't also sensors to check the resulting pH downstream, with alarms and cut-offs. It's not like such systems could be prone to failure of a more prosaic nature (such as mechanical), and would therefore have fail-safes.
The net result, if it hadn't been noticed would probably have been the dumping of a few hundred litres of NaOH solution into a processing tank, and then the pump shutting off as it tries to pump air, followed by another pump shutting off when that gets a short distance downstream, a cut-over to a secondary system and a very brief dip in the town's water pressure.
The biggest risk would probably be from the fact that diluting NaOH is exothermic, and thus you'd potentially end up with a whole load of dilute warm NaOH to dispose of, and a load of processing infrastructure that needs a good scrub down before it can be brought back online.
Ahh, another shit scholar! I've been trying to find the resonant frequency of shit for years now! I came up with this equation but it's not solving right just yet—maybe you can take a look?
f = ((Bx+2² × 2OL²) + O2C) / K - S
I know it's stained, but those stains are testament to my research.
That wouldn't happen, past a certain point of increase in revs, a cavitation would form. Sucking air (or rather vacuum) into the pump restricting its flow as well as damaging the pump. Most likely it wouldn't be able to sustain the speed for long before thermal controls would shut the pump down, so most likely just causing an overflow event instead.
(I used to help a water modelling company with their scale testing reports in the days before fluid dynamics where computable to a practical level).
I know a lot of industrial systems run Windows in some form or another, its because the foundation system that's customized for the particular plant or applciation was built on Windws "becuase it was the best platform avaialable (back in 1992)". Its tolerable but only if you realize that these are not office systems running Office but special purpose systems that won't have frequently patched versions of Windows. Management doesn't understand this -- I long gave up trying to explain why it was necessary to keep an old box with XP on it (specially patched drivers and kernel from manufacturer needed to support weird bit of hardware -- could upgrade, I daresay, but at a huge cost of $$$$$ and learning curve (also $$$$$)).
These systems should never be connected to the Internet. If someone desperately wants to see what they're doing get them a webcam.
I haven't given you a downvote, simply because of the ending, but the rest of your post us pointless and irrelevant Windows bashing.
If you had actually READ the article, you would of realised it was accessed by an insecure Teamviewer session.
The same would of happened if it had been Linux, BSD, OSX or Unix. An insecure remote access connection is an insecure remote access connection. End of.
But the point is the same, such PCs have no business anywhere near the Internet.
We still have analysis equipment that is run by Windows XP, same for production kit. Heck, we even have a metal and plastic printer, for printing industrial signs, that requires MS-DOS and an RS232 port!
But none of it is allowed anywhere near the Internet.
An IT manager I know runs a CNC machine, which requires Windows XP. The manufacturer of the CNC machine has no upgrade software for the machine (you have to buy a new CNC machine for high 6-figures to get software compatible with modern Windows). When they need support, the manufacturer tells them to give them the TeamViewer number for the machine. After quietly explaining that XP is not going anywhere near the Internet, she then tells them they will have to "remote control" the machine operator to solve the problem. If they want to use TeamViewer, they will have to provide her with a modern, secure version of the software.
At another company I know, they were still selling an ERP system in 2015 that was based on SUSE Enterprise Linux from 2000! They only, finally, got around to upgrading to a more modern OS because they couldn't get any RAID cards anymore that would work with the old Kernel! And that was connected to the Internet at their customer's sites and was remote managed by the company.
"But the point is the same, such PCs have no business anywhere near the Internet."
Actually they do.
Think edge computing.
Your actual controller is remote and you want to control multiple water facilities from a central location. Or your facilities are remote in a area w no internet/telco and you need to use cell / sat for communication.
Note that since there isn't really a lot of data going between the remote device and the central controller, you could use data on 3G networks and above. (You could use point to point or go across a secured linked on the internet.)
There are a couple of designs where going over the internet will work however it would be the exception than the norm.
The issue is the controls on the PC and how the PC interacts with the controller.
While this hacker jumped up the amount of chemicals to be added... it would be the same as if someone fat fingered it. The underlying system should have rejected the input.
There's more but without know more about the systems. its pure speculation.
Before Windows, a number of these systems used DOS and QuickBASIC because it was much, much, cheaper than the dedicated units or minicomputers that preceded them. The good news was that QuickBASIC was quite good at opening a serial line, sending or receiving data, closing the line and crunching the received data. The really good news was that they weren’t connected to the internet...
30 yrs ago, I wrote the OS for an embedded controller that controlled the amount of chlorine used to treat drinking water. It was for a 6809 embedded controller.
No windows here.
Later, I found out that another group wrote some software which captured the sensor readings to be stored in an oracle database. This is where my controller was connected to the 'net.
The safe thing is that in order to modify the system, you had to download a net new program to be run.
(I wrote the OS and the first gen of the controlling software.) While my OS would allow for sensor readings to happen ~200 times a second, the sensors actually tested the water once every 3 minutes.
Also the rate of change was controlled so that you weren't constantly adding chlorine or water to maintain a steady state.
Windows was never the OS or involved unless it interacted w the Oracle database.
While this brings back memories... I can think of half a dozen ways this could have been prevented even if the hack was successful.
30 years ago I worked for a water company, quite possibly using the controllers for which you wrote the software. Plants were networked and remote monitored in those days, but the connections were all "private circuits" rented from the Telco. Point to point, using proprietary modems, from PLC to PLC or similar. Yes, there was no security or encryption, but unless someone opened up the green cabinet in the street or climbed the telephone pole to physically tap into the wires, then decode the protocol, they were pretty secure. At that time there were no Windows anywhere near operational kit - stuff ran on dedicated hardware and or control centre SCADA stuff ran on Vaxes and the like.
Then windows made everything cheaper..... No need for dedicated and expensive hardware, do it in software and the only thing required is a dedicated ISA/PCI interface card to connect to the outside world. Then windows and PC's make it cheap and easy to network things. Then the internet arrives to make it even easier and cheaper..... you get the picture. The skill set required to run it all and connect it up gets lower.....
As mentioned above Teamviewer can be installed by anyone on any PC without any specialist knowledge or understanding of security or attack vectors. It must be secure, they argued, it needs a password to access it, sort of thing.
Private circuits are now becoming a thing of the past, as ubiquitous and cheap internet connections take over the world, exposing more and more to the big bad internet. Yes, these should all be on VPN's with access logs, but again, as other posters comment here, some of this plant has a functional life of 20-40 years, and will generally need to run with whatever it was supplied with for that time. At one time when service was required an expensive engineer would arrive in his car to do maintenance, now someone would expect the same maintenance to be done remotely from halfway around the world, at significantly reduced cost.
Oh, and those spinning things that spread the sh!t around the filter bed - they are almost always driven by gravity feed water pressure, no motor or pumps required, so no chance of flinging it in a 10km radius.
You know what this means of course "apparently: an operator logging into a PC at the facility early on Friday morning said he had noticed the system had been accessed but had assumed it was his supervisor and thought nothing of it. "
It means when he logged in it said "thank you for using the free sponsored version of teamviewer"....
'Lye' doesn't specifically mean sodium hydroxide - it can also refer to potassium hydroxide. These days, though it usually is NaOH it is used for. It used to be used in soap-making, and it didn't matter whether it was NaOH or KOH (or a mixture, given how it used to be made).
It's an old-fashioned word in British English. It isn't obsolete, but it is uncommon.
Incidentally, kudos to the operator who spotted the problem (taking the story at face value, of course). Whatever the shortcomings of the system, at least he caught it.
It will be a word familiar to anyone who has read the Gashlycrumb Tinies.
You're correct in thinking it's not in common use in the UK any more. It's more commonly known as caustic soda, or occasionally sodium hydroxide. I think lye is no longer in common use, as people don't tend to use wood ash to make their own soap from tallow any more. You might still find the word in use in some industries such as tanning or soap making, as a technical term, but you'd have to ask them...
In my time, I have been involved in soap making on an industrial scale. We always used pure sodium hydroxide (it was cheaper than potassium hydroxide).
Interestingly, the lye obtained from wood ash would have been a mixture of various compounds depending on the composition of the initial wood. There's typically about ten times more potassium in the ash than sodium and other metal ions, so the lye produced would be a combination of hydroxides, though mainly KOH.
This post has been deleted by its author
My bet is on a script kiddie somewhere. This was not any sort of true hacker, otherwise they wouldnt have been doing this during work hours and in such a way as to allow one of the users to see their actions so easily.
I'm reminded of the scene in Hackers (that good ol' 90's movie), where one of the "kackers" is saying how they got on a server, no idea what it was, started throwing commands at it, and then the next day it was all over the news that some ATM's started throwing cash into the street.
It certainly has that sort of feel to it...
Idiots! When will entities with critical, do-or-die data or equipment realise that the internet is NOT safe. If they need remote access they should spring the dime and set up dedicated lines. You know. Those things that ran the whole country for decades before DARPANet was realsed into the wild. At least until they either incorporate at least 1024 bit security or completely remove the biggest threat to their computers - humans.
One of the first hacks of 'internet of things' was the Maroochy Shire sewage spill in Queensland where an employee of the equipment supplier hacked a wireless system used to actuate valves, discharging millions of litres of untreated sewage into local water systems. That was back in 2000 - and it looks like we're still connecting everything for [reasons].
A certain website covered it back in the mists of time:
https://www.theregister.com/2001/10/31/hacker_jailed_for_revenge_sewage/
Strangely, the hack is described in terms of ppm of Caustic Soda. Dosing is usually via a dosing pump (variable speed and stroke) and would be set by adjusting a pH target. It is not possible to dose Caustic Soda directly - it consists of very hygroscopic pearls or flakes.
It is probably possible to set up such a system to work with ppm, by calculating ppm from the water flow rare and the dosig rate but who on earth would bother? !!
In any sanely engineered system you would set the target pH ( usually 7.4 to 7.8) and let the pump and pH meter sort things out. That's maybe 10 lines of code. Then you add another 100 or so to cope with all the error conditions you can think up but essentially if pH drops below 7 or rises above 8 the plant should stop and raise an alarm. The operator should not be able to set insane dosing values.
I've done, or had done for me, several such mini plants dosing caustic, acid or sodium hypochlorite.
Forget the plant security, Windows or Team Viewer. The adjustment should simply not have been possible.
The UI may have allowed the entry of those numbers, but I seriously doubt the dosing system would get anywhere near that before it topped out. I also doubt the feedstock would last very long if it could. After all, if they need to add a small amount.
Assuming the 100ppm is by molar amount, not weight, that works out as about 45g per tonne of water (1,000 L, or one cubic metre) - Lets say this is a processing plant for 10,000 people (apparently the population of Oldsmar is about 14,000), and the average US person apparently uses 0.4 m3 of water a day, so the daily usage for that level of water treatment would be 180kg.
Up that by a factor of 1000, and that's 180 tonnes. Let's surmise that this starts with a hopper containing pelleted NaOH, which might contain 1 tonne, or enough for about 5 1/2 days. It's presumably someone's job to keep that topped up. I'd guess at the start of their shift, they check the level and top it up. It's going to empty well before the concentration gets anywhere near 100,000 ppm, and no doubt there would be alarms going off everywhere, from pH monitors, feedstock level indicators, flow measurement, et al.
> Why did these hackers need to poison the water in the whole city, it's scary and strange.
That's a good question. Normally the US poisons its own city water supplies without recourse to hackers.
Who recalls the story posted on thier very website of a recently-fired employee who managed to use his remote access to direct raw sewage into the water supply?
Western Australia about a decade ago if I recall correctly...
(But.... "Terrorists" and "foregn hackers" makes for better headlines when Muricans do it)