back to article Water utility hackers destroy pump, expert says

Hackers destroyed a pump used by a US water utility after gaining unauthorized access to the industrial control system it used to operate its machinery, a computer security expert said. Joe Weiss, a managing partner for Applied Control Solutions, said the breach was most likely performed after the attackers hacked into the …


This topic is closed for new posts.
  1. Mike 140


    YTF was a water company's control system attached to the net?

    1. Mark 65

      Exactly. It is not the job of the US Government to protect this stuff it is the job of the utilities to not connecting to the rest of the World so we can all have a crack at it. Seriously, these SCADA systems are connected to the internet through shear fucking laziness so these wankers don't need to get off of their fat arses in their comfy offices in order to tweak shit. Well that's a mighty high price to pay if you ask me.

      1. Black Betty

        Not lazy. Cheap.

        At the very least, a manually operated system reuqires a nearby "on call" operator. More likely a permannent on site operator, who would spend most of his time with his heels up waiting for something to happen.

        Truly dedicated communications infrastructure is also prohibitively expensive.

        The true problem is piggybacking something as bloody simple as industrial control systems on top of any complex operating system. Water pumps just plain don't need to be able to run a word processor, a database and 50 active porn windows.

        1. James Micallef Silver badge

          Yep, the real issue is about cost-cutting. They could make this thing secure* by having a dedicated WAN or a VPN instead of using the open web. That would cost a lot more, so they cut corners.

          *or as close to secure as is possible

          1. Black Betty

            Still too effing complex. Too stupid to be hacked...

   the only way to go.

            1. Hardwired (not coded) refusal to exceed "normal operational parameters" by more than x%.

            2. A very limited command set.

            3. Chained encryption of commands and responses with a null operation failure mode, and a hardwired restart sequence.

          2. Vic

            > VPN instead of using the open web. That would cost a lot more

            A VPN would cost *pennies*

            They really are very cheap. But you need a PHB who will actually *listen* to such proposals, rather than just ignoring everything he doesn't understand...


          3. Drew V.

            Yes. Public infrastructure in the US is almost by definition old, decaying, underfunded, understaffed and neglected, or outsourced to private companies where all of the above applies. One obvious symptom is how the American electrical grid from time to time suffers enormous blackouts that should be easily prevented. The real surprise to me is that things like these don't happen more often.

        2. James O'Brien


          " Water pumps just plain don't need to be able to run a word processor, a database and 50 active porn windows."

          Sure they do. So the operator can draft his resum^H^H^H^H^H CV as outlined by some twat named Dominic Conner from The Register, the database to index the pirated music and movies because you can never be too organized and the porn windows for the down time.....

      2. Stuart Castle Silver badge

        When does the government need to get involved?

        So far, we've had hackers able to disrupt Nuclear power stations, open death row and now destroy water pumps.

        In short, everything a foreign power would need to do to ensure that the country was, in effect, disabled.

        Before you say, that the armed forces would still be available, they should be. However, I suspect they'd be busy either distributing the remaining supplies to the population, as well as attempting to control uprisings.

        1. Anonymous Coward
          Anonymous Coward

          wait till we get smart meters!

      3. Tom 13

        It's not up to the government to protect or fix it,

        but under the umbrella of ensuring homeland security, it is a responsibility of the DHS to notify critical infrastructure stakeholders whenever and wherever a system vulnerability has been discovered.

        I'm not discounting the lazy/cheap factor for the utilities, I just don't want DHS's lazy/cheap factor disregarded either. Particularly since the DHS statement about not being aware of any such vulnerability further enables the lazy/cheap utility.

  2. Anonymous Coward
    Anonymous Coward

    Firewall? Guys!

    So, these guys that have water pumps connected to the Internet doesn't know what a firewall is?

    Surely anyone worth their salt would have systems screwed down - with a basic IP access control list if nothing else - to accept connections only from other authorised IP ranges elsewhere within the organisation?



    1. Martin

      But a firewall is still a vulnerability....

      It's a water pump for a water system. The water pump and everything it needs to be connected to should be on a completely separate, unconnected to the internet, network.

      1. GitMeMyShootinIrons

        Whatever happened to...

        Good old fashioned point-to-point modems? Perhaps with a secured PC on the end to SSH onto the controller. No new-fangled Internet to worry about.

        I doubt bandwidth is needed in this application - its a pump - not exactly streaming HD porn....

        1. JGT

          You'd be surprised

          Over a decade and a half ago I took a short flight in the LA basin and sat next to a water district employee. We ended up talking computers communications and he mentioned that a city's water system used microwave links for command and control. I asked what kind of encryption they used. He was confused, wanting to know why they should be encrypting the link, it was only doing water system stuff. I mentioned how it wouldn't be hard to override the HQ signal with a correctly oriented and stronger signal. He asked why they should be concerned. As this was before 9/11, I struggled to find a convincing reason and came up with, what if they override the pump controls and over pressurize the system?

          He was shocked at the idea.

          Seems I wasn't too far off the mark.

      2. M7S

        @ Martin, see Black Betty's comment above

        I've a small unmanned pumping station near me, but over a mile from any other significant comms infrastructure. I suspect its not the only one like this and I'm not far outside the M25. I would expect distances in the USA to be even greater so connecting to the "public" net is the only viable option, not that this would excuse any other security failings, but I dont have enough information on that to feel able to comment reasonably.

    2. alain williams Silver badge

      Firewall is not the whole solution

      The chances are that some malware found its way onto a PC used by one of the maintainance engineers and the attack was launched from there. This would have been deemed 'safe' by any firewall.

      This is as much a MS Windows problem as a SCADA problem.

    3. Tom 13

      Grendel, Martin, Hello? Anybody Home?!

      Stuxnet was designed specifically to jump the air gap for the ideal system you describe. So Firewalls and air gaps are irrelevant. And the new malware is just a variant using a different hole.

  3. Anonymous Coward
    Anonymous Coward

    So unnamed ebil actors from far, far away gone and done did...

    ... destroyed the thing by turning it off and on again?

    Some IT "expert", this.

    Then again this is the land of dumping a lake worth of drinking water because one drunk yoof peed in it. Yet they couldn't be arsed to put the simplest of fences around it. It was also right open to the elements. Acid rain and all that. You know, acid? And what about the birds? Anybody think of that, huh? And the fish? They fscking fsck in it! And... oh dear I just realised yoof will have been drunk on American Beer[tm]. I can see they feel there's a problem right there.

    1. RocketBook

      American Beer

      Of course they could have just sold the water direct to the breweries and they could have bottled it without further processing.

    2. Paul_Murphy

      You forgot the homeopathic problem.

      2 million fish making their messes in a reservoir? no problem, ditto with flocks of 'bird flu' victims crapping all over it.

      One person's pee? I'm surprised there wasn't some sort of homeopathic event horizon thing going on..


      1. Ru


        Ach, beat me to it. Cursed moderation delays.

    3. Ru

      One word for you:


      Dilution a billionfold will not save you from the wee.

      1. Anonymous Coward
        Anonymous Coward

        re homeopathic dilution

        the magick ingredients that the homeopaths use, do they rapidly breed and multiply like some of the organic stuff in the snow-yellowing fluid?

        just asking, like....

        1. Anonymous Coward
          Anonymous Coward


          Homeopathic ingredients grow by getting fewer and fewer.

          Anyway, the moral of this story is that computers can automate everything, including attacks.

    4. I ain't Spartacus Gold badge

      Some IT expert...?

      Yes you can destroy a pump by turning it off and on again. I'm sure he's capable of talking to a pump expert, and finding out these things.

      I can think of many easy ways to break a pump from a remote controller.

      Pump motors are designed not to be started too often. Usually they're set up not to be able to start more than 12 times an hour. This is either enforced in software, or often a run-on timer that runs the motor for a set time. Thus, after a start, the pump will idle until it's needed again, or the timer runs out. Bypassing these controls, and constantly re-starting the motors would break them.

      If you could get sufficient control of a water supplier's systems you could cause some interesting explosions in the pipework. Water is heavy, and under tens of atmospheres of pressure you can do some serious damage with it. They're possibly lucky they're only having to replace a burnt-out pump motor.

      1. Anonymous Coward
        Anonymous Coward

        "Yes you can destroy a pump by turning it off and on again."

        Sure. But I was talking about IT "experts", not pump experts. And that makes it (a poorly executed, I freely admit, but still) a joke of sorts.

        On a more serious point, there probably ought to have been interlocks preventing software from destroying hardware, and if the lesson is that this requires hardware interlocks that you can't override in software, well, then maybe we should require exactly that. In law if we have to.

        This, or even stuxnet, isn't quite the first time software deliberately and spectacularly broke hardware. Is it really too much to ask of experts in this field to know this and learn from lessons past?

  4. Destroy All Monsters Silver badge

    "What the hell is going on with DHS? Why aren't people being notified?"

    Someone is being stupid here. And it isn't fat-assed bureaucrats safely encysted in the inner belly of an *enormous* dinosaur, blowing tens of billions and busy seizing spanish soccer websites (diversification into IP problems is always good), setting drones on Mexicans trying to cross the southern rabbit fence, using government credit cards to build a home brewery, checking out the 99% or warning about the mighty danger of poisoned buffets.

    Did I mention that DHS was involved in blowing a whole New Orleans?

    Really, we need a FAIL + NUKE + TRASH icon, all mashed together into a single gigantic clusterfuck. Probably a TAX icon, too.

    1. Pete 8
      Thumb Up

      You Sir, are a very cunning linguist.

  5. Paul Crawford Silver badge


    "raised serious concerns about the ability of the US government to secure critical infrastructure"

    It is not there job to do so, it is the water company.

    But maybe if the US gov made the CEO & MD of such corporations liable for gaol time for allowing such a serious breach of good practice, i.e. putting critical infrastructure on the 'net WITHOUT the software suppliers (MS et al) backing that up with a matching warranty of fitness for purpose, might just help to get such things fixed though.

    1. Pete 8

      Toilet pwnage!

      "But maybe if the US gov made the CEO & MD of such corporations liable for gaol time for allowing such a serious breach of good practice, i.e. putting critical infrastructure on the 'net WITHOUT the software suppliers (MS et al) backing that up with a matching warranty of fitness for purpose, might just help to get such things fixed though."

      How does a government or a court imprison itself?

      1. Anonymous Coward
        Anonymous Coward

        Meaningful fines would go a long way, perhaps something like 10% to 50% of sales for the year -- or more.

        1. Tom 13

          There is NO SUCH THING AS A MEANINGFUL FINE on a PUBLIC Utility.

          The government guarantees a certain level of profitability for them to operate the utility, and thus the costs of the fines always gets passed to the consumer.


      2. Dodgy Geezer Silver badge

        How does a government or a court imprison itself?

        The same way they always do.

        Set up a commission of enquiry, find that they have been a naughty boy, and promise faithfully never to do it again. Ever.

        Then award themselves more taxpayers money to expand their empire.......

        I wish that, when a shortcoming is found in government work, people wouldn't be quite so free with calls for increased money to be spent in that area. It's a recipe for shortcomings to be found everywhere.....

      3. Paul Crawford Silver badge

        @Toilet pwnage!

        I had assumed that most US utilities were private companies doing the gov work. Even so, you find those who made the decisions and work up to the top, as you can still gaol government or court employees:

        Why was it on the net? Ah, probably to save money.

        Were the risks considered? Probably not, or ignored to save money.

        Who ultimately took the decision (or applied budgetary pressure) that traded-off safety for running cost, and was that an acceptable risk or one that represents criminal negligence? If is was a windows-based box with hard-coded passwords, then negligence is the only answer.

    2. Voland's right hand Silver badge

      Not quite

      You will find that in order to operate bits of critical national infrastructure like water, sewerage, leccy, gas, etc you need to do comply with some reqs. So in fact, the CEO and MD are liable for at least something as they are in breach of their regulatory regime. Similarly, even in the USA the government has quite enough leverage to make such companies do things.

      In any case, this just goes to confirm something I have been saying for ages - SCADA security is sh*t. The scariest bit is that the same companies and people who write scada now write smart metering software. So a system with the same lousy level of security as the one on that pump (or worse) will be in every house in a few years in control of leccy, gas and water.

    3. Anonymous Coward
      Anonymous Coward

      But maybe if the US gov made the CEO & MD of such corporations liable for gaol time for allowing such a serious breach of good practice, i.e. putting critical infrastructure on the 'net WITHOUT the software suppliers (MS et al) backing that up with a matching warranty of fitness for purpose, might just help to get such things fixed though.

      Can we just get real.How serious was this? Everyone makes mistakes, anyone who works in software or IT shoudl be very conscious of that. The level of checking, redundancy and analysis of a system should depend on its criticality and impact on safety. If everything has to be checked to the level that a safety related system has to be checked everything would grind to a halt. There is no indication that this pump was a safety critical device. If every time any employee in a company did something which was not good practice whether the system concerned was important or not then every CEO and MD would be in jail. There already (at least in the UK) neligence laws about failing to take precasutions when dealing with safety related systems.Company directors are personally responsible in these cases.

      It may even be that at the current time putting such a system on the net without VPNs etc wasthe correct decision. If the system is non-critical, and if there is a substantial saving in time and effort from remote servicing, and if the additional cost of securing it outweighs the risk of an attack, then actually the appropriate things is to connect it to the internet. The problem with risk is we never know for certain.This is deliberate critminal damage to the pumop and it is very difficult to see any benefit to the perpetrator so why the F**** did they do it. WIth all the publicity I am sure that the risk benefits are now heavily skewed so that all similar devices for that company must be secured.

      1. Paul Crawford Silver badge

        @AC 12:16

        "Can we just get real.How serious was this? Everyone makes mistakes, anyone who works in software or IT should be very conscious of that."

        You are right to a point, in this case no serious damage was caused to the population, etc. However, we are in 2011 and the vulnerability of computer systems, in particular anything using Windows, has been amply demonstrated for all of the last decade.

        What this incident shows is a system that might have been fine off-line, without a half billion PCs potentially able to probe it, but clearly was not good enough. With a bit more effort & synchronisation perhaps a determined perpetrator could have wreaked havoc on most of the pups in a region, leading to the possibility of death or injury from disease or dehydration caused by a failure of such a fundamental human need: fresh water.

        My point comes down to poking those in charge with a big pointy legal stick (not unnecessary prosecution of genuine mistakes) so that changes are made, and stupidly vulnerable systems (think Siemens and their SCADA's hard-coded passwords) are kept well detached from the internet in the future.

        "This is deliberate critminal damage to the pumop and it is very difficult to see any benefit to the perpetrator so why the F**** did they do it."

        Two possible answers spring to mind:

        1) There is no reason. Just done for idle amusement.

        2) Practice for a cyber-attack or a blackmail attempt.

        1. Paul Crawford Silver badge


          Seems we both struggle with 'pumps'

        2. Ian Stephenson
          Thumb Up

          @Paul Crawford

          3) Proof of ability - You can see we can do it, just deposit $1million in this numbered account along with the name of your target.....

      2. Tom 13

        Sorry, it really isn't difficult for me

        to imagine this being leveraged to a major incident with catastrophic consequences. I just don't see need to put fuel on the fire by publishing it.

      3. TheOtherHobbbes


        "If every time any employee in a company did something which was not good practice whether the system concerned was important or not then every CEO and MD would be in jail."

        You say that like it would be a bad thing.

  6. Anonymous Coward


    Even people with tiny recording studios know not to put their critical machines on the net.

  7. Wombling_Free

    Yes, but...

    "Even people with tiny recording studios know not to put their critical machines on the net."

    Yes. WE know that.

    Unfortunately the 1% 36sqm walk-in-robe owning senior management haven't got a clue beyond what wine they will drink for dinner tonight. They expect the untermenschen lot will take care of this, and correspondingly cut the middle-mungers budgets, who then immediately outsource to China, India or Smellistanumboto.

    Then everyone wonders why their city water supply is run by a PIC16F84A with a USB interface nailed to a Nokia 8210.

    1. Blofeld's Cat

      "Then everyone wonders why their city water supply is run by a PIC16F84A with a USB interface nailed to a Nokia 8210."

      I suspect that such an arrangement would be more secure than some of the real solutions out there.

    2. Anonymous Coward
      Anonymous Coward

      ya know, that 'F84/8210 mashup is probably more secure....

  8. Anonymous Coward
    Anonymous Coward

    Siemens again?

    Or someone else?

  9. Anonymous Coward
    Anonymous Coward

    Hindsight is 20 20

    It's easy to criticise the people who set this up; but the reality is that these systems started out as purely internal networks of RS485 connections, with external access for engineers provided by modems. These have grown in complexity over decades, and probably got connected to the internet long before it became apparent that this would be a threat of any kind.

    You can't just insist that systems aren't accessible from the Internet either; there are good reasons, in a country the size of the USA, to ensure that experts can access systems remotely. The challenge is to allow remote access only to authorised persons.

    Like a lot of IT specialities, security is coming to the SCADA game a long time after it started; experts in SCADA aren't necessarily experts in security, it is IT management's job to link the two groups together to solve the problem.

    1. Anonymous Coward
      Anonymous Coward

      I just don't buy that argument

      It's been pointed out for years that nobody in charge put half a thought to securing their systems yet already had them hooked to various private networks, dialin modems (wardialing, anyone?), whatever, and then proceeded to hook them up to the wider internet. Meaning that they did have externally accessible systems and just didn't stop and think what making them more and much easier accessible would mean.

      That is deliberately ignoring the consequences of your actions.

      And for critical infrastructure that's pretty much inexcusable. We knew the internet is not a safe place; that's been painfully apparent for decades now. Saying "we didn't know" should count as criminal negligence. Having been alerted years ago to possible impending doom but wilfully ignoring it, wondering why the shit hits the fan now is not hindsight at all. It's being too bloody late reading the signs on the wall.

  10. Anonymous Coward
    Anonymous Coward


    @Wombling_Free You owe me a new keyboard.

    Memo to self, NEVER drink coffee, especially hot coffee when reading El Reg, I had hot coffee coming out of my nose! OW!

    BTW is it true that Nokia 3310's are really used to control stuff? I always wondered why the price on Greedbay seems to be so high.

    -AC/DC 6EQUJ5

  11. JDX Gold badge

    Stole passwords

    So they stored them in plaintext?

  12. Anonymous Coward
    Anonymous Coward

    Did I miss something?

    Joe Weiss is complaining that the DHS didn't warn him? Does he read the news? Even if you have been wiring your hardware into the interspazz for some crazy reason, there've been ample hints that, as a strategy, it needs a rethink...

  13. Anonymous Coward
    Anonymous Coward

    DHS priorities

    Well as long as they keep giving people a really hard time at airports (particularly Russians it seems!) I'm sure it'll all work out OK in the end.

  14. Dr Patrick J R Harkin

    I don't suppose there's any hope...

    ...that the X-Factor broadcast system is SCADA controlled?

    1. Anonymous Coward
      Anonymous Coward

      They pulled off something of that nature in "Hackers", at least.

  15. Nigel 11

    Hard-wired safety limiters needed?

    Surely critical infrastructure ought to be designed so that there are limiter systems which cannot be over-ridden by any computer? (Or indeed, by human operators following any procedure short of using screwdrivers and wirecutters).

    I once worked at a synchrotron light facility. I'll spare you most of the details, but the light is generated by relativistic electrons circulating in an ultra-hard vacuum, and the various experiments took light, X-rays, etc. out through "beam lines" which at one end were open to that ultra-hard vacuum. If any air ever got into a beam line, a series of vacuum sensors had to cause valves to slam shut faster than air could travel down the line. If that ever failed there could be expensive damage to repair and days, weeks or even months of down-time while the vacuum was re-established.

    What controlled this emergency safety system? Hard-wired relay logic (with its power-fail fail-safe). No digital decision-making. This system could not and should not be overridden. If it tripped when it shouldn't have, it was an annoyance. The converse, a disaster. Relays are fast enough, fail safe on power failure, and have extremely high noise immunity so they never "glitch". The right technology for the job!

    One line was operated by a big computer company who refused to use relays. They had a special multiple-redundant computer system doing the job. They claimed it couldn't fail. One day, it did: spectacularly so. The facility was down for weeks and the big computer company was on the hook for all the bills. Hubris and Nemesis at its best!

    1. Anonymous Coward
      Anonymous Coward

      This kind of thinking, good sir, is why the nice people regulating the late and overbudget EPR nuclear power station at Olkiluoto wanted separate computer systems for normal operation and for safety shutdown (unfortunately even the safety shutdown is too complicated to be hard wired).

      Last I heard, they'd been over-ruled by commercial interests who wanted a combined control and safety system.

  16. Anonymous Coward
    Anonymous Coward

    re "don't put it on the net"

    Follow Stuxnet much? If you didn't, think about shutting up, please.

    You don't need a Net connection (firewalled or not) to get caught out. You may not even need infested USB sticks and Sneakernet, though obviously they do simplify propagation.

    But it probably makes exploits easier if the PLC programming tools (which run on on a box which necessarily sometimes IS connected to the Net and sometimes IS connected to the "secure" automation LAN) run on a historically hopelessly insecure OS.

    It probably also makes exploits easier if the PLC programming tools and user applications are *required* to use widely documented default passwords for authentication.

    But no, blame it on being connected to the Net. It's so much easier than having a clue.

    1. I ain't Spartacus Gold badge

      Up to a point...

      >>>But no, blame it on being connected to the Net. It's so much easier than having a clue.

      Well, yes and no. Sure, you can get an infection without a net connection. But an attacker can't control in real-time if there's no net access. So unless they know what hardware they're hacking in advance, they can't easily code it to give commands that will be universally damaging.

      Therefore if you're targeting particular systems (say for example the Iranian nuclear program), and you've got good intel, you can write code to attack the specific hardware used, in designed ways. If on the other hand you've just released a virus into the wild, and hoped for the best, then you've no idea what kit your malware is interacting with.

  17. Anonymous Coward
    Anonymous Coward

    No hardware interlocks?

    Therac disaster, anyone?

    1. Disco-Legend-Zeke

      And They Want To Build...

      ...giant orbiting solar panels beaming microwaves at targe^H^H^H^H antenna farms on earth.

      Why is my beer so warm?

  18. Why Not?
    Thumb Down

    SOX anyone?

    Maybe use the Sarbanes Oxley legislation?

    I assume once its researched the errors will be a combination of default passwords and open end points.

    Lets have a few examples made.

  19. john 112

    I can think of one reason no one has reacted to this incident. If you follow the link you get a report, but no detail whatso ever. So how to react? What SCADA package? Network details? default passwords?

    Its very common to have mfg/scada networks separate from the office network and theoretically with no access to the www. But we all know that there is no perfect isolation with Humans involved.

  20. SeniorMoment


    Vulnerabilities like described in the article do raise questions about American's vulnerability to government sponsored hackers out to criple the USA economically and thus militarily.

    Mission critical, internal control systems should never be attached to the internet. If a programmer or other authorized person needs remote access to address problems, it should be through a dedicated line or better through an on-site operator. That operator could communicate with a programmner, etc. by fax, phone, or even by a completely different internet attached computer to make changes to the software or controls. Mission critical systems need to be closed computer operated and have failsafe software against equipment abuse.

    It would be humiliating for a manager to be recognized for making such a stupid decision as to allow internet access over open lines to a mission critical system, so it is easy to understand why the specific organization was not identified directly. Even high tech 128 bit encryption of financial passwords is not unbreakable with enough computer resources, so that encryption will have to change by the time quantum computers hit the market with multiple processors equivalent to a network of super computers today. Physical security always need to be considered first.

    Moving water is certainly a misson critical internal operation for a water utility. I do wonder though why the pumps were not designed to shut off until manually restarted in response to too many on off cycles, since such things can also happen with pump systems damaged in earthquakes or floods creating intermittent electrical connections. It is very likey the destroyed pump will be costly to replace.

    If it were not for the security breaches at defense contractors I would recommend the Pentagon give free classes to businesses on appropriate security, but apparently they haven't even been able to secure their network with their contractors who have lost secrets through lack of sufficient computer security.

  21. Fill

    Smart meters?

    Curiously enough, just a couple weeks ago the local power utility came to my house and replaced my meter with a 'smart' power meter which they tout now is networked all the way to the mothership. If somebody figures out an exploit to set them on fire, they could burn entire neighborhoods down. Of course, the hardware probably won't let such a thing happen, but it's funny the sort of trust you have to have with what utilities are attaching or wiring to your home.

  22. stp

    we are also the country that won't hire/train our own programmers but

    want to import them all from our competitor countries. so no big surprise!

    lots of companies fired their experienced American it professionals to

    hire cheap ones from overseas. its only a matter of time until the

    consequences begin to show.

  23. Anonymous Coward
    Anonymous Coward


    Burning out a pump motor via repeated stopping and starting is difficult if the correct circuit breakers/fuses are installed.

This topic is closed for new posts.

Other stories you might like