back to article US declares emergency after ransomware shuts oil pipeline that pumps 100 million gallons a day

One of the USA’s largest oil pipelines has been shut by ransomware, leading the nation's Federal Motor Carrier Safety Administration to issue a regional emergency declaration permitting the transport of fuel by road. The Colonial Pipeline says it carries 100 million gallons a day of refined fuels between Houston, Texas, and …

  1. jake Silver badge

    Presumably the fuckwits in charge ...

    ... had the pipeline's SCADA connected the TehIntraWebTubes at large, so management could impress their friends when fondling their iFad down at the club.

    It's going to get worse, until said fuckwits in charge realize that when an actual IT professional tells them "that is not today, has never been, and cannot ever be made safe, UNLESS we re-design The Internet from the ground up, from scratch!" it's actually the truth, not an excuse to get out of working.

    1. Anonymous Coward
      Anonymous Coward

      Re: Presumably the fuckwits in charge ...

      Until fuckwits in charge are personally held accountable by businesses (And or law enforcement where it's wilful) critical national infrastructure will continue to be operated poorly.

      I've no doubt "lessons will be learned" but that should include that the top two people in charge of securitys jobs are on the line when there's a breach of this scale.

      1. steelpillow Silver badge
        FAIL

        Re: Presumably the fuckwits in charge ...

        "lessons will be learned"

        One sap in the wrong place at the wrong time will be rounded on by the pack and sacrificed.

        The very lowest-hanging infosec fruit will be picked amid many sanity-defying press announcements.

        The whole pile of shite will be buried under NDAs and nobody will learn anything.

        1. Drew Scriver

          Re: Presumably the fuckwits in charge ...

          Where's that PR-template again? Google: "press release template for hacked comanies".

          Oh, here it is.

          "While our [service name] has experienced a limited and temporary reduction in service, [company name] remains committed to the highest standards of security. We are working diligently to resolve the situation and we have no evidence that our core operational systems were compromised."

          1. Marc 13

            Re: Presumably the fuckwits in charge ...

            FTFY

            "While our [service name] has experienced a limited and temporary reduction in service ***due to a sophisticated cyber attack***, [company name] remains committed to the highest standards of security. We are working diligently to resolve the situation and we have no evidence that our core operational systems were compromised."

    2. D@v3

      Re: Presumably the fuckwits in charge ...

      I know far too many places, where a statement such as..

      "UNLESS we re-design The Internet from the ground up, from scratch!"

      would be met with, "jolly good idea, get on with it"....

      1. jake Silver badge

        Re: Presumably the fuckwits in charge ...

        ""jolly good idea, get on with it"...."

        I can do that, if you have the budget for it

        We had the basics worked out with a secure version of TCP/IP back when the Internet was still based on NCP running on IMPs ... A dude by the name of Vint Cerf nixed it back then (at the urging of his lords and masters in Washington), but I still have the code and documentation. Are you ready to upgrade all your kit that stupidly implemented TCP/IP in hardware?

    3. Snake Silver badge

      Re: Presumably the fuckwits in charge ...

      "that is not today, has never been, and cannot ever be made safe, UNLESS we re-design The Internet from the ground up, from scratch!"

      I've said that before. Got downvoted, often.

      People don't want change, even when change is good. The internet was designed with positive thoughts of cooperation in mind. It was never considered that, decades later, malcontents would intentionally try to ruin things for fun or profit.

      IPv6 should have been encrypted from the very first glimmer of a concept in someone's mind. But they were more concerned with that utopian ideal of perfect interconnectivity, everywhere and all the time, to believe that people could act...poorly. The consortium still can not believe that people don't want their smart lamp IP device accessible from anywhere in the world; they cannot understand that people are getting more and more concerned about security and that the idea of a worldwide uniquely-addressable device is no longer a good idea.

      20 years ago we were still naive, believing that people would want to do the "right thing". Today a lot less so - especially once we woke up to one modern another's political objectives, where "right thing" has been replaced with "It's all about ME".

      1. FlamingDeath Silver badge

        Re: Presumably the fuckwits in charge ...

        The internet was at its best in the 90’s, for me at least

        1. Relatively new still, a frontier

        2. No moron masses and social media

        3. Commercialism was there, but it was a lot smaller

        4. Malware happened, but it was fun malware, your bank account didn’t get emptied

        5. Anonymity as standard, tracking tech was less mature than today

        6. It was a place for geeks and nerds, too complicated and not enough instant gratification for the moron masses.

        Web 2.0 has a lot to answer for

        1. Snake Silver badge
          Devil

          Re: Internet was at its best in the 90's...

          you do realize that only proves that [large groups of] humans manage to fsck up everything they touch??

          1. jake Silver badge

            Re: Internet was at its best in the 90's...

            Of course. It's the only thing that Humanity has proven it is good at.

        2. Anonymous Coward
          Anonymous Coward

          Re: Presumably the fuckwits in charge ...

          re: "internet was at its best in the 90's"

          I was a relative latecomer (late 90s), but still recall a co-worker's T-shirt that said "The internet is full -- go away!". Agreed then and even more so now.

        3. jake Silver badge

          Re: Presumably the fuckwits in charge ...

          The Internet was at its best from roughly 1983 to roughly 1993. Prior to that, it was too hard for even the smart non-techies to use, after that it was easy enough for the LCD of humanity to use. In that butter-zone, though ... That was awesome.

      2. SImon Hobson Silver badge

        Re: Presumably the fuckwits in charge ...

        IPv6 should have been encrypted from the very first glimmer of a concept in someone's mind.

        Guess what, it was !

        Then someone came along and wanted a feature that was incompatible with that. AIUI IPv6 still technically supports E2E encryption at the network level, but it's not widely implemented.

        1. jake Silver badge

          Re: Presumably the fuckwits in charge ...

          Actually, early on in the proto-TCP/IP (before IPV4, even) we were going for encryption, and trying to make the network secure. The "intelligent" lads in Washington DC got wind of it, and had Vint Cerf tell us to stop that. No security or encryption allowed.

          The rest, as they say, is history ... and the current insecure by design(!!) clusterfuck.

          Hope all y'all are still enjoying all your online transactions ...

          1. MachDiamond Silver badge

            Re: Presumably the fuckwits in charge ...

            "Hope all y'all are still enjoying all your online transactions ..."

            What online transactions? I shop local and pay with cash.

            I tell a lie, but I use prepaid credit cards when I do pay online. I had my debit card compromised when out of town at a conference once. I learned not trust the bastages after that. My then bank refused to do squat to get me out of that jam. I couldn't even present myself at a branch to get some funds or have them issue me a new card (temp or otherwise). "A new card will be sent to your registered address in 7-10 business days". F!

      3. Keven E

        Re: Presumably the fuckwits in charge ...

        "...where "right thing" has been replaced with "It's all about ME"..."

        Add 20 years to that and you'll find Reaganomics... a serious contender for the ultimate atmosphere of "me-based" greed.

        1. jake Silver badge

          Re: Presumably the fuckwits in charge ...

          "Add 20 years to that and you'll find Reaganomics... a serious contender for the ultimate atmosphere of "me-based" greed."

          I'm fairly certain you'll find that the Trumpaholics trump that.

          1. Swarthy

            Re: Presumably the fuckwits in charge ...

            The Trumpists are nothing more than distilled Reagonistas.

            It's the same mind-set, except they care even less don't even pretend to care about other people.

      4. jake Silver badge
        Pint

        Re: Presumably the fuckwits in charge ...

        "I've said that before. Got downvoted, often."

        Me too. Doesn't stop it from being reality, though. Maybe, eventually, if enough of us keep pointing out the obvious, people will stop using this network for things it wasn't designed for.

        I'm not betting the farm on it, though. People as a group are stupid.

        "It was never considered that, decades later, malcontents would intentionally try to ruin things for fun or profit."

        Actually, it was considered very early on, pre-TCP/IP going live, even. See my comment above, re: Cerf. We knew we were reaching a critical mass when the trolls started in on Usenet. When AOL opened up to Usenet, it was all over.

        Have a beer :-)

    4. vtcodger Silver badge

      Re: Presumably the fuckwits in charge ...

      Presumably the fuckwits ...

      After some thought, I think it's possible that the IT community reaction may be a bit off-focus here.

      Look, we're dealing with a 2600km pipeline and storage complex here. How else would you propose to control it other than over "phone" lines? Alternatives like a private, separate, wired or wireless communication system would likely be costly, failure prone and require the same constant maintenance that the phone system does. And it might not help in cases like this.

      I think that in this case, the system level problem possibly doesn't come from remote control per se. It sounds like it comes from failure to isolate the physical delivery/storage system from the company's administrative, management, etc systems. The latter fails -- and it is very likely that it will from time to time. ... And it takes the product storage/delivery system with it. ... Ooops

      I have never worked with pipeline systems, but my guess would be that their is some need to extract billing and operating data from the product delivery system and probably some need to input some commands at times. So a completely separate system might be difficult/impractical/impossible. I did work at one time with some folks who were trying to extract an unclassified data subset from a classified system in real time. The difficulties were legion.

      So I don't think partitioning the product system from the other systems would be anywhere near as easy as it sounds. But I think it might be possible. A quick internet search leads me to believe that partitioning is not a very common practice. Maybe it should be.

      Partitioning/Isolation looks to be an entirely different problem than the well known potential problem of SCADA vulnerabilities. I suspect that fixing one will do little to solve the other.

      1. tip pc Silver badge

        Re: Presumably the fuckwits in charge ...

        Everyone thinks that all network connections are connected directly to the internet.

        Its easier to think of things the other way around.

        example being private circuits like MPLS circuits.

        the internet is really a mass of interconnected networks.

        imagine I have an international network spanning 1000 end points evenly spread across Europe (inc UK) & the US.

        my carrier of choice is BT.

        they can hook me up with private MPLS connections to all my sites across all the nations. I can even go ipvpn if I want. (https://www.globalservices.bt.com/en/solutions/products/ip-connect), they manage the connectivity at the local level with the local carriers ensuring the end point is on their network & intern is configured for my network, all transparent to me.

        My international MPLS service is private to my network, is carried across multiple carriers etc but is kept separate from the internet due to the way MPLS works and if needed IPVPN across MPLS.

        The great thing about MPLS is that carriers can also route the internet across their systems using MPLS ensuring separation of traffic.

        The point being that carriers are effectively a huge MPLS network ensuring separation of traffic until it reaches points where its intentionally shared.

        Its trivial for a carrier to supply a private MPLS service without it connecting to the internet. The company who pays for the service then does internet break out at controlled points.

        Thats the way its been done for decades.

        there is now a huge push to put everything directly on the internet, using cheap domestic internet connections & SD WAN to try and keep it all secure.

        It doesn't need to be that way, & for systems like this pipeline no auditor should permit direct internet connectivity to any component other than an assigned DCs with specific codes of connection in place.

        1. RegGuy1 Silver badge

          Re: Presumably the fuckwits in charge ...

          ... the internet is really a mass of interconnected networks.

          They should call it an inter-network of networks, each linking at its gateway.

          Perhaps 'internet' is a suitable name! Whatdayathink?

          Oh ok, 'information superhighway?' Ok then. :-)

      2. SImon Hobson Silver badge

        Re: Presumably the fuckwits in charge ...

        have never worked with pipeline systems, but my guess would be that their is some need to extract billing and operating data from the product delivery system and probably some need to input some commands at times

        It's the "need to input some commands" bit that's the problem.

        A pipeline typically isn't just a long metal tube. Something like this will have valves, pumps, etc, etc which all need to be correctly operated - e.g." don't run pump A unless valve B and valve C are open, and valve D is shut" sort of thing. And of course, the operators need to be able to see what state everything is in before commanding any changes. Get it wrong as you risk over-pressuring a pipe (potential for burst), dead-heading a pump (bad for the pump), and many other things that (especially for a pipe carrying environment unfriendly materials) that can cause a bad day.

        As already mentioned, the network controlling and monitoring this should be airgapped from all non-critical networks - and this is achievable if you put the effort into your network design. However, as Stuxnet proved, even this is not sufficient. Also, even if the actual SCADA system is OK, without the "general IT", you might not have access to the information needed such as what needs to be delivered to where.

      3. Alpine_Hermit

        Re: Presumably the fuckwits in charge ...

        "...Alternatives like a private, separate, wired or wireless communication system would likely be costly..."

        Not as costly as this shutdown.

        The internet is totally inappropriate for many use-cases, but because it's there, ubiquitous, cheap/free then the CFOs and CIOs insist on it. Hopefully this will now change.

        1. Doctor Syntax Silver badge
          Unhappy

          Re: Presumably the fuckwits in charge ...

          "Hopefully this will now change."

          Hope springs eternal.

      4. Doctor Syntax Silver badge

        Re: Presumably the fuckwits in charge ...

        "I have never worked with pipeline systems, but my guess would be that their is some need to extract billing and operating data from the product delivery system and probably some need to input some commands at times. So a completely separate system might be difficult/impractical/impossible"

        That pipeline has been in operation for longer than an internet-based control system seems feasible. It has been proven to be operable without exposure to the net. It has now been proven to be inoperable with exposure to the net.

        We can, then, eliminate impossible and impractical. Difficult, maybe. Probably an option you left out - inconvenient. But far better than what they've got now.

      5. MachDiamond Silver badge

        Re: Presumably the fuckwits in charge ...

        "Look, we're dealing with a 2600km pipeline and storage complex here. How else would you propose to control it other than over "phone" lines? Alternatives like a private, separate, wired or wireless communication system would likely be costly,"

        A pipeline that transporting billions of dollars of petroleum products at a time. I expect that it's precisely the application for an independent, redundant and private comms network all of its own.

    5. Tron

      Re: Presumably the fuckwits in charge ...

      You do not need to redesign the internet.

      Just don't connect your fundamental infrastructure to it. Because doing so is extremely stupid, as it was from Day 1.

      Anyone handling large amounts of personal data needs to retain it offline or think about using a distributed topology so that customers' data is held on customer's own systems.

      Then you have designed out much of the problem. Bears like honey, so don't keep lots of it in one place, making ransomware so lucrative.

    6. MachDiamond Silver badge

      Re: Presumably the fuckwits in charge ...

      "Presumably the fuckwits in charge ...

      ... had the pipeline's SCADA connected the TehIntraWebTubes at large, so management could impress their friends when fondling their iFad down at the club."

      Which is why it is extremely important to document every request you get in hardcopy if you work in IT when you see these mandates. CYA in big red neon letters! If you get hauled into court, you want to show a judge that you advised against doing it in no uncertain terms and were ordered to do it anyway. It would also help to show that if you refused, you be out on your arse.

      I left a company where I was the safety manager when the company told me I could be overruled during operations. This wasn't the sole reason I left but a big one. In my case, people could be seriously hurt or killed. I guess that could be said for stuffing up a major petroleum pipeline.

  2. Stanislav Bonita

    One word:

    Airgap.

    1. jake Silver badge

      Re: One word:

      Easily defeated by any old idiot using sneakernet. And usually is.

      1. Paul Crawford Silver badge

        Re: One word:

        Only for very high value targets.

        Or, of curse, for systems where some muppet allows auto-run on USB drives and for staff to play use critical machines for entertainment. Which is probably more common...

        1. Alpine_Hermit

          Re: One word:

          "Only for very high value targets"

          This pipeline is more than valuable for a nation State attacker.

          1. doublelayer Silver badge

            Re: One word:

            But it is not a nation-state attacker who took it down this time. This reminds me of a person I used to work with whose attitude toward security was that every attacker was either someone too stupid to figure out anything where the password wasn't 123456 or a coordinated effort by at least three countries, therefore basically anything was fine because the Chinese military or NSA could figure a way in. He was wrong. You can accept that a system cannot ever be perfectly secure and still secure it to a level where it's a lot harder for them to get in. If the nation-state attacker had to launch a sneakernet attack to shut down the pipeline, this ransomware gang wouldn't have gotten in and it would be working today. It would also be a lot less likely that a nation-state attacker would do that, because such attacks are costly for them so they won't do it as often.

      2. CAPS LOCK

        Re: One word:

        Plastic padding. (Type Elastic natch.)

        1. Doctor Syntax Silver badge

          Re: One word:

          Errrm....

          That was two.

        2. Anonymous Coward
          Anonymous Coward

          Re: One word:

          In my long lost youth I used a lot of that on my cars.

          1. Anonymous Coward
            Anonymous Coward

            Re: One word:

            Heck, I had a girlfriend with that for awhile.

      3. Alan Brown Silver badge

        Re: One word:

        which is exactly what happened in Iran with the centrifuges

    2. Tom 7 Silver badge

      Re: One word:

      There is in the bosses heads.

    3. hoola Silver badge

      Re: One word:

      But that minor detail is seen as an inconvenience in this world of everything connected. There is the belief that everything must be online, accessible or remotely managed and it is all cool. This is hot helped by the generation now working there way up the ranks of management that see all this IoT crap and think it is the answer to everything. They can turn their heating on from a phone, it is an app so it must be secure.

      This is despite what the (usually older & experienced) IT Professionals are constantly them but too many just don't want to know.

      Then when it all goes wrong it is IT's fault and problem to sort it out.

    4. Nick Ryan Silver badge

      Re: One word:

      It doesn't particularly have to be an air gap, just some form of very well controlled gap. Because, of course, any form of gap can be bridged somehow (sneakernet reference below).

      Locking down the control systems of the oil pipeline and ensuring that only very limited and controlled communications are allowed in and out should not be difficult. The management system can still be on an office network, but should just be a system that either reports on what is happening or can provide direction to what is happening, the actual operations should be self contained.

      Oh, and backups. If it's critical, it requires a backup.

      Reads like a standard collection of fail where security is an after thought and can be cobbled on later if needed and everything is wide open.

      1. Doctor Syntax Silver badge

        Re: One word:

        "security is an after thought and can be cobbled on later if needed"

        I wonder if sometimes these systems start out as secure and it's convenience cost-saving insecurity that's cobbled on later.

        1. Nick Ryan Silver badge

          Re: One word:

          That's probably quite true.

          Security is incovenient. Having to use a key to open my front door is inconvenient. But then is having any form of latching mechanism, hell just a swing door. Nope, that's inconvenient too, just get rid of the door. Hell, why stop there, forcing people to go through a single entry, remove the walls too... (I may be getting a bit sarcastic here but it's how these things work)

        2. HellDeskJockey-ret

          Re: One word:

          From my experience they usually start of with "get it working." Security is an afterthought if at atl.

          1. fredesmite2

            Re: One word: billy-bob

            So Billy-Bob was jerking off in the back room to some x-rated web site that down loaded a virus

            lol

      2. GraXXoR

        Re: One word:

        Yes.. Backups... Hell, even if it's not critical... Backups backups backups....

        1. bombastic bob Silver badge
          Devil

          Re: One word:

          thanks for the reminder, I need to do backups today. not the automatic daily "do the dumps and copy the incremental changes to 3 different machines on the network" kind, but the "burn it to DVD and put it in the safe" kind. just my own stuff, but still...

          (I admit, I've been a bit lax on the DVD burning)

      3. steelpillow Silver badge
        Pirate

        Re: One word:

        Oh, and backups. If it's critical, it requires a backup.

        Bwahahahahaha!!!!!!

        1. 0laf Silver badge
          FAIL

          Re: One word:

          Voice of experience....

          Backups aren't worth shit if they're not tested and used regularly.

          I've seen backups that were logged as sucessful that had 0k or 64kb of a multi terabyte backup.

          I've seen backups that IT didn't know how to restore

          I've seen attack ships on fire off the shoulder or Orion

          Backups should be tested, restoration should be tested and loss of systems should be tested as exercises.

          1. Swiss Anton

            Re: One word:

            You need to do more than check that you can recover from a backup, you need to make sure that the backup isn't compromised. There is no fun in restoring a system with a dormant virus which then wakes up and re-infects the restored software and data.

          2. Doctor Syntax Silver badge

            Re: One word:

            "I've seen backups that IT didn't know how to restore"

            If it isn't tested it isn't a backup.

          3. macjules Silver badge

            Re: One word:

            Exactly that.

            "@Devops Have you done the backup?"

            "Yes"

            "@Devops Have you restored the backup to a new image?"

            "Yes"

            "@QA Did you test it?"

            "Yes"

            "@QA Did you actually look at the restored site, rather than accept the result of an automated test?"

            "Err, no"

    5. Potemkine! Silver badge

      Re: One word:

      Airgap

      Agreed, but making myself the devil's advocate, how do you update your systems with an airgap? Upload new configuration files? Backup when you have no backup infrastructure?

      For a real airgap, you require a lot of hardware for two environments, development and production. I see many companies whose beancounters don't like to invest in the IT part we can kick.

      1. Cuddles Silver badge

        Re: One word:

        It's not completely trivial, but it's not as comlicated as it might be. When it comes down to it, the vast majority of software updates these days are either security updates or pointless faffing around with the UI. An airgapped control system doesn't need either of those. Once you have your fuel line, or whatever, up and running, there should be relatively little that needs changing in the future. After all, the main reason security is such a problem with these systems is that most of them haven't been updated for years or even decades.

        1. MericanMan
          WTF?

          Re: One word:

          I spent almost 20 years working for a multi-state mid-stream refining and pipeline company here in the US. While some of these comments that have been made are entirely accurate that you need some information flow to/from the control systems these days, there are certainly ways to make it "fairly" secure.

          Our systems started out as old, completely isolated industrial control systems that were only accessible from dedicated workstations in the control room, which were not in any way connected to business systems. When the systems were upgraded, however, that was no longer practical, as they did need to get certain data out of it, and did need to at least be able to see the status of the systems remotely (*changes* to the system were not wanted or needed remotely, as like most companies of this type, the control room is manned 24/7).

          The new ICS was still an *almost* completely separate network, including its own sets of redundant physical switches and fiber paths in the main facilities, plus private fiber-based ethernet transport from an ISP for connections between sites (even though private, those connections all still went through firewalls on each end that only allowed the expected control traffic), plus some number of private radio links for remote sites.

          The only point of connectivity with the business networks, since there had to be one, was a (redundant) server provided by the manufacturer, with a firewall between it and the business network, which was configured to only allow the very specific traffic that needed to flow to and from it, which consisted only of application-specific ports and protocols for the ICS "view only" software application. That server, at an application level, did not allow any changes to be made to the ICS, and the ICS required custom network cards for connecting to the ICS network, which had physical dip switches on them to set an ICS ID number for that station, and only authorized stations could make changes on the network regardless of software restrictions.

          All that, plus the fact that every ICS site had a firewall across its traffic to other sites, made for a relatively secure setup. Certainly random ransomware that got into the business network never would have been able to spread to the ICS. That's not to say that skilled, targeted attackers wouldn't maybe find some way in, but it at least would not be easy.

          We also made use of 2FA for most things, had the business inter-VLAN traffic going through firewalls that limited it to what should flow, etc, so even on the business network a ransomware attack would have a limited reach.

          It was, of course, a significant amount of effort to set up and maintain all of this, and I know there are many organizations that don't bother to do it properly. Having seen the state of inherent insecurity on many pieces of ICS equipment, I think that's a foolish decision on any company's part. ICS equipment is nearly as bad as IoT junk from a security standpoint, with things like hardcoded passwords, insecure designs, infrequent patching of any type, etc being the norm.

          It continues to surprise me though, in this day and age, how little some companies secure themselves. It's even more unforgivable when you're running critical infrastructure. I now work for a fairly small financial firm that is NOT responsible for anything critical other than to us and our customers, yet we've taken a very security-focused posture normally only seen at larger enterprises. Hopefully more of you get to work with a company that understands the importance of security rather than just wanting to be able to tick some audit boxes and move on.

          1. fredesmite2
            Mushroom

            Re: One word:

            Yet someone can open a blind email in WIndows Exploder email and bring down the entire system

            1. Doctor Syntax Silver badge

              Re: One word:

              Not if the sort of precautions outlined are taken.

          2. Nick Ryan Silver badge

            Re: One word:

            A long time ago I worked for a company that installed hardware into (NHS) hospitals and so on.

            We started with a PC supplied by us that worked as a basic server with a serial link to a PC sitting in the private network that we fitted. Reasonably safe until the PC supplied by us got hammered by whatever was lurking around the system and for supplier "reasons" applying security updates to any of the kit including the external PC were not permitted.

            Later we switched to deploying an edge firewall with all of our PCs operating within the private network and no serial link. The firewall doesn't have to be expensive, or even with lots of features, all it needed to do was to only allow an incoming communication on the port(s) that we permitted and nothing else, there were no communications permitted outbound at all. We sold it as "protecting the NHS network" from whatever was in the private network however our main priority was the other way round of course...

            It was almost an amusing day that I turned up on site with the local IT staff running around reimaging systems all around the department and they weren't happy when I told them that they were under no circumstances to be allowed near the systems that we supplied. Luckily for me even though I got there after they started they couldn't access the (locked in cabinets) PCs that we supplied so they didn't trash our systems. They weren't happy about not being able to reimage our PCs but eventually had to back down.

            It was a somewhat less amusing day when I found that an engineer had introduced an auto-run virus to our systems because he'd used an infected USB storage stick. It was at that point I found that despite the morons at Microsoft introducing a policy and settings to "not auto-run" that the stupid OS "auto-ran" regardless. This "functionality" was only fixed by a later OS update/patch which, of course, the original vendor refused to be permitted to be installed as it might impact operation - although only if their code was crap. I had three visits to that site for this one reason before I got very grumpy with the situation and things changed...

      2. Doctor Syntax Silver badge

        Re: One word:

        "how do you update your systems with an airgap?"

        Carefully. With a proper, tested process.

    6. DS999 Silver badge

      Airgap need not reduce functionality

      Use a one way serial connection for data to get OUT of the SCADA system, while not allowing any data in.

      That way you can remotely monitor what is going on, but can't make any changes remotely.

      1. Jonathan Richards 1 Silver badge
        Thumb Up

        Re: Airgap need not reduce functionality

        Exactly. Data Diode

  3. sanmigueelbeer Silver badge
    WTF?

    Lessons learnt? I doubt it.

    But it’s still a scary event, because attacks on critical infrastructure have long been identified as having the potential for enormous harm. This one has halted operations of infrastructure that impacts the lives of millions of people, every day.

    What lessons were learnt? Nothing.

    If "critical infrastructure" are deemed critical then take them off the f**king internet.

    NOTE: I am not going to talk about "patching" or "updates" because there are custom-made applications that cannot be updated or patched.

    1. jake Silver badge

      Re: Lessons learnt? I doubt it.

      But we do patch and/or update them. Just not by clicking a link in email.

    2. Brian Miller

      Re: Lessons learnt? I doubt it.

      It depends on who does the learning, and who does the managing of what has been learned. Usually there is a village missing its idiot, who is to be found wearing a suit and tie.

      One time I had a brief chat with a fellow who worked for Big Oil, and he said his main job was to play "hide the (huge) profits." It's not like these companies lack resources, they lack managers who will do the job they were hired to do.

      I'm guessing that the whole PC network got infected, and then it doesn't matter that the actual controllers are fine. The PCs are the machines that are used to communicate with the critical infrastructure. Even if a PC is used just for its browser, if you can't use the browser, then the PC is toast.

      It's past time to move back to punch cards and paper tape! Let the miscreants try to take over OS/360 and a stack of punch cards!

      1. Natalie Gritpants Jr Silver badge

        Re: Lessons learnt? I doubt it.

        It was not difficult to make a card that would cause a reboot on certain mainframes. There was a Boroughs machine at Warwick Uni in the early 80s that was susceptible (allegedly). Scatter enough of those in the pigeonholes used by students to submit their jobs and the computer will slow to crawl until they are all found.

        1. Doctor Syntax Silver badge

          Re: Lessons learnt? I doubt it.

          Go on, you know it'd be a good "Who, me?".

      2. vtcodger Silver badge

        Re: Lessons learnt? I doubt it.

        " Let the miscreants try to take over OS/360 "

        Good point. Internet hackers are clever, but I doubt they are clever enough ever to master JCL. On the downside, identifying malicious JCL would be harder than identifying malicious Javascript. Pretty much impossible I should think.

        I still remember a chance lunchtime encounter in the 1960s with an early OS/360 user. He described JCL as the world's first syntax free language. What more can you say about a scripting language that frequently requires a null operator and calls its null operator "IEFBR14"?

        1. Anonymous IV
          Thumb Down

          Re: Lessons learnt? I doubt it.

          > What more can you say about a scripting language that frequently requires a null operator and calls its null operator "IEFBR14"?

          Years and years ago, in the days of OS/MVT 18.0 (or near offer), a new programmer was hired by IBM to work on MVT. This clever lad (for he was a non-female person) realised that IEFBR14 was a massive 4 bytes in size, and consisted of the two IBM Assembler instructions:

          SR 15,15 ; clear register 15 to zero

          BR 14 ; branch to the return address held in register 14

          He thought that he would optimise the program by removing the first instruction, leaving only the two byte instruction:

          BR 14

          This he did, and put the 'optimised' version of IEFBR14 into the next MVT update. No testing or change-control was needed - "obviously" - since it was such a simple change.

          However, he did not realise that register 15 was the return-code register, which would now contain an unspecified number, but not zero, and consequently all jobs which tested for a zero return code began to fail spectacularly.

          I am not aware what happened to the hapless programmer.

          1. vtcodger Silver badge

            Re: Lessons learnt? I doubt it.

            I am not aware what happened to the hapless programmer.

            Odds are that he became either a manager or marketer.

            1. David 132 Silver badge
              Trollface

              Re: Lessons learnt? I doubt it.

              He went on to write a widely-deployed Linux init system.

      3. Doctor Syntax Silver badge

        Re: Lessons learnt? I doubt it.

        "they lack managers who will do the job they were hired to do"

        It might be managers doing what they were hired to do - cut costs.

      4. sanmigueelbeer Silver badge

        Re: Lessons learnt? I doubt it.

        Usually there is a village missing its idiot, who is to be found wearing a suit and tie

        Petya and WannaCry(pt) was the wake-up call.

        Unfortunately, some went back to sleep after hitting the "snooze" button.

    3. Anonymous Coward
      Anonymous Coward

      Re: Lessons learnt? I doubt it.

      In the UK the NHS is critical national infrastructure yet it's going MORE online every day, more interconnected and trust me there's a huge lack of resources in the security/governance areas as these aren't considered critical.. oddly enough.

      If they were we'd have far more staff working in trusts with info sec / cyber sec roles.

      1. Anonymous Coward
        Anonymous Coward

        Re: Lessons learnt? I doubt it.

        Having just spent the last three months trying to use their systems I heartily support this. Eight unconnected systems with different requirements and so many workarounds I wonder how anyone gets things done. If the medical staff worked this way we'd all be pushing up daisies within a week.

  4. Anonymous Coward
    Anonymous Coward

    Stick that it your pipe

    But please, don’t smoke it!

    It’s an oil pipeline after all.

  5. croc

    Hey - at least they didn't reverse the flow from one end... I mean, a few million bucks to buy off a ransom is cheap compared to the Valdez.

    1. Grunchy

      They may have reversed the flow. Luckily you can disconnect power while resetting the system.

    2. FlamingDeath Silver badge

      People are mentioning SCADA, you’re mentioning reversing flow. Do we even know this was the entry point?

      The details are scarce, except that the event has caused them to shut down some systems as a precaution

      My bet is on the ‘meet our dumb fuck executive team who haven’t had a cyber awareness compliancy training in their whole lives’ website page

      Every organisation I’ve heard about having become victims of ransomware, all had an ego stroking ‘meet the team’ webpage, with first and last names, positions in company, maybe some personal details about their interests, etc.

      The way you get ransomeware, is you want ransomware

      1. FlamingDeath Silver badge

        The world has fundamentally changed, people post all sorts of useful information for potential hackers & social engineers alike.

        Oh looks a meet the team page

        Oh looks they have a twitter

        Oh looks they have some marketing event with x details

        Useful for crafting phishing context

        Hi {meet-team-page}

        {insert twitter related content}

        {dont put a link yet, start a convo, insert link after reply}

        Kind regards,

        {meet-team-page}

  6. ShadowDragon8685

    Meanwhile, over in DarkSide Land, the would-be Robin Hood malefactors are shitting very large bricks and trying to figure out if they can actually send an anonymous decryption key to Uncle Sam, or if it would be better to just emit their cryptography keys required to decrypt anything they've hit with this shit.

    And if this is something that can't be decrypted, they're shitting masonry bricks and booking plane tickets to somewhere they can book more plane tickets.

    Uncle Sam will quite happily drop JDAMs on houses for THIS shit, and frankly, it's about time the ransomware crook gang started getting their houses exploded for attacks that will be taken as attacks on national security interests and responded-to as if they'd flown a plane into the pipeline.

    1. Androgynous Cupboard Silver badge

      Ah, bless. Run and hide from the unimaginable worldwide power of the US government, that's your message? Might be time to wean yourself off the Hollywood blockbusters, my friend.

      No. The US will not be bombing Russia, or whichever former soviet-bloc republic is host to this crew.

      1. Anonymous Coward
        Anonymous Coward

        No. The US will not be bombing Russia, or whichever former soviet-bloc republic is host to this crew.

        Maybe just a terrible accident with some home-made vodka, fatally poisoning a few computer nerds who'd got together at a friends house for some "gaming"?

    2. Rich 11 Silver badge

      Twat

      Uncle Sam will quite happily drop JDAMs on houses for THIS shit, and frankly, it's about time the ransomware crook gang started getting their houses exploded for attacks

      How many resident children would you be happy to see 'exploded' along with their houses? Should neighbours or people in the street who also get 'exploded' be seen as unfortunate collateral damage or as entirely the fault of those goddam' cyber turrists for choosing to live in a residential neighbourhood?

      1. hoola Silver badge

        Re: Twat

        Anyone see that film, "Eye in the sky"?

        Helen Mirrem trying to remove some bigwig terrorist with a drone strike.....

        Exactly this problem, there is no such thing as a targeted air strike in built up areas without collateral damage & casualties. If governments feel the need to do this then unfortunately it requires boots on the ground and risk. They are not prepared to do that because of the political fallout if (when) if goes wrong and those in power have to take responsibility.

        In this era of conflicts reported in near real-time by the media too much is seen in the context of a computer game.

      2. fajensen Silver badge

        Re: Twat

        All of them, of course. Only Americans are People to The US World Police (and even that hangs by a thread). Everyone else are either targets or collateral damage.

        What's new is that they have whacked a couple of American citizens (Anwar al-Awlaki, Abdulrahman al-Awlaki, Nawar al-Awlaki, plus the usual bystanders), to kinda test the waters one assume. Domestic US police kills Americans with impunity and have for years, the CIA must be getting envious.

    3. FuzzyTheBear
      Joke

      You ain't serious

      JDAM's are manufactured by Boeing , chances are they will not hit the target .. Chances are they will hit 26.67768005289034, -80.03704457336461 .. That would be our luck but wont solve the problem we're concerned with :D

      1. Anonymous Coward
        Anonymous Coward

        Re: You ain't serious

        You're a bit north of intended target: 26.67707258489724, -80.03698016247051 is better.

      2. spireite
        Devil

        Re: You ain't serious

        JDAMs?

        Boeings latest, the 737MAX has proven to be not very accurate in its targetting mechanism,

        1. jake Silver badge

          Re: You ain't serious

          Congratulations. You have won this month's "Stupidest Post" award ... and we're not even in week three yet.

      3. Rich 11 Silver badge

        Re: You ain't serious

        26.67768005289034, -80.03704457336461

        I might have miscounted the decimal places there (it's too early in the morning), but I think that's a claimed precision on the order of 10 nanometres.

    4. Anonymous Coward
      Anonymous Coward

      no need for collateral damage, a couple well placed grams of lead will be enough.

    5. Anonymous Coward
      Anonymous Coward

      Suppose a missile can hit a physical address with a precision of 2 meters. But which address? The address of the orphanage supplied by a double agent? Or the agent's mother in law's address? Such "intelligence" was always a problem in Afghanistan, and was one of the reasons why trillions of dollars went down the drain.

      Furthermore, the cost of war per year with any of the big 4 non-US-ally state malware producers, Russia+satellites, China, NK, and Iran, would be more expensive by several orders of magnitude than the cost of dealing with malware. So instead it is dealt with in like, by using the internet to spy or perhaps attack, although that news generally doesn't make it to the press due to the internal secrecy of the agencies and the countries involved. (Although Iran seems to be an exception).

      The only long term solution is defensive.

  7. Grunchy

    Shut down all pipelines!

    Shut down ALL pipelines, beginning with all city sewers.

    1. Alan Brown Silver badge

      Re: Shut down all pipelines!

      That's been done at least once

    2. NXM

      Re: Shut down all pipelines!

      All you need is a nice fatberg for that!

      1. spireite
        Coat

        Re: Shut down all pipelines!

        The reason the US doesn't seem to have many reported Fatbergs, is because they are all above ground waddling/walking/mostly driving between home and McDs

        1. jake Silver badge

          Re: Shut down all pipelines!

          Be careful who you are dissing, and about what ... The only country in the world with a bigger obesity problem in men than the UK is Malta ...

          https://renewbariatrics.com/uk-obesity-statistics/

  8. cantankerous swineherd Silver badge

    this is what happens when you rely on the internet.

    it'll get worse before the lesson is actually learnt.

    1. ghp

      Plenty of governments seem to take the internet as a given, while there are others that can easily take it down. Better to leave or create some alternatives.

    2. Warm Braw Silver badge

      It seems the pipeline has been operating since 1964, so the assumption has to be that there was a deliberate choice at some point to replace the control systems that previously existed with something more vulnerable.

      1. vtcodger Silver badge

        In 1964 ...

        In 1964, the control was probably provided by some dude driving out in a war surplus jeep to manually open or close one or more valves. ... After getting a phone call or teletype message. Probably more secure, but hardly bulletproof. And probably kind of expensive and not all that reliable.

        Just a guess.

        1. Doctor Syntax Silver badge

          Re: In 1964 ...

          "probably kind of expensive and not all that reliable"

          But not too bad compared with the current situation. Apart from anything else, where do you magic up all those road tankers when you need them? And if you succeed, where do you magic up the tankers to replace whatever it was they were doing before?

          1. Anonymous Coward
            Anonymous Coward

            Re: In 1964 ...

            From China, right after you apologise.

  9. FlamingDeath Silver badge

    Which one of these do you think were phished?

    The ego of these wankers, I see a pattern emerging

    https://www.colpipe.com/about-us/our-company/executive-team

    1. Paul Crawford Silver badge

      When the board of directors get massive fines and/or (preferable) some gaol time for failing to ensure a secure system leading to this sort of thing, only then we might see a bit more proactive security.

      1. sgp Bronze badge

        "Excellence in Everything We Do"

        Right.

      2. Doctor Syntax Silver badge

        Even then it might only happen in some cases after some examples are made which will probably take tome to drag through the courts.

    2. Emir Al Weeq

      Thanks for the link.

      Loved the CIO's entry...

      Marie is recognized as a leader in the energy industry for her technology leadership, cybersecurity expertise

      1. Lon24 Silver badge
        Mushroom

        Just managementspeak. More worryingly she had a handle or not on 'Southern Nuclear'.

        When oil stops flowing to New York other things just stop. When water stops flowing to a reactor things might go a little more iconic.

    3. steviebuk Silver badge

      Did Angela Kolar

      Vice President, Operations Services & Chief Risk Officer skip work recently or just not bother to notice the this risk?

      I wonder if its like the oil in Texas where the companies were given, I believe, funds to weather proof their shit but didn't bother and used the money on other things, then we saw what happened in winter.

      Nothing will be learned, as said, until the big bosses are fined if it can be showed they are incompetent and that it wasn't due to some lower down rogue employee.

      1. David Neil

        To be fair to Angela she had one eye on the risk, unfortunately the other eye was 3 streets over and looking in a shop window

      2. Doctor Syntax Silver badge

        "and that it wasn't due to some lower down rogue employee"

        In such circumstances it can always be shown that it was some lower down rogue employee.

    4. Xalran

      Accountants and Financials

      Well, since none of them has a Degree/Msc/Phd in computer science or assimilated but all of them except the head of operation are accountants of finance people of some kind, we can only assume that they have never worked on risk assessment and mitigation below manglement level.

      No surprise they got hit.

      1. Lunatic Looking For Asylum

        Re: Accountants and Financials

        But shouldn't they have been told ?

        Realistically, somebody somewhere down the food chain will have warned them about the possibility of the attack - I bet there's some techy guy rubbing his hands with glee and running round shouting I told you so - naa naa na naa naaa :-)

        1. Anonymous Coward
          Anonymous Coward

          Re: Accountants and Financials

          No.

          I mean, sure someone warned them, but the techie guy that saw this coming isn't running around with glee, he's running around trying to fix this shitshow while getting angry enquiries from upper management asking for "status". When the storm blows over, the techies won't get credit for the fix, they'll just get the blame.

      2. Anonymous Coward
        Anonymous Coward

        Re: Accountants and Financials

        The CIO appears to be trained as a teacher.

        1. Doctor Syntax Silver badge

          Re: Accountants and Financials

          That should help with all the lessons to be learned.

  10. Ken Moorhouse Silver badge

    The Internet...

    Very convenient as everywhere is connected to everywhere else.

    A pipeline has the ideal infrastructure for a "private net". Run the comms along the pipeline for the most part, and not connect it to the internet. Problem there is the necessity to communicate with people on the internet. Some kind of trapdoor is needed to ensure data flows one way. However, the overall business needs to have feedback back into manage the demand, etc. Basic control theory says that, even without disruptive influences, feedback is a problem without proper damping. That damping can be ensured by using humans to close the feedback loop.

    Use technology to do what technology is good at. Use humans to do what they are good at.

    1. Anonymous Coward
      Anonymous Coward

      Re: The Internet...

      Use humans to do what they are good at.

      Suggestions on a postcard to...

      1. quxinot Silver badge

        Re: The Internet...

        On a postcard?

        The majority of the internet seems to be hosting that particular answer!

    2. Doctor Syntax Silver badge

      Re: The Internet...

      "Run the comms along the pipeline for the most part, and not connect it to the internet."

      Have you no idea of the number of executive bonuses you could pay with the amount it would cost to maintain that when all you have to do with the internet is pay the phone bill?

    3. Anonymous Coward
      Anonymous Coward

      Re: The Internet...

      Smart meter anyone?

    4. Jellied Eel Silver badge

      Re: The Internet...

      A pipeline has the ideal infrastructure for a "private net". Run the comms along the pipeline for the most part, and not connect it to the internet. Problem there is the necessity to communicate with people on the internet.

      Often there are wayleaves for fibre along pipeline routes. But problem is often not a necessity to communicate with people, but machines. So you've got routers, or systems running software that 'needs' to communicate with license servers, analytics, ad servers etc etc. So basically the kind of cloudybollox that blows holes in perimeter security. People are simpler to manage, ie access via controlled gateways, but there can still be risks if someone gets physical access to an out-of-the-way site and plugs into a craft terminal or console port.

      Some kind of trapdoor is needed to ensure data flows one way.

      Back in the good'ol days of serial connections, you could sometimes just snip TX or RX pins for that, give or take any need for flow control. But harder to do with IP.

      1. John H Woods Silver badge

        Re: "Harder to do"

        Well the expense is in buying data diodes rather than having people who know which pins to snip, and the management issue is then "do we really need this box?" rather than "do we really need Bob?"

        Said it before and will say it again: the chief bean counters in IT outfits should be actuaries rather than accountants, because at least the former know how (or at least that some attempt should be made) to price risk, whereas the latter tend just to see excess costs that can be trimmed to increase profit.

    5. Anonymous Coward
      Anonymous Coward

      Re: The Internet...

      In some ways long pipelines are ideal cases for using public infrastructure for communication, not a private net . Running your own wire or fibre along the route is possible, but just one more thing to maintain and for the squirrels to eat . A single point of failure & sure to fail at the most remote and difficult to access point :) . Or the archetypal pipe line failure - hit by digger - will knock out all your communications at the same time.

      So for some years now, pipeline industry has been using public networks widely. Satellite in the past, but these days mobile phone networks are pretty reliable even in out of the way areas and of course cheaper. In many ways much more robust to have a comms path which isn't along the pipeline.

  11. steviebuk Silver badge

    I wonder

    If one of the big wigs demanded "I need access to gmail. I need internet access now from this station"

    But that is air gabbed sir/madem for safety.

    "I don't care. You know who I am. Get it done. I need to book this dinner date for my partner I promised them about and forgot"

    Opens to the Internet.

    "Oooo I got sent an email for a discount for a dinner date. There is a link. Thats a funny address, thisismalware.com. Never mind, should save me some bucks. They need me to run this file for the discount? Sure."

    Noooooooooooooooooooo!!!!!!!!

  12. Anonymous South African Coward Silver badge

    Forn Parts.

    The US Justice Department two weeks ago established a Ransomware and Digital Extortion Task Force to fight the scourge

    Good luck with that, chaps.

    Especially if said ne'er-do-well is operating from Forn Parts, where you cannot go without causing a Major Diplomatical Incident and Prelude To World War III.

  13. Andy The Hat Silver badge

    The oil pipeline is not the issue

    The real issue here is that it is becoming apparent that these "attacks" are totally out of judicial control. I don't believe *most* are specifically targeted but simply operated wherever they land and it looks profitable to operate.

    This particular case is basically a logistics issue for the US, it was potentially more of an issue for the NHS. At what point will these activities start taking lives either by bad luck or deliberate action - by shutting down life support machines, fire emergency response systems, a nuclear reactor control system or an ICBM control system? I would love to know what the various judicial investigatiors are doing in the background ... 'Follow the money' is the usual refrain but the sheer number of actors, 'untraceable' digital currencies, and judicial boundaries without cross-border cooperation make this difficult if not impossible. The other problem is the willingness to allow these actors to profit from their actions - as long as it makes money and the initiators are free to do it the activity will not stop.

    There are lots of global problems in this world but, if nobody gets a handle on it, ransomware will be the next global pandemic but this time under the control of a few "kingpins" (the big fish will rapidly eat the small fish) and resembling something out of a sc-fi film ...

    1. Anonymous Coward
      Anonymous Coward

      Re: something out of a sc-fi film ...

      One of the later "Die Hard"s, I think.

  14. EvilOldBugger
    FAIL

    Internet links

    It seems like a number of commentards are assuming that the infection came in directly from the Internet, whereas the article actually does not mention the vector (as I recall). It could have been a link on an email, that infected the (probably high level Managers) PC, and was transferred from there to their infrastructure, or any other route.

    Still, I agree it is unlikely anything will be done about it until the incident has some major effect on the person involved (by that I mean a sacking for the manager, not the innocent means of transfer), but will that happen? Unfortunately, probably not. The article does mention that 'Colonial Pipeline appears not to have worked towards short recovery time objectives'.

    1. Nick Ryan Silver badge

      Re: Internet links

      The most likely route would be the office network being compromised somehow and that this was connected to what should have been a secure control network/infrastructure allowing what should be secure to be compromised too. How the office network was compromised? Any number of ways, but assuming that the office network won't be compromised and therefore having no segragation is a common stupid.

    2. Anonymous Coward
      Anonymous Coward

      Re: Internet links

      If anyone wants to mess with the SCADA systems for the electrical grid, a good starting point might be to kick in the door of some remote substation, install a Raspberry Pi or similar in one of the switches, leave, and then sniff around via WiFi+SSH+Yagi. Probably a good idea to leave a tatty sleeping bag and some other stuff behind so they will think some bum broke in and ran when he heard the service car.

      1. Doctor Syntax Silver badge

        Re: Internet links

        The Pi would be noticed. Scene examinations look for more than sleeping bags.

    3. Kev99

      Re: Internet links

      Even if it was an email link that link would have needed to connect to bunch of holes to send the trash beck to the central controller.

  15. Real Ale is Best
    Boffin

    When will they learn not to connect critical infrastructure to the Internet?

    Really this is just asking for trouble.

    1. Anonymous Coward
      Anonymous Coward

      Yes much better to not connect critical machines to networks. That way they can't catch anything nasty, except via the usb sticks that you then need to move data in and out on. Of course you will want to avoid using USB sticks for all except your key data, so probably best not to try patching OSes or updating virus protection. There are some upsides, as the system isn't connected to the Internet you don't need to worry about passwords 'admin/1234' will do fine.

      And yes really, seen it and it was just as bad as it sounds. Million dollar system 'has never worked right as we keep getting failures'.

      1. Doctor Syntax Silver badge

        Ah, yes. Don't connect it to the internet == don't use networks.

        The starting point for building a secure network is to recognise the fallacy in that.

  16. Anonymous Coward
    Anonymous Coward

    That reminds me...

    of the time when I had a contract to audit the fiscal metering systems of one of the UK's main pipeline systems. Metering systems traditionally used a specialist computer for each metered stream, continually taking readings of flow rate, pressure, temperature and density, to compute the mass exported. A separate loop took regular samples for weekly analysis of composition (each well's production is different, and can vary over time). There were over 20 installations feeding into the pipeline and a further metering station onshore. The oil going in had to balance with what came out, taking into account it took differing times for each installations' output to reach shore (from 2 to 14 days). Each installation operator would be paid (and taxed) on their share of the total throughput (and charged their share of pipeline costs). If I found their measuring wasn't within the agreed tolerances, it could cost them $millions in lost revenue - which sometimes happened.

    Anyway, the IT angle is that the computers (typically 3-5 for each installation) were stand-alone and their only external connection was output to data loggers; all other interfacing was digital (in the form of a technician's fingers pressing buttons or twiddling dials). It worked well. One company decided a different route: to go to virtual metering computers. Replace the individual units with virtual ones running on a Windows PC, connected back to the onshore office. What could possibly go wrong with that? My report expressed concerns over the move, as long-term system integrity was yet to be demonstrated. Not an external intrusion but, by my next year's visit, it was "interesting" to learn that a Windows update was found to be incompatible with the specially written drivers that interfaced with the metering hardware. That was my last visit there as my 5-year contract came to an end and I decided not to bid for a new one. (That job had meant regular trips offshore, each just a couple nights, but amounting to hundreds of times getting into the yellow body bags (aka survival suits) followed by an hour or two in a noisy, uncomfortable upside-down Flymo. I had been offered an onshore contract that paid even more - not a difficult decision.)

    To illustrate the scale of financial risk, one company decided to dispense with the usual dedicated "metering tech" and add the monitoring role onto the "tiffy" team (instrument techs). Unfortunately, for the company, they discovered (well, I did, retrospectively during my annual visit) that restarting from a plant trip* was a busy time for the tiffies and they forgot to restart the metering and lost a whole day's revenue (several mission USD). That was well over several years of keeping the metering tech role and it was quickly reinstated.

    *Plant trips, when production is automatically shut down, are not uncommon as systems are designed to fail safe. Drastic but the lessons from Piper Alpha are taken seriously.

    1. Anonymous Coward
      Anonymous Coward

      Re: That reminds me...

      This story sounds disturbingly familiar; in a closely related industry. Modems dumping out a raw text file to an ancient Oracle database were used historically to bring data back; later replaced by a WinXP solution allegedly supposed to feed information to multiple interested parties. Yes, XP is still in production, mission critical environments, connected to multiple users. Bloody madness.

      I personally dealt with one incident where a badly-configured instrument lead to the under-metering of about £42M.

      The nature of the transfer was such that the "downstream" party were responsible for the meter, but the upstream party would periodically audit it.

      You cannot tell me that the downstream party did not know they were selling "more" than they were being billed for for several years in a row. And they had the temerity after the event to push to change the reconciliation rules (to limit the number of years you were allowed to go back and correct on discovery of the error).

      Economics aside; errors of this scale have safety implications too.

      The teams responsible for searching for these kinds of errors have been greatly diminished in capabilities between cost-cutting and retirements. I'm sure it's only a matter of time before an incident happens, and the findings recommend reinstating these capabilities.

      Sometimes, knowing stuff is a curse.

    2. blah@blag.com

      Re: That reminds me...

      Interesting read.

      Back in the day I worked in the UK Coal industry for a short time. Actually back in the 80's UK Coal tech was the best in the world, the German & Chinese industries looked here for innovation.

      So in the pit where I worked they were testing air sampling networks all over the pit, basically air lines pumped sampled air over miles of tunnels to the electronics where the sampling was done. One of the miners worked out you could fart on the input tube and trigger an alarm.

      One of the projects I worked on was analysing coal quality (moisture & ash) from face(s), up pit to processing, on over to the local power station. I spent weeks collecting samples then got to one of the main outbound conveyers, and found a constant rain falling on the conveyor from old workings. Hence the "high moisture content" and weird chemical signatures from the ironstone bands in those workings.

      Lessons were learned.

      1. Doctor Syntax Silver badge

        Re: That reminds me...

        Measuring things properly is hard. And that's in laboratory conditions.

  17. Anonymous Coward
    Anonymous Coward

    Why?

    Is anyone actually able to explain why corporate networks and critical systems coexist on a network - beyond 'stupidity?' I mean really, why?

    Is it that the suppliers of the systems that power these things are all updated by wire?

    1. Paul Crawford Silver badge

      Re: Why?

      Is anyone actually able to explain why corporate networks and critical systems coexist on a network - beyond 'stupidity?

      Money. Trying to save the cost of duplicated air-gapped/firewalled networks, or the time to manually check/reconcile things.

      Stupidity and greed cover the vast majority of disasters.

  18. codejunky Silver badge

    Hmm

    'The gang has also shared evidence that it has made charitable donations, and said it feels an obligation to share some of the ransoms it wins.'

    As in shrek- Rob from the rich and give too the needy, I keep a wee percentage but I'm not greedy.

    Probably better if they dont screw with domestic fuel supply which is probably of more benefit to the charities than a donation of illegal gains.

    1. JetSetJim Silver badge
      Headmaster

      Re: Hmm

      Presumably they are quoting the ransomware folks with the word "wins", rather than using the more appropriate "extorts"

  19. Anonymous Coward
    Anonymous Coward

    Microsoft has some serious questions to answer

    Right there, in the public Windows API is a section on "named pipes". Even more worrying, there's a function named DisconnectNamedPipe.

    https://docs.microsoft.com/en-us/windows/win32/api/namedpipeapi/nf-namedpipeapi-disconnectnamedpipe

    You see where I'm going with this? All it takes is one bad actor (and I don't mean that slimy coffee creep George Clooney) calling DisconnectNamedPipe and... carnage.

    No wonder Melania Gates is divorcing Bill.

    1. fredesmite2

      Re: pedos the same registration list as GOP voters

      Think Melania is sure Billy kept his underwear on while getting a back rub at Trump's friends house ?

  20. Anonymous Coward
    Anonymous Coward

    No good punk kids

    Like all these things its normally youth to blame.

    Either the ones launching the attack of the go getting whizz kids that get hired and promptly dismantle any security to get their new game to run.

    Solution is to only hire old old people . Youngsters can work in the warehouse moving boxes around till at least 60 yo before graduating to hands on IT systems.

  21. Anonymous Coward
    Anonymous Coward

    "Our pipelines can be hacked but the election was the most secure ever”

    A comment I saw earlier today.

    1. Scott Pedigo

      Re: "Our pipelines can be hacked but the election was the most secure ever”

      >> "Our pipelines can be hacked but the election was the most secure ever”

      Could easily be true.

      People might choose to infer that "most secure ever" equates to "was secure", but that would be just one interpretation, the other being "our election had slightly fewer gaping security holes than the previous elections".

      1. fredesmite2

        Re: "Our pipelines can be hacked but the election was the most secure ever”

        Prove it

        In court

        In front of a jury

  22. Val Halla

    Total Inability To Supply Under-priced Petrol.

  23. mmccul

    Air gap can be hard

    Several times, I've heard of supposedly air gapped systems that were connected to a command and control network, legitimately, which was connected to an administrative network, with all the sysadmins knowing that, but that admin network was connected to the ... and eventually, to the public network. Each link seemed appropriate in isolation, but not together, and no one realized the overall chain of links until something happened to demonstrate it.

    It's not a new problm, and it doesn't have to be malicious incompetence. Air gapping a single server is easy. Air gapping a network of systems that need to talk to each other to do their primary function is much harder.

    1. DevOpsTimothyC Bronze badge

      Re: Air gap can be hard

      "I've heard of supposedly air gapped systems that were connected to"

      As soon as you add "that were connected to" then there was no air gap. As soon as you're also saying "each link" some alarm bells should be ringing for multiple people.

  24. FuzzyTheBear
    FAIL

    LOLOL

    " Safety is our most important priority at Colonial Pipeline Company. Our focus on excellence means excellence in safety. At Colonial, we integrate best practices in safety into every element of operational and occupational processes. This enhances our robust asset integrity program, which is designed to prevent incidents from happening in the first place. "

    Yeah .. sure .. think they should reread the crap they post and fact check them :D :D :D

    1. FlamingDeath Silver badge

      Re: LOLOL

      Thats some robust integrity they got there

  25. Scott Pedigo
    Big Brother

    The gang has also shared evidence that it has made charitable donations...

    Al Capone was also viewed by some as a modern day Robin Hood. The St. Valentine's Day Massacre got him a new name: "Public Enemy Number One".

    As noted in previous comments, the hackers involved might be Russian. If so they might be state-sponsored, state look-the-other-way, or otherwise. They could be from anywhere. But wherever they are from, one the U.S. authorities know their identities, then there future travel prospects are limited. A vacation even years later to a seemingly safe former block country could result in an arrest on an Interpol warrant and deportation to the U.S.

    1. Anonymous Coward
      Anonymous Coward

      Re: The gang has also shared evidence that it has made charitable donations...

      That's their weak point. Who can resist the kids demands to go to Disneyworld? That's America's secret weapon.

  26. fredesmite2
    Mushroom

    See - Stuxnet

    THE ENTIRE "IT" AND "SECURITY TEAM " FROM THAT RED-NECK SHITTH0LE NEEDS FIRED IMMEDIATELY

  27. fredesmite2
    Mushroom

    I'd like to THANK PUTIN and his Russian Hackers

    For showing everyone what a 3rd world shittH0le looks like when left up to a bunch of poorly educated, red-neck, inbred assholes in the modern age.

    Anyone want to bet they were using unpatched Windows machines ?

  28. Boris the Cockroach Silver badge
    FAIL

    Air gap!

    Actually... I suspect one of the c-level twonks did'nt read the sign when surfing for adult entertainment

    "DONT DOWNLOAD SHIT FROM A DODGY WEBSITE!"

    Or completely missed the one saying

    "DONT OPEN EMAIL ATTATCHMENTS FROM 'FRIENDS' "

    We had one of the former... borrowed a works PC for surfing 'stuff' d/l this to watch latest US tv shows....

    And there went our network... and every PC attatched... thus he became a former employee.

    And my boss wonders why I glue up the USB ports when he gets some new PCs in......

  29. Muppet Boss Bronze badge
    Black Helicopters

    They better run and hide now

    These Darkside crims, they better run and hide, and hide well. And when found pray this is FBI and not headhunters, in the original meaning, from a private security company hired by oil tycoons. They better be very careful about not getting into fatal accidents from this moment.

  30. grizewald

    Who let all the children in here?

    I remember when El Reg had a much better class of commentard. Far too many posts here appear to have been penned by people who were up way past their bed time.

  31. Lorribot Silver badge

    IT is too complicated for most

    I remember when a CEO of a company rang up teh service desk and demanded they fix the Internet when BT was having an outage. This was serious request and shows the level of understanding of many top business folks that just don't understand technology. They also had a habit of throwing phones at walls when they didn't do what was expected.

    However a lot of this is that there are way too many people working in technology that know enough to be dangerous but don't care enough to be secure.

  32. sanmigueelbeer Silver badge

    The Colonial Pipeline says it carries 100 million gallons a day of refined fuels between Houston, Texas, and New York Harbor, or 45 percent of all fuel needed on the USA’s East Coast. The pipeline carries fuel for cars and trucks, jet fuel, and heating oil.

    Imagine that. A random ransomware attack to an industry which feeds 45% of fuel into the USA's East Coast. And this is in a peacetime environment.

    Now imagine what would happen if the US would be involved in a "shooting war".

    Some state actor have already demonstrated adept skills in shutting down several Saudi Arabian petrol plants with ransomware.

    No soup gasoline for you!

  33. sanmigueelbeer Silver badge

    DarkSide ransom group linked to Colonial Pipeline hack

    DarkSide ransom group linked to Colonial Pipeline hack

    DarkSide is one of a number of increasingly professionalised groups of digital extortionists, with a mailing list, a press centre, a victim hotline and even a supposed code of conduct intended to spin the group as reliable, if ruthless, business partners

    1. jake Silver badge

      Re: DarkSide ransom group linked to Colonial Pipeline hack

      They are nothing but extortionists. Stop spreading their propaganda.

      "It'd be a shame if your legs accidentally broke ... "

  34. Dr Paul Taylor

    gallons

    The Imperial gallon is 4.546 litre, the US gallon is 3.785 litres and the US dry gallon is 4.405 litres. Which one do you mean? The metric system was invented to clear away the confusion of different units of measurement across the (European) continent. Let's use it!

    1. Binraider Bronze badge

      Re: gallons

      I definitely agree with your sentiment about gallons. Energy is often traded in terms of units of energy (kW).

      Natural gas for example, if you burn 1 cubic meter of it at one standard atmosphere, the useful energy output varies with respect to the ambient air temperature when you try and ignite it.

      There are at least 6 sets of standard conditions used around Europe for differing standard temperatures. This not without good reason, burning an otherwise identical unit of natural gas in Barcelona with it's higher ambient temperature will give marginally more useful energy out than burning it in Stavanger. There is obviously a historical difference also to be found in gas sources from region to region e.g. North sea oil derived gas is usually a bit more energy rich than LNG per unit volume.

      Can lead to some interesting debates where the uneducated see 100MW of gas leaving one country but only 99.7MW arriving on the other side! (Because of the difference in reference conditions).

      As was noted above, measuring stuff is hard, and stupid non-metric unit systems only make it harder.

      1. jake Silver badge

        Re: gallons

        'stupid non-metric unit systems only make it harder."

        MUMMY! THE BAD PEOPLE MADE ME READ ABOUT SOMETHING I DON'T UNDERSTAND! MAKE THEM STOP! WAAAAAAAAAA ...

        1. Binraider Bronze badge

          Re: gallons

          Remind me who crashed a spacecraft into Mars because they couldn't work with Metric and Imperial at the same time?

          The SI system is there for a good reason. Use it.

    2. jake Silver badge
      Pint

      Re: gallons

      Have a pint? Or does the bathroom scale say that you weigh too many stone in these sedentary Covid times? Perhaps you should be walking/jogging/running a few miles per day? Did the milkman deliver your pint this morning? How many cubic feet of gas do you use boiling your kettle per month? If you're on the electrics, how many kWh does it take to boil the kettle?

      And etc. You Brits are funny when it comes to teh metrics :-)

      "The nice thing about standards is that there are so many of them to choose from." —Andrew S. Tanenbaum

  35. Captain Badmouth
    Holmes

    Been there, done that...

    Meanwhile, the Russian/ Chinese/ Iranian/ N Korean (delete as/if necessary) state-employed Black Hats are spitting blood waiting to see if their little compromise software, sitting quietly in a dark corner of a server somewhere, will be uncovered by the security measures introduced following this mal(ware)arkey.

    1. Jan 0 Silver badge

      Re: Been there, done that...

      You forgot the USA state-employed black hats. Be fair.

      1. jake Silver badge

        Re: Been there, done that...

        And the British. And the Israelis. And the French. And the Canadians. And the Australians. And the Germans. And ... Pretty much everybody, no?

        Note I'm not condoning it, just acknowledging the reality.

      2. Captain Badmouth
        WTF?

        Re: Been there, done that...

        I didn't think they'd be compromising their own oil company.

  36. Silverburn

    PR cringe

    Wow, an ransomware crew with a PR department trying to sound like a legit company, instead the sleazy bunch of ****bags they are. What a time to be alive.

    1. jake Silver badge

      Re: PR cringe

      "ransomware crew with a PR department"

      ::shrugs:: Most of us are used to that.

      See: Capitol Hill in your own jurisdiction.

  37. fredesmite2
    FAIL

    I'd like to know

    Which poorly educated dumbfkk #TrumpBillie opened an e-mail ad featuring " NUDE RUSSIAN BRIDES FOR FREE " on an 10 year old unpatched Windows machine that introduced the randomson virus.

    ..

  38. fredesmite2
    Unhappy

    Their nationwide passwd for all IOT devices

    was admin/password

  39. sanmigueelbeer Silver badge

    Top Russian Submarine Design Bureau Hit By Cyber Attack With Chinese Characteristics

    Top Russian Submarine Design Bureau Hit By Cyber Attack With Chinese Characteristics

  40. grumpy-old-person

    Cybersecurity?

    How is it so easy for the 'bad guys' to access ANYTHING, let alone critical infrastructure?

  41. sanmigueelbeer Silver badge

    Toshiba unit struck by DarkSide ransomware group

    Toshiba unit struck by DarkSide ransomware group

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Biting the hand that feeds IT © 1998–2021