US declares emergency after ransomware shuts oil pipeline that pumps 100 million gallons a day One of the USA’s largest oil pipelines has been shut by ransomware, leading the nation's Federal Motor Carrier Safety Administration to issue a regional emergency declaration permitting the transport of fuel by road. The Colonial Pipeline says it carries 100 million gallons a day of refined fuels between Houston, Texas, and …
... had the pipeline's SCADA connected the TehIntraWebTubes at large, so management could impress their friends when fondling their iFad down at the club.
It's going to get worse, until said fuckwits in charge realize that when an actual IT professional tells them "that is not today, has never been, and cannot ever be made safe, UNLESS we re-design The Internet from the ground up, from scratch!" it's actually the truth, not an excuse to get out of working.
Until fuckwits in charge are personally held accountable by businesses (And or law enforcement where it's wilful) critical national infrastructure will continue to be operated poorly.
I've no doubt "lessons will be learned" but that should include that the top two people in charge of securitys jobs are on the line when there's a breach of this scale.
"lessons will be learned"
One sap in the wrong place at the wrong time will be rounded on by the pack and sacrificed.
The very lowest-hanging infosec fruit will be picked amid many sanity-defying press announcements.
The whole pile of shite will be buried under NDAs and nobody will learn anything.
Where's that PR-template again? Google: "press release template for hacked comanies".
Oh, here it is.
"While our [service name] has experienced a limited and temporary reduction in service, [company name] remains committed to the highest standards of security. We are working diligently to resolve the situation and we have no evidence that our core operational systems were compromised."
FTFY
"While our [service name] has experienced a limited and temporary reduction in service ***due to a sophisticated cyber attack***, [company name] remains committed to the highest standards of security. We are working diligently to resolve the situation and we have no evidence that our core operational systems were compromised."
""jolly good idea, get on with it"...."
I can do that, if you have the budget for it
We had the basics worked out with a secure version of TCP/IP back when the Internet was still based on NCP running on IMPs ... A dude by the name of Vint Cerf nixed it back then (at the urging of his lords and masters in Washington), but I still have the code and documentation. Are you ready to upgrade all your kit that stupidly implemented TCP/IP in hardware?
"that is not today, has never been, and cannot ever be made safe, UNLESS we re-design The Internet from the ground up, from scratch!"
I've said that before. Got downvoted, often.
People don't want change, even when change is good. The internet was designed with positive thoughts of cooperation in mind. It was never considered that, decades later, malcontents would intentionally try to ruin things for fun or profit.
IPv6 should have been encrypted from the very first glimmer of a concept in someone's mind. But they were more concerned with that utopian ideal of perfect interconnectivity, everywhere and all the time, to believe that people could act...poorly. The consortium still can not believe that people don't want their smart lamp IP device accessible from anywhere in the world; they cannot understand that people are getting more and more concerned about security and that the idea of a worldwide uniquely-addressable device is no longer a good idea.
20 years ago we were still naive, believing that people would want to do the "right thing". Today a lot less so - especially once we woke up to one modern another's political objectives, where "right thing" has been replaced with "It's all about ME".
The internet was at its best in the 90’s, for me at least
1. Relatively new still, a frontier
2. No moron masses and social media
3. Commercialism was there, but it was a lot smaller
4. Malware happened, but it was fun malware, your bank account didn’t get emptied
5. Anonymity as standard, tracking tech was less mature than today
6. It was a place for geeks and nerds, too complicated and not enough instant gratification for the moron masses.
Web 2.0 has a lot to answer for
IPv6 should have been encrypted from the very first glimmer of a concept in someone's mind.
Guess what, it was !
Then someone came along and wanted a feature that was incompatible with that. AIUI IPv6 still technically supports E2E encryption at the network level, but it's not widely implemented.
Actually, early on in the proto-TCP/IP (before IPV4, even) we were going for encryption, and trying to make the network secure. The "intelligent" lads in Washington DC got wind of it, and had Vint Cerf tell us to stop that. No security or encryption allowed.
The rest, as they say, is history ... and the current insecure by design(!!) clusterfuck.
Hope all y'all are still enjoying all your online transactions ...
"Hope all y'all are still enjoying all your online transactions ..."
What online transactions? I shop local and pay with cash.
I tell a lie, but I use prepaid credit cards when I do pay online. I had my debit card compromised when out of town at a conference once. I learned not trust the bastages after that. My then bank refused to do squat to get me out of that jam. I couldn't even present myself at a branch to get some funds or have them issue me a new card (temp or otherwise). "A new card will be sent to your registered address in 7-10 business days". F!
"I've said that before. Got downvoted, often."
Me too. Doesn't stop it from being reality, though. Maybe, eventually, if enough of us keep pointing out the obvious, people will stop using this network for things it wasn't designed for.
I'm not betting the farm on it, though. People as a group are stupid.
"It was never considered that, decades later, malcontents would intentionally try to ruin things for fun or profit."
Actually, it was considered very early on, pre-TCP/IP going live, even. See my comment above, re: Cerf. We knew we were reaching a critical mass when the trolls started in on Usenet. When AOL opened up to Usenet, it was all over.
Have a beer :-)
Presumably the fuckwits ...
After some thought, I think it's possible that the IT community reaction may be a bit off-focus here.
Look, we're dealing with a 2600km pipeline and storage complex here. How else would you propose to control it other than over "phone" lines? Alternatives like a private, separate, wired or wireless communication system would likely be costly, failure prone and require the same constant maintenance that the phone system does. And it might not help in cases like this.
I think that in this case, the system level problem possibly doesn't come from remote control per se. It sounds like it comes from failure to isolate the physical delivery/storage system from the company's administrative, management, etc systems. The latter fails -- and it is very likely that it will from time to time. ... And it takes the product storage/delivery system with it. ... Ooops
I have never worked with pipeline systems, but my guess would be that their is some need to extract billing and operating data from the product delivery system and probably some need to input some commands at times. So a completely separate system might be difficult/impractical/impossible. I did work at one time with some folks who were trying to extract an unclassified data subset from a classified system in real time. The difficulties were legion.
So I don't think partitioning the product system from the other systems would be anywhere near as easy as it sounds. But I think it might be possible. A quick internet search leads me to believe that partitioning is not a very common practice. Maybe it should be.
Partitioning/Isolation looks to be an entirely different problem than the well known potential problem of SCADA vulnerabilities. I suspect that fixing one will do little to solve the other.
Everyone thinks that all network connections are connected directly to the internet.
Its easier to think of things the other way around.
example being private circuits like MPLS circuits.
the internet is really a mass of interconnected networks.
imagine I have an international network spanning 1000 end points evenly spread across Europe (inc UK) & the US.
my carrier of choice is BT.
they can hook me up with private MPLS connections to all my sites across all the nations. I can even go ipvpn if I want. (https://www.globalservices.bt.com/en/solutions/products/ip-connect), they manage the connectivity at the local level with the local carriers ensuring the end point is on their network & intern is configured for my network, all transparent to me.
My international MPLS service is private to my network, is carried across multiple carriers etc but is kept separate from the internet due to the way MPLS works and if needed IPVPN across MPLS.
The great thing about MPLS is that carriers can also route the internet across their systems using MPLS ensuring separation of traffic.
The point being that carriers are effectively a huge MPLS network ensuring separation of traffic until it reaches points where its intentionally shared.
Its trivial for a carrier to supply a private MPLS service without it connecting to the internet. The company who pays for the service then does internet break out at controlled points.
Thats the way its been done for decades.
there is now a huge push to put everything directly on the internet, using cheap domestic internet connections & SD WAN to try and keep it all secure.
It doesn't need to be that way, & for systems like this pipeline no auditor should permit direct internet connectivity to any component other than an assigned DCs with specific codes of connection in place.
have never worked with pipeline systems, but my guess would be that their is some need to extract billing and operating data from the product delivery system and probably some need to input some commands at times
It's the "need to input some commands" bit that's the problem.
A pipeline typically isn't just a long metal tube. Something like this will have valves, pumps, etc, etc which all need to be correctly operated - e.g." don't run pump A unless valve B and valve C are open, and valve D is shut" sort of thing. And of course, the operators need to be able to see what state everything is in before commanding any changes. Get it wrong as you risk over-pressuring a pipe (potential for burst), dead-heading a pump (bad for the pump), and many other things that (especially for a pipe carrying environment unfriendly materials) that can cause a bad day.
As already mentioned, the network controlling and monitoring this should be airgapped from all non-critical networks - and this is achievable if you put the effort into your network design. However, as Stuxnet proved, even this is not sufficient. Also, even if the actual SCADA system is OK, without the "general IT", you might not have access to the information needed such as what needs to be delivered to where.
"...Alternatives like a private, separate, wired or wireless communication system would likely be costly..."
Not as costly as this shutdown.
The internet is totally inappropriate for many use-cases, but because it's there, ubiquitous, cheap/free then the CFOs and CIOs insist on it. Hopefully this will now change.
"I have never worked with pipeline systems, but my guess would be that their is some need to extract billing and operating data from the product delivery system and probably some need to input some commands at times. So a completely separate system might be difficult/impractical/impossible"
That pipeline has been in operation for longer than an internet-based control system seems feasible. It has been proven to be operable without exposure to the net. It has now been proven to be inoperable with exposure to the net.
We can, then, eliminate impossible and impractical. Difficult, maybe. Probably an option you left out - inconvenient. But far better than what they've got now.
"Look, we're dealing with a 2600km pipeline and storage complex here. How else would you propose to control it other than over "phone" lines? Alternatives like a private, separate, wired or wireless communication system would likely be costly,"
A pipeline that transporting billions of dollars of petroleum products at a time. I expect that it's precisely the application for an independent, redundant and private comms network all of its own.
You do not need to redesign the internet.
Just don't connect your fundamental infrastructure to it. Because doing so is extremely stupid, as it was from Day 1.
Anyone handling large amounts of personal data needs to retain it offline or think about using a distributed topology so that customers' data is held on customer's own systems.
Then you have designed out much of the problem. Bears like honey, so don't keep lots of it in one place, making ransomware so lucrative.
"Presumably the fuckwits in charge ...
... had the pipeline's SCADA connected the TehIntraWebTubes at large, so management could impress their friends when fondling their iFad down at the club."
Which is why it is extremely important to document every request you get in hardcopy if you work in IT when you see these mandates. CYA in big red neon letters! If you get hauled into court, you want to show a judge that you advised against doing it in no uncertain terms and were ordered to do it anyway. It would also help to show that if you refused, you be out on your arse.
I left a company where I was the safety manager when the company told me I could be overruled during operations. This wasn't the sole reason I left but a big one. In my case, people could be seriously hurt or killed. I guess that could be said for stuffing up a major petroleum pipeline.
But it is not a nation-state attacker who took it down this time. This reminds me of a person I used to work with whose attitude toward security was that every attacker was either someone too stupid to figure out anything where the password wasn't 123456 or a coordinated effort by at least three countries, therefore basically anything was fine because the Chinese military or NSA could figure a way in. He was wrong. You can accept that a system cannot ever be perfectly secure and still secure it to a level where it's a lot harder for them to get in. If the nation-state attacker had to launch a sneakernet attack to shut down the pipeline, this ransomware gang wouldn't have gotten in and it would be working today. It would also be a lot less likely that a nation-state attacker would do that, because such attacks are costly for them so they won't do it as often.
But that minor detail is seen as an inconvenience in this world of everything connected. There is the belief that everything must be online, accessible or remotely managed and it is all cool. This is hot helped by the generation now working there way up the ranks of management that see all this IoT crap and think it is the answer to everything. They can turn their heating on from a phone, it is an app so it must be secure.
This is despite what the (usually older & experienced) IT Professionals are constantly them but too many just don't want to know.
Then when it all goes wrong it is IT's fault and problem to sort it out.
It doesn't particularly have to be an air gap, just some form of very well controlled gap. Because, of course, any form of gap can be bridged somehow (sneakernet reference below).
Locking down the control systems of the oil pipeline and ensuring that only very limited and controlled communications are allowed in and out should not be difficult. The management system can still be on an office network, but should just be a system that either reports on what is happening or can provide direction to what is happening, the actual operations should be self contained.
Oh, and backups. If it's critical, it requires a backup.
Reads like a standard collection of fail where security is an after thought and can be cobbled on later if needed and everything is wide open.
That's probably quite true.
Security is incovenient. Having to use a key to open my front door is inconvenient. But then is having any form of latching mechanism, hell just a swing door. Nope, that's inconvenient too, just get rid of the door. Hell, why stop there, forcing people to go through a single entry, remove the walls too... (I may be getting a bit sarcastic here but it's how these things work)
thanks for the reminder, I need to do backups today. not the automatic daily "do the dumps and copy the incremental changes to 3 different machines on the network" kind, but the "burn it to DVD and put it in the safe" kind. just my own stuff, but still...
(I admit, I've been a bit lax on the DVD burning)
Voice of experience....
Backups aren't worth shit if they're not tested and used regularly.
I've seen backups that were logged as sucessful that had 0k or 64kb of a multi terabyte backup.
I've seen backups that IT didn't know how to restore
I've seen attack ships on fire off the shoulder or Orion
Backups should be tested, restoration should be tested and loss of systems should be tested as exercises.
Airgap
Agreed, but making myself the devil's advocate, how do you update your systems with an airgap? Upload new configuration files? Backup when you have no backup infrastructure?
For a real airgap, you require a lot of hardware for two environments, development and production. I see many companies whose beancounters don't like to invest in the IT part we can kick.
It's not completely trivial, but it's not as comlicated as it might be. When it comes down to it, the vast majority of software updates these days are either security updates or pointless faffing around with the UI. An airgapped control system doesn't need either of those. Once you have your fuel line, or whatever, up and running, there should be relatively little that needs changing in the future. After all, the main reason security is such a problem with these systems is that most of them haven't been updated for years or even decades.
I spent almost 20 years working for a multi-state mid-stream refining and pipeline company here in the US. While some of these comments that have been made are entirely accurate that you need some information flow to/from the control systems these days, there are certainly ways to make it "fairly" secure.
Our systems started out as old, completely isolated industrial control systems that were only accessible from dedicated workstations in the control room, which were not in any way connected to business systems. When the systems were upgraded, however, that was no longer practical, as they did need to get certain data out of it, and did need to at least be able to see the status of the systems remotely (*changes* to the system were not wanted or needed remotely, as like most companies of this type, the control room is manned 24/7).
The new ICS was still an *almost* completely separate network, including its own sets of redundant physical switches and fiber paths in the main facilities, plus private fiber-based ethernet transport from an ISP for connections between sites (even though private, those connections all still went through firewalls on each end that only allowed the expected control traffic), plus some number of private radio links for remote sites.
The only point of connectivity with the business networks, since there had to be one, was a (redundant) server provided by the manufacturer, with a firewall between it and the business network, which was configured to only allow the very specific traffic that needed to flow to and from it, which consisted only of application-specific ports and protocols for the ICS "view only" software application. That server, at an application level, did not allow any changes to be made to the ICS, and the ICS required custom network cards for connecting to the ICS network, which had physical dip switches on them to set an ICS ID number for that station, and only authorized stations could make changes on the network regardless of software restrictions.
All that, plus the fact that every ICS site had a firewall across its traffic to other sites, made for a relatively secure setup. Certainly random ransomware that got into the business network never would have been able to spread to the ICS. That's not to say that skilled, targeted attackers wouldn't maybe find some way in, but it at least would not be easy.
We also made use of 2FA for most things, had the business inter-VLAN traffic going through firewalls that limited it to what should flow, etc, so even on the business network a ransomware attack would have a limited reach.
It was, of course, a significant amount of effort to set up and maintain all of this, and I know there are many organizations that don't bother to do it properly. Having seen the state of inherent insecurity on many pieces of ICS equipment, I think that's a foolish decision on any company's part. ICS equipment is nearly as bad as IoT junk from a security standpoint, with things like hardcoded passwords, insecure designs, infrequent patching of any type, etc being the norm.
It continues to surprise me though, in this day and age, how little some companies secure themselves. It's even more unforgivable when you're running critical infrastructure. I now work for a fairly small financial firm that is NOT responsible for anything critical other than to us and our customers, yet we've taken a very security-focused posture normally only seen at larger enterprises. Hopefully more of you get to work with a company that understands the importance of security rather than just wanting to be able to tick some audit boxes and move on.
A long time ago I worked for a company that installed hardware into (NHS) hospitals and so on.
We started with a PC supplied by us that worked as a basic server with a serial link to a PC sitting in the private network that we fitted. Reasonably safe until the PC supplied by us got hammered by whatever was lurking around the system and for supplier "reasons" applying security updates to any of the kit including the external PC were not permitted.
Later we switched to deploying an edge firewall with all of our PCs operating within the private network and no serial link. The firewall doesn't have to be expensive, or even with lots of features, all it needed to do was to only allow an incoming communication on the port(s) that we permitted and nothing else, there were no communications permitted outbound at all. We sold it as "protecting the NHS network" from whatever was in the private network however our main priority was the other way round of course...
It was almost an amusing day that I turned up on site with the local IT staff running around reimaging systems all around the department and they weren't happy when I told them that they were under no circumstances to be allowed near the systems that we supplied. Luckily for me even though I got there after they started they couldn't access the (locked in cabinets) PCs that we supplied so they didn't trash our systems. They weren't happy about not being able to reimage our PCs but eventually had to back down.
It was a somewhat less amusing day when I found that an engineer had introduced an auto-run virus to our systems because he'd used an infected USB storage stick. It was at that point I found that despite the morons at Microsoft introducing a policy and settings to "not auto-run" that the stupid OS "auto-ran" regardless. This "functionality" was only fixed by a later OS update/patch which, of course, the original vendor refused to be permitted to be installed as it might impact operation - although only if their code was crap. I had three visits to that site for this one reason before I got very grumpy with the situation and things changed...
But it’s still a scary event, because attacks on critical infrastructure have long been identified as having the potential for enormous harm. This one has halted operations of infrastructure that impacts the lives of millions of people, every day.
What lessons were learnt? Nothing.
If "critical infrastructure" are deemed critical then take them off the f**king internet.
NOTE: I am not going to talk about "patching" or "updates" because there are custom-made applications that cannot be updated or patched.
It depends on who does the learning, and who does the managing of what has been learned. Usually there is a village missing its idiot, who is to be found wearing a suit and tie.
One time I had a brief chat with a fellow who worked for Big Oil, and he said his main job was to play "hide the (huge) profits." It's not like these companies lack resources, they lack managers who will do the job they were hired to do.
I'm guessing that the whole PC network got infected, and then it doesn't matter that the actual controllers are fine. The PCs are the machines that are used to communicate with the critical infrastructure. Even if a PC is used just for its browser, if you can't use the browser, then the PC is toast.
It's past time to move back to punch cards and paper tape! Let the miscreants try to take over OS/360 and a stack of punch cards!
It was not difficult to make a card that would cause a reboot on certain mainframes. There was a Boroughs machine at Warwick Uni in the early 80s that was susceptible (allegedly). Scatter enough of those in the pigeonholes used by students to submit their jobs and the computer will slow to crawl until they are all found.
" Let the miscreants try to take over OS/360 "
Good point. Internet hackers are clever, but I doubt they are clever enough ever to master JCL. On the downside, identifying malicious JCL would be harder than identifying malicious Javascript. Pretty much impossible I should think.
I still remember a chance lunchtime encounter in the 1960s with an early OS/360 user. He described JCL as the world's first syntax free language. What more can you say about a scripting language that frequently requires a null operator and calls its null operator "IEFBR14"?
> What more can you say about a scripting language that frequently requires a null operator and calls its null operator "IEFBR14"?
Years and years ago, in the days of OS/MVT 18.0 (or near offer), a new programmer was hired by IBM to work on MVT. This clever lad (for he was a non-female person) realised that IEFBR14 was a massive 4 bytes in size, and consisted of the two IBM Assembler instructions:
SR 15,15 ; clear register 15 to zero
BR 14 ; branch to the return address held in register 14
He thought that he would optimise the program by removing the first instruction, leaving only the two byte instruction:
BR 14
This he did, and put the 'optimised' version of IEFBR14 into the next MVT update. No testing or change-control was needed - "obviously" - since it was such a simple change.
However, he did not realise that register 15 was the return-code register, which would now contain an unspecified number, but not zero, and consequently all jobs which tested for a zero return code began to fail spectacularly.
I am not aware what happened to the hapless programmer.
In the UK the NHS is critical national infrastructure yet it's going MORE online every day, more interconnected and trust me there's a huge lack of resources in the security/governance areas as these aren't considered critical.. oddly enough.
If they were we'd have far more staff working in trusts with info sec / cyber sec roles.
Having just spent the last three months trying to use their systems I heartily support this. Eight unconnected systems with different requirements and so many workarounds I wonder how anyone gets things done. If the medical staff worked this way we'd all be pushing up daisies within a week.
People are mentioning SCADA, you’re mentioning reversing flow. Do we even know this was the entry point?
The details are scarce, except that the event has caused them to shut down some systems as a precaution
My bet is on the ‘meet our dumb fuck executive team who haven’t had a cyber awareness compliancy training in their whole lives’ website page
Every organisation I’ve heard about having become victims of ransomware, all had an ego stroking ‘meet the team’ webpage, with first and last names, positions in company, maybe some personal details about their interests, etc.
The way you get ransomeware, is you want ransomware
The world has fundamentally changed, people post all sorts of useful information for potential hackers & social engineers alike.
Oh looks a meet the team page
Oh looks they have a twitter
Oh looks they have some marketing event with x details
Useful for crafting phishing context
Hi {meet-team-page}
{insert twitter related content}
{dont put a link yet, start a convo, insert link after reply}
Kind regards,
{meet-team-page}
Meanwhile, over in DarkSide Land, the would-be Robin Hood malefactors are shitting very large bricks and trying to figure out if they can actually send an anonymous decryption key to Uncle Sam, or if it would be better to just emit their cryptography keys required to decrypt anything they've hit with this shit.
And if this is something that can't be decrypted, they're shitting masonry bricks and booking plane tickets to somewhere they can book more plane tickets.
Uncle Sam will quite happily drop JDAMs on houses for THIS shit, and frankly, it's about time the ransomware crook gang started getting their houses exploded for attacks that will be taken as attacks on national security interests and responded-to as if they'd flown a plane into the pipeline.
Uncle Sam will quite happily drop JDAMs on houses for THIS shit, and frankly, it's about time the ransomware crook gang started getting their houses exploded for attacks
How many resident children would you be happy to see 'exploded' along with their houses? Should neighbours or people in the street who also get 'exploded' be seen as unfortunate collateral damage or as entirely the fault of those goddam' cyber turrists for choosing to live in a residential neighbourhood?
Anyone see that film, "Eye in the sky"?
Helen Mirrem trying to remove some bigwig terrorist with a drone strike.....
Exactly this problem, there is no such thing as a targeted air strike in built up areas without collateral damage & casualties. If governments feel the need to do this then unfortunately it requires boots on the ground and risk. They are not prepared to do that because of the political fallout if (when) if goes wrong and those in power have to take responsibility.
In this era of conflicts reported in near real-time by the media too much is seen in the context of a computer game.
All of them, of course. Only Americans are People to The US World Police (and even that hangs by a thread). Everyone else are either targets or collateral damage.
What's new is that they have whacked a couple of American citizens (Anwar al-Awlaki, Abdulrahman al-Awlaki, Nawar al-Awlaki, plus the usual bystanders), to kinda test the waters one assume. Domestic US police kills Americans with impunity and have for years, the CIA must be getting envious.
Suppose a missile can hit a physical address with a precision of 2 meters. But which address? The address of the orphanage supplied by a double agent? Or the agent's mother in law's address? Such "intelligence" was always a problem in Afghanistan, and was one of the reasons why trillions of dollars went down the drain.
Furthermore, the cost of war per year with any of the big 4 non-US-ally state malware producers, Russia+satellites, China, NK, and Iran, would be more expensive by several orders of magnitude than the cost of dealing with malware. So instead it is dealt with in like, by using the internet to spy or perhaps attack, although that news generally doesn't make it to the press due to the internal secrecy of the agencies and the countries involved. (Although Iran seems to be an exception).
The only long term solution is defensive.
In 1964, the control was probably provided by some dude driving out in a war surplus jeep to manually open or close one or more valves. ... After getting a phone call or teletype message. Probably more secure, but hardly bulletproof. And probably kind of expensive and not all that reliable.
Just a guess.
"probably kind of expensive and not all that reliable"
But not too bad compared with the current situation. Apart from anything else, where do you magic up all those road tankers when you need them? And if you succeed, where do you magic up the tankers to replace whatever it was they were doing before?
Did Angela Kolar
Vice President, Operations Services & Chief Risk Officer skip work recently or just not bother to notice the this risk?
I wonder if its like the oil in Texas where the companies were given, I believe, funds to weather proof their shit but didn't bother and used the money on other things, then we saw what happened in winter.
Nothing will be learned, as said, until the big bosses are fined if it can be showed they are incompetent and that it wasn't due to some lower down rogue employee.
Well, since none of them has a Degree/Msc/Phd in computer science or assimilated but all of them except the head of operation are accountants of finance people of some kind, we can only assume that they have never worked on risk assessment and mitigation below manglement level.
No surprise they got hit.
But shouldn't they have been told ?
Realistically, somebody somewhere down the food chain will have warned them about the possibility of the attack - I bet there's some techy guy rubbing his hands with glee and running round shouting I told you so - naa naa na naa naaa :-)
No.
I mean, sure someone warned them, but the techie guy that saw this coming isn't running around with glee, he's running around trying to fix this shitshow while getting angry enquiries from upper management asking for "status". When the storm blows over, the techies won't get credit for the fix, they'll just get the blame.
Very convenient as everywhere is connected to everywhere else.
A pipeline has the ideal infrastructure for a "private net". Run the comms along the pipeline for the most part, and not connect it to the internet. Problem there is the necessity to communicate with people on the internet. Some kind of trapdoor is needed to ensure data flows one way. However, the overall business needs to have feedback back into manage the demand, etc. Basic control theory says that, even without disruptive influences, feedback is a problem without proper damping. That damping can be ensured by using humans to close the feedback loop.
Use technology to do what technology is good at. Use humans to do what they are good at.
A pipeline has the ideal infrastructure for a "private net". Run the comms along the pipeline for the most part, and not connect it to the internet. Problem there is the necessity to communicate with people on the internet.
Often there are wayleaves for fibre along pipeline routes. But problem is often not a necessity to communicate with people, but machines. So you've got routers, or systems running software that 'needs' to communicate with license servers, analytics, ad servers etc etc. So basically the kind of cloudybollox that blows holes in perimeter security. People are simpler to manage, ie access via controlled gateways, but there can still be risks if someone gets physical access to an out-of-the-way site and plugs into a craft terminal or console port.
Some kind of trapdoor is needed to ensure data flows one way.
Back in the good'ol days of serial connections, you could sometimes just snip TX or RX pins for that, give or take any need for flow control. But harder to do with IP.
Well the expense is in buying data diodes rather than having people who know which pins to snip, and the management issue is then "do we really need this box?" rather than "do we really need Bob?"
Said it before and will say it again: the chief bean counters in IT outfits should be actuaries rather than accountants, because at least the former know how (or at least that some attempt should be made) to price risk, whereas the latter tend just to see excess costs that can be trimmed to increase profit.
In some ways long pipelines are ideal cases for using public infrastructure for communication, not a private net . Running your own wire or fibre along the route is possible, but just one more thing to maintain and for the squirrels to eat . A single point of failure & sure to fail at the most remote and difficult to access point :) . Or the archetypal pipe line failure - hit by digger - will knock out all your communications at the same time.
So for some years now, pipeline industry has been using public networks widely. Satellite in the past, but these days mobile phone networks are pretty reliable even in out of the way areas and of course cheaper. In many ways much more robust to have a comms path which isn't along the pipeline.
If one of the big wigs demanded "I need access to gmail. I need internet access now from this station"
But that is air gabbed sir/madem for safety.
"I don't care. You know who I am. Get it done. I need to book this dinner date for my partner I promised them about and forgot"
Opens to the Internet.
"Oooo I got sent an email for a discount for a dinner date. There is a link. Thats a funny address, thisismalware.com. Never mind, should save me some bucks. They need me to run this file for the discount? Sure."
Noooooooooooooooooooo!!!!!!!!
The US Justice Department two weeks ago established a Ransomware and Digital Extortion Task Force to fight the scourge
Good luck with that, chaps.
Especially if said ne'er-do-well is operating from Forn Parts, where you cannot go without causing a Major Diplomatical Incident and Prelude To World War III.
The real issue here is that it is becoming apparent that these "attacks" are totally out of judicial control. I don't believe *most* are specifically targeted but simply operated wherever they land and it looks profitable to operate.
This particular case is basically a logistics issue for the US, it was potentially more of an issue for the NHS. At what point will these activities start taking lives either by bad luck or deliberate action - by shutting down life support machines, fire emergency response systems, a nuclear reactor control system or an ICBM control system? I would love to know what the various judicial investigatiors are doing in the background ... 'Follow the money' is the usual refrain but the sheer number of actors, 'untraceable' digital currencies, and judicial boundaries without cross-border cooperation make this difficult if not impossible. The other problem is the willingness to allow these actors to profit from their actions - as long as it makes money and the initiators are free to do it the activity will not stop.
There are lots of global problems in this world but, if nobody gets a handle on it, ransomware will be the next global pandemic but this time under the control of a few "kingpins" (the big fish will rapidly eat the small fish) and resembling something out of a sc-fi film ...
It seems like a number of commentards are assuming that the infection came in directly from the Internet, whereas the article actually does not mention the vector (as I recall). It could have been a link on an email, that infected the (probably high level Managers) PC, and was transferred from there to their infrastructure, or any other route.
Still, I agree it is unlikely anything will be done about it until the incident has some major effect on the person involved (by that I mean a sacking for the manager, not the innocent means of transfer), but will that happen? Unfortunately, probably not. The article does mention that 'Colonial Pipeline appears not to have worked towards short recovery time objectives'.
The most likely route would be the office network being compromised somehow and that this was connected to what should have been a secure control network/infrastructure allowing what should be secure to be compromised too. How the office network was compromised? Any number of ways, but assuming that the office network won't be compromised and therefore having no segragation is a common stupid.
If anyone wants to mess with the SCADA systems for the electrical grid, a good starting point might be to kick in the door of some remote substation, install a Raspberry Pi or similar in one of the switches, leave, and then sniff around via WiFi+SSH+Yagi. Probably a good idea to leave a tatty sleeping bag and some other stuff behind so they will think some bum broke in and ran when he heard the service car.
Yes much better to not connect critical machines to networks. That way they can't catch anything nasty, except via the usb sticks that you then need to move data in and out on. Of course you will want to avoid using USB sticks for all except your key data, so probably best not to try patching OSes or updating virus protection. There are some upsides, as the system isn't connected to the Internet you don't need to worry about passwords 'admin/1234' will do fine.
And yes really, seen it and it was just as bad as it sounds. Million dollar system 'has never worked right as we keep getting failures'.
of the time when I had a contract to audit the fiscal metering systems of one of the UK's main pipeline systems. Metering systems traditionally used a specialist computer for each metered stream, continually taking readings of flow rate, pressure, temperature and density, to compute the mass exported. A separate loop took regular samples for weekly analysis of composition (each well's production is different, and can vary over time). There were over 20 installations feeding into the pipeline and a further metering station onshore. The oil going in had to balance with what came out, taking into account it took differing times for each installations' output to reach shore (from 2 to 14 days). Each installation operator would be paid (and taxed) on their share of the total throughput (and charged their share of pipeline costs). If I found their measuring wasn't within the agreed tolerances, it could cost them $millions in lost revenue - which sometimes happened.
Anyway, the IT angle is that the computers (typically 3-5 for each installation) were stand-alone and their only external connection was output to data loggers; all other interfacing was digital (in the form of a technician's fingers pressing buttons or twiddling dials). It worked well. One company decided a different route: to go to virtual metering computers. Replace the individual units with virtual ones running on a Windows PC, connected back to the onshore office. What could possibly go wrong with that? My report expressed concerns over the move, as long-term system integrity was yet to be demonstrated. Not an external intrusion but, by my next year's visit, it was "interesting" to learn that a Windows update was found to be incompatible with the specially written drivers that interfaced with the metering hardware. That was my last visit there as my 5-year contract came to an end and I decided not to bid for a new one. (That job had meant regular trips offshore, each just a couple nights, but amounting to hundreds of times getting into the yellow body bags (aka survival suits) followed by an hour or two in a noisy, uncomfortable upside-down Flymo. I had been offered an onshore contract that paid even more - not a difficult decision.)
To illustrate the scale of financial risk, one company decided to dispense with the usual dedicated "metering tech" and add the monitoring role onto the "tiffy" team (instrument techs). Unfortunately, for the company, they discovered (well, I did, retrospectively during my annual visit) that restarting from a plant trip* was a busy time for the tiffies and they forgot to restart the metering and lost a whole day's revenue (several mission USD). That was well over several years of keeping the metering tech role and it was quickly reinstated.
*Plant trips, when production is automatically shut down, are not uncommon as systems are designed to fail safe. Drastic but the lessons from Piper Alpha are taken seriously.
This story sounds disturbingly familiar; in a closely related industry. Modems dumping out a raw text file to an ancient Oracle database were used historically to bring data back; later replaced by a WinXP solution allegedly supposed to feed information to multiple interested parties. Yes, XP is still in production, mission critical environments, connected to multiple users. Bloody madness.
I personally dealt with one incident where a badly-configured instrument lead to the under-metering of about £42M.
The nature of the transfer was such that the "downstream" party were responsible for the meter, but the upstream party would periodically audit it.
You cannot tell me that the downstream party did not know they were selling "more" than they were being billed for for several years in a row. And they had the temerity after the event to push to change the reconciliation rules (to limit the number of years you were allowed to go back and correct on discovery of the error).
Economics aside; errors of this scale have safety implications too.
The teams responsible for searching for these kinds of errors have been greatly diminished in capabilities between cost-cutting and retirements. I'm sure it's only a matter of time before an incident happens, and the findings recommend reinstating these capabilities.
Sometimes, knowing stuff is a curse.
Interesting read.
Back in the day I worked in the UK Coal industry for a short time. Actually back in the 80's UK Coal tech was the best in the world, the German & Chinese industries looked here for innovation.
So in the pit where I worked they were testing air sampling networks all over the pit, basically air lines pumped sampled air over miles of tunnels to the electronics where the sampling was done. One of the miners worked out you could fart on the input tube and trigger an alarm.
One of the projects I worked on was analysing coal quality (moisture & ash) from face(s), up pit to processing, on over to the local power station. I spent weeks collecting samples then got to one of the main outbound conveyers, and found a constant rain falling on the conveyor from old workings. Hence the "high moisture content" and weird chemical signatures from the ironstone bands in those workings.
Lessons were learned.
Is anyone actually able to explain why corporate networks and critical systems coexist on a network - beyond 'stupidity?
Money. Trying to save the cost of duplicated air-gapped/firewalled networks, or the time to manually check/reconcile things.
Stupidity and greed cover the vast majority of disasters.
'The gang has also shared evidence that it has made charitable donations, and said it feels an obligation to share some of the ransoms it wins.'
As in shrek- Rob from the rich and give too the needy, I keep a wee percentage but I'm not greedy.
Probably better if they dont screw with domestic fuel supply which is probably of more benefit to the charities than a donation of illegal gains.
Right there, in the public Windows API is a section on "named pipes". Even more worrying, there's a function named DisconnectNamedPipe.
https://docs.microsoft.com/en-us/windows/win32/api/namedpipeapi/nf-namedpipeapi-disconnectnamedpipe
You see where I'm going with this? All it takes is one bad actor (and I don't mean that slimy coffee creep George Clooney) calling DisconnectNamedPipe and... carnage.
No wonder Melania Gates is divorcing Bill.
Like all these things its normally youth to blame.
Either the ones launching the attack of the go getting whizz kids that get hired and promptly dismantle any security to get their new game to run.
Solution is to only hire old old people . Youngsters can work in the warehouse moving boxes around till at least 60 yo before graduating to hands on IT systems.
>> "Our pipelines can be hacked but the election was the most secure ever”
Could easily be true.
People might choose to infer that "most secure ever" equates to "was secure", but that would be just one interpretation, the other being "our election had slightly fewer gaping security holes than the previous elections".
Several times, I've heard of supposedly air gapped systems that were connected to a command and control network, legitimately, which was connected to an administrative network, with all the sysadmins knowing that, but that admin network was connected to the ... and eventually, to the public network. Each link seemed appropriate in isolation, but not together, and no one realized the overall chain of links until something happened to demonstrate it.
It's not a new problm, and it doesn't have to be malicious incompetence. Air gapping a single server is easy. Air gapping a network of systems that need to talk to each other to do their primary function is much harder.
" Safety is our most important priority at Colonial Pipeline Company. Our focus on excellence means excellence in safety. At Colonial, we integrate best practices in safety into every element of operational and occupational processes. This enhances our robust asset integrity program, which is designed to prevent incidents from happening in the first place. "
Yeah .. sure .. think they should reread the crap they post and fact check them :D :D :D
Al Capone was also viewed by some as a modern day Robin Hood. The St. Valentine's Day Massacre got him a new name: "Public Enemy Number One".
As noted in previous comments, the hackers involved might be Russian. If so they might be state-sponsored, state look-the-other-way, or otherwise. They could be from anywhere. But wherever they are from, one the U.S. authorities know their identities, then there future travel prospects are limited. A vacation even years later to a seemingly safe former block country could result in an arrest on an Interpol warrant and deportation to the U.S.
Actually... I suspect one of the c-level twonks did'nt read the sign when surfing for adult entertainment
"DONT DOWNLOAD SHIT FROM A DODGY WEBSITE!"
Or completely missed the one saying
"DONT OPEN EMAIL ATTATCHMENTS FROM 'FRIENDS' "
We had one of the former... borrowed a works PC for surfing 'stuff' d/l this to watch latest US tv shows....
And there went our network... and every PC attatched... thus he became a former employee.
And my boss wonders why I glue up the USB ports when he gets some new PCs in......
These Darkside crims, they better run and hide, and hide well. And when found pray this is FBI and not headhunters, in the original meaning, from a private security company hired by oil tycoons. They better be very careful about not getting into fatal accidents from this moment.
I remember when a CEO of a company rang up teh service desk and demanded they fix the Internet when BT was having an outage. This was serious request and shows the level of understanding of many top business folks that just don't understand technology. They also had a habit of throwing phones at walls when they didn't do what was expected.
However a lot of this is that there are way too many people working in technology that know enough to be dangerous but don't care enough to be secure.
The Colonial Pipeline says it carries 100 million gallons a day of refined fuels between Houston, Texas, and New York Harbor, or 45 percent of all fuel needed on the USA’s East Coast. The pipeline carries fuel for cars and trucks, jet fuel, and heating oil.
Imagine that. A random ransomware attack to an industry which feeds 45% of fuel into the USA's East Coast. And this is in a peacetime environment.
Now imagine what would happen if the US would be involved in a "shooting war".
Some state actor have already demonstrated adept skills in shutting down several Saudi Arabian petrol plants with ransomware.
No soup gasoline for you!
DarkSide ransom group linked to Colonial Pipeline hack
DarkSide is one of a number of increasingly professionalised groups of digital extortionists, with a mailing list, a press centre, a victim hotline and even a supposed code of conduct intended to spin the group as reliable, if ruthless, business partners
I definitely agree with your sentiment about gallons. Energy is often traded in terms of units of energy (kW).
Natural gas for example, if you burn 1 cubic meter of it at one standard atmosphere, the useful energy output varies with respect to the ambient air temperature when you try and ignite it.
There are at least 6 sets of standard conditions used around Europe for differing standard temperatures. This not without good reason, burning an otherwise identical unit of natural gas in Barcelona with it's higher ambient temperature will give marginally more useful energy out than burning it in Stavanger. There is obviously a historical difference also to be found in gas sources from region to region e.g. North sea oil derived gas is usually a bit more energy rich than LNG per unit volume.
Can lead to some interesting debates where the uneducated see 100MW of gas leaving one country but only 99.7MW arriving on the other side! (Because of the difference in reference conditions).
As was noted above, measuring stuff is hard, and stupid non-metric unit systems only make it harder.
Have a pint? Or does the bathroom scale say that you weigh too many stone in these sedentary Covid times? Perhaps you should be walking/jogging/running a few miles per day? Did the milkman deliver your pint this morning? How many cubic feet of gas do you use boiling your kettle per month? If you're on the electrics, how many kWh does it take to boil the kettle?
And etc. You Brits are funny when it comes to teh metrics :-)
"The nice thing about standards is that there are so many of them to choose from." —Andrew S. Tanenbaum
Meanwhile, the Russian/ Chinese/ Iranian/ N Korean (delete as/if necessary) state-employed Black Hats are spitting blood waiting to see if their little compromise software, sitting quietly in a dark corner of a server somewhere, will be uncovered by the security measures introduced following this mal(ware)arkey.
Top Russian Submarine Design Bureau Hit By Cyber Attack With Chinese Characteristics
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
