Hackers pop German steel mill, wreck furnace

Talented hackers have caused "serious damage" after breaching a German steel mill and wrecking one of its blast furnaces. The hack of the unnamed mill, detailed in the annual report of the German Federal Office of Information Security, was pulled off after a victim fell for a phishing email. Hackers then pivoted to the …

[Anonymous Coward]

I can't see whats wrong...

"a feat that should not be possible according to best practice that requires separation between industrial control systems and the public internet."

Well it was separated; a windows XP SP1 machine sat in between the furnace and the public internet, what's wrong with that?

[Ole Juul]

Is there something missing?

According to the article, the report says "The result was that a blast furnace could be shut down," but I don't see where it says that one was, or that anything was damaged.

[Anonymous Coward]

Re: Is there something missing?

Page 31:

"Schadenswirkung

Es häuften sich Ausfälle einzelner Steuerungskomponenten oder ganzer Anlagen. Die Ausfälle führten dazu, dass ein Hochofen nicht geregelt heruntergefahren werden konnte und sich in einem undefinierten Zustand befand. Die Folge waren massive Beschädigungen der Anlage."

Blast furnace couldn't be shut down in a controlled fashion. Massive damage.

[vagabondo]

Re: Is there something missing?

Just switching off a furnace full of molten metal and you get a massive slug of scrap metal wrapped in a fire-brick jacket. It takes a long time to remove the solidified metaland build a new furnace.

[Tim Worstal]

Re: Is there something missing?

Yeah, I was thinking about that. A blast furnace that is FCUK'd is a very big deal indeed. Hundreds of millions of dollars. and there aren't that many of them around either, not even in Germany.

An arc furnace would be a very different thing.....some millions, possibly tens of millions.

My German is completely terrible so I've no idea where the error is, if there is an error. Was it a blast furnace damaged? Because one ruined would have been front page news. Or an arc or plasma furnace ruined, which wouldn't make the front page?

[Destroy All Monsters]

Re: Is there something missing?

My German is completely terrible

Unfortunately there is scant details in the report. But it clearly implies targeted sabotage ("hackers worked forward into the production networks ... they showed detailed know-how about the attacked industrial process and control systems"), not someone crashing a process by kiddie-oops.

Could be the Thyssen Krupp plant in Brazil, who knows: Panne zur Unzeit im ThyssenKrupp-Werk in Brasilien. It blew out in 2013, exactly when there was talk about selling it. Reason given: "Instability of Processes", 500 million EUR for reparations, 5% shaved off the shareholder value. Noice.

The Galactica Solution starts to look very good.

[Anonymous Coward]

@vagabondo

Dumb question I'm sure, but if you turn off a furnace full of molten metal and it solidifies, can't you turn it on and re-heat that solidified metal? Does the cooling metal cause something to crack so it can't be safely turned back on again?

[Destroy All Monsters]

Re: @vagabondo

If it is a blast furnace, the "chimney" is essentially the whole of interior, which is continually refilled from the top with layers of carbon and other remainders of supernovae. If the interior is full of solidified cold slag, it's game over: There is nothing to refill, the chimney is gone.

[Longrod_von_Hugendong]

Why the fcuk...

Does the control software allow this to happen? Its like in bond films where the machines have so much leeway they can kill and mash a human at like 2/3rds power. Seems a dumb idea to have that in software.

[Anonymous Coward]

Re: Why the fcuk...

More importantly, why the fuck does a furnace NEED to be connected at all????

Seems we are connecting things up for the sake of it, not for any good technical reason sometimes.

[Anonymous Coward]

Re: Why the fcuk...

Of course you can shut down industrial plant via software. You don't actually need to turn the power off to cause massive damage. Just fiddle with the code to shutdown part of the machine will do the trick. Or disable the controlled shutdown procedure. Or reboot a PLC. The possibilities are endless, especially with a blast furnace. Imagine what you get when you turn off the heat with molten steel inside - a massive lump of steel that has to be hacked out by hand, damaging the lining that all has to be replaced. Plus you have to wait weeks before it's cold enough to work on.

We make industrial control systems. If you know what you are doing, you can break any machine.

These people obviously were well versed in Siemens PCS7. After all - it was Germany, so I can't see it being anything else.

[Anonymous Coward]

Re: Why the fcuk...

@AC, In your industrial control systems do you include such things as physical key switches that are required to be human activated before any actual switching can take place? We have had to add such switches to almost all of the new control systems that our client has bought because the suppliers couldn't/wouldn't see the necessity for them.

This incident has proved the requirement of our client was eminently justified.

[thames]

Re: Why the fcuk...

With many large process industries there is a long, complex procedure involved in shutting down the equipment and the shutdown may take hours or days. Some processes contain enormous amounts of heat or chemical energy which either power the reaction or are a result of it, Due to the laws of physics, the realities of economic efficiencies, or the need for compliance with pollution control regulations, the reaction may be operating on the limits of the abilities of modern science and engineering. If you simply "pull the plug" on the equipment, that energy is still present, but it isn't being controlled. The safety systems may save the lives of the operators, but the equipment itself may be damaged beyond repair.

As for why production equipment is connected to office equipment, it's so the ERP system can get real time production data and feed customer orders into the plant. Modern JIT production methods mean that when you click "buy now" on Amazon, that purchase ripples all through the supply chain.

The biggest problem in industrial control systems is that almost control system manufacturers are strongly wedded to highly proprietary hardware, software, and especially communications protocols. Their idea of being "open" is to sell you (at eye watering prices) a special driver or piece of middleware to let you talk to it. These drivers almost always involve running a version of MS Windows that is several versions out of date, usually using COM/DCOM, and often only work if all Windows security features (such as they are) are turned off.

Most SCADA systems from the big vendors are descendents of 1990s code bases and all the vendors had a big love-in with Microsoft which meant that they are locked in to all the wonderful "brain wave of the month" features that Microsoft added and then deprecated over the years. The industry moves so slowly that they often don't adopt the latest whizzy fad from Microsoft until after it's out of date.

The vendors of industrial control systems are very adamant about not wanting anything to do with security. They see this as being strictly a customer problem. Given the highly proprietary nature of these control systems, the customers often have very little flexibility in how they are deployed.

The biggest problem really is the lack of open data access to the controller CPUs (the boxes that run the equipment). You are given very limited options in how you do this, because the vendors are deathly afraid that somehow, somewhere, someone might do something useful with their equipment without the vendor extracting their pound of flesh.

[Anonymous Coward]

Re: Why the fcuk...

"The biggest problem in industrial control systems is that almost control system manufacturers are strongly wedded to highly proprietary hardware, software, and especially communications protocols."

Standardise interfaces not implementations.

GM, Boeing, and other big players saw this issue coming almost three decades ago.

The answer was to open things up, so that simple secure multivendor interoperability could become the norm rather than the exception.

Go read about MAP (Manufacturing Automation Protocol) and TOP (Technical Office Protocol) and such like, back in the days when it was cool to talk about Computer Integrated Manufacturing as well as CAD and CAM. This was ages ago, even before the days of da Interwebz, so relatively little stuff is readily available online.

Many (most?) of the major automation and SCADA vendors joined the party, even Siemens (who also had their own conceptual equivalent of MAP, Siemens AP). Some computer vendors came along too. The Microsoft camp obviously wanted to go their own way with the aforementioned COM/DCOM etc, but MS were happy to recruit key people from other industry players, as you do (buongiorno Lorenzo! Come sta?).

But widespread implementation would have required lots of investment in various places, with limited short term benefit, so the ball was kicked down the road, and in due course the MAP and TOP products largely disappeared. After all, nobody's ever going to need more than Modbus are they.

Three decades later, people are visibly paying the price.

[thames]

Re: Why the fcuk...

"Go read about MAP (Manufacturing Automation Protocol) and TOP (Technical Office Protocol) and such like"

I remember MAP/TOP. It was a complete flop. It was supposed to be a competitor to Ethernet, but went pretty much nowhere.

There's a "standard" for industrial communications based around part of IEC-61131, but the industrial automation market is like one of those children's events where everybody has to get a prize. They simply took the 7 different and incompatible protocols from the dominant vendors and declared them all to be "the standard". If you implement any one of them, then you are compliant with the official IEC standard. The fact that these 7 different protocols are incompatible and cannot communicate with each other and most are highly secretive and proprietary matters not a bit. All the vendors really wanted was to "open wash" their existing product lines, and that's what they did. They did the same with other things as well by the way.

About the only industrial protocol which is both widespread, runs on modern media, and is actually open is Modbus/TCP. It's an extension of a protocol that originated on RS-232 back in the days before the market leaders thought of locking everything up. It's also simple enough that pretty much anyone can implement what they need without a massive software engineering effort.

However, most of the major vendors either won't support it, or will support it only in very limited ways. The main concern for the major vendors is that the big money is in the I/O (the "peripherals"). An open protocol that would let you mix and match networked I/O would put a lot of overpriced hardware vendors out of business. They have vendor lock-in and legacy customers down to a science, and they don't want to open the door to competition.

In many ways the industrial automation market is like what the mainframe market was like back in the 1960s. Each mainframe vendor had its own hardware, peripherals, OS, programming languages, and development tools, and guarded "their" customers jealously. The automation market is like that, only more so.

For exchanging data between the factory floor and the IT systems (e.g. ERP), what's really needed is something like an HTTP REST protocol with JSON data. It would piggy-back off existing IT technology, be well understood, and we would know how to secure it. All those things however make it anathema to automation vendors.

[Anonymous Coward]

Re: Why the fcuk...

"I remember MAP/TOP. [snip see later] It was supposed to be a competitor to Ethernet,"

Then I'm afraid you remember it a bit wrong, but as I said, reference material is hard to find. The original MAP people included a group of lower layer companies whose names I forget and can't easily find, who were obsessed with having MAP based on 802.4 token bus cables and interfaces, which was a generally daft idea (unless you were an 802.4 vendor).

Siemens and many other sensible outfits said "we can do this using industrial strength Ethernet". Siemens as far as I know went ahead with their AP protocol suite over Ethernet, addressing exactly the MAP requirements. Computer vendors such as DEC would happily talk to either or both, using the same software.

"Each [automation] vendor had its own hardware, peripherals, OS, programming languages, and development tools, and guarded "their" customers jealously. "

Exactly, and that is exactly what MAP and TOP were supposed to address (cabling aside).

The important bits were not how the devices were physically connected (802.3 vs 802.4), but what the connection allowed you to do, in a device-independent, vendor-independent, interoperable manner (without the lock-in which otherwise arises, which you clearly recognise). Sadly, lots of people failed to see beyond the cabling.

The MAP/TOP idea was that if you didn't play according to the MAP/TOP rules set by GM, Boeing, and other significant automation purchasers, you didn't sell automation (devices or applications) to them. Once you've got standardised interfaces and protocols, the rest becomes relatively simple.

"It was a flop".

It was undeniably a flop in commercial terms. It addressed a device<->application integration (not cabling) problem that most people with chequebooks didn't see a need to fix, and those that saw the need didn't like the price at the time. You clearly see the problem. Maybe now computers and networks are cheaper, the people with the chequebooks would appreciate the situation too.

"For exchanging data between the factory floor and the IT systems (e.g. ERP), what's really needed is something like an HTTP REST protocol with JSON data."

Maybe. But I hope you're not suggesting putting that functionality in the factory floor devices and making it directly accessible from the corporate desktops. There lies anarchy. On the other hand it could quite happily fit in a secure "shop floor gateway" box which has the shop floor kit on one side and the corporate network on the other side and some clever software in the middle.

These boxes were around in the late 1980s, using MAP protocols (sometimes over 802.3, sometimes over 802.4), Siemens AP over 802.3, and other proprietary protocols (such as the inevitable Modbus).

Back then, this kind of functionality required minicomputer or workstation class hardware and a real OS (this is before Windows NT) to do a worthwhile job. There were similar compute requirements in the network interface+software in the automation device. It didn't stand much chance of success. Collectively this led to interesting oddnesses like Allen Bradley putting a rebadged VAX in the same backplane as a high end PLC (the Pyramid Integrator) so your enterprise apps (which weren't called that back then) could run on the shop floor boxes.

Compute power and network performance is not the problem it was back in the 80s, and the alternatives to Windows are more acceptable these days (or should be) than they have been for a decade or two.

Sometimes the solution is there, waiting for you. Or at least it might have lessons to offer to those who might be tempted to re-invent a few wheels.

Some of the people who did the useful work in the 1980s might not have retired yet, if you're quick.

[thames]

Re: Why the fcuk...

Anonymous Coward - "The original MAP people included a group of lower layer companies whose names I forget and can't easily find, who were obsessed with having MAP based on 802.4 token bus cables and interfaces, which was a generally daft idea"

As I recall it, the MAP/TOP stuff included hardware and protocol, but when the hardware failed in the market, some vendors tried to salvage the software layers. That too, died a death. Not a lot of information survived to say where it all went wrong. There was a small group at GM promoting open ideas, but they saw it as a way to make the big vendors compete with each other through inter-operable standards rather than producing something that was independent of vendors. That went pretty much nowhere, and their efforts to base everything on Microsoft products (especially for the soft logic system) simply introduced yet another proprietary vendor into the mix - and one who wasn't really interested in the market to begin with. It was the wrong approach to the problem and doomed to failure.

Anonymous Coward - "Maybe. But I hope you're not suggesting putting that functionality in the factory floor devices and making it directly accessible from the corporate desktops. "

This is why I said to the "IT systems (e.g. ERP)". The raw data straight out of the PLC simply isn't something that can be digested by a desktop anyway. It needs to be distilled down into usable reports. But if you use genuinely open methods that are borrowed from the IT world, then you can use all the standard IT tools and techniques to secure it. Expecting the automation world to come up with their own parallel but incompatible imitations of everything in the IT business and expecting it to be secure is hopeless.

Anonymous Coward - "Back then, this kind of functionality required minicomputer or workstation class hardware"

At the time the first PLC was invented, people were doing programmable industrial control on minicomputers, with the PDP-8 being a popular choice. The early Modicon and A-B PLCs were based on minicomputer bit slice processors, the AMD 2900 series being a popular choice. Siemens had their own M-Code logic co-processor CPU. Microprocessors were simply too slow.

Anonymous Coward - "Collectively this led to interesting oddnesses like Allen Bradley putting a rebadged VAX in the same backplane as a high end PLC (the Pyramid Integrator) so your enterprise apps (which weren't called that back then) could run on the shop floor boxes."

I investigated the Pyramid, but it was really too inflexible and too focused on doing specific things for specific customers and ludicrously expensive. A-B liked that sort of solution because it kept what you did with the PLC under their control. They had lots of expensive boxes that did specific things, but very limited and very slow options for simply getting the data out.

One option was putting a Datahighway+ card into a PC and using that as a gateway, which meant you had these gateway PCs scattered about the plant running hacked-together bridge software so you could get the data back out to send somewhere else. These tended to be poorly maintained so far as patches and updates were concerned, and despite the reputation of PLC vendors for long term legacy support, their record was poor when it came to products in this area. You haven't lived until you've had to babysit an old PC running MS-DOS with a DH+ card, a driver which talks using DDE, and a copy of Lotus-123 being used as a database glued together by macros (because that was what A-B sold the equipment builder), all of which were obsolete and could only be repaired by re-engineering the system from scratch.

[Anonymous Coward]

Re: Why the fcuk...

>As for why production equipment is connected to office equipment, it's so the ERP system can get real

>time production data and feed customer orders into the plant. Modern JIT production methods mean that

>when you click "buy now" on Amazon, that purchase ripples all through the supply chain.

Sorry, not good enough. A blast furnace can take hours to get going. Don't tell me the orders need to get to the control system in a microsecond when they could quite easily and with no effect on production send someone down to the factory floor to key in the parameters. Its just Grade A lazyness. I feel sorry for the company but frankly it serves them right. Some people just have to learn lessons the hard way.

[Boris the Cockroach]

Re: Why the fcuk...

"The industry moves so slowly that they often don't adopt the latest whizzy fad from Microsoft until after it's out of date."

The reason is simple.... the equipment costs lots and lots and lots. and the payback on the machinery is a minimum use time of 10 years, more often than not in excess of 15.Eg the smaller 5 axis CNC mill I use costs £150 000 ... and its a cheap one and its been in service 8 years.

Plus the minor fact that the software has to run on reasonably reliable chippery that is well understood by designers/builders so its not going to return a divide by zero error when asked to do Y= (COS(theta)*Tool radius)/ (PI*Workpiece diameter)/360* the moon's current phase angle.... or something even more complex

So its pentium 3's all round at the moment running a custom Linux OS from 1999 ..

In any case it still does'nt excuse the lack of an air gap and filling up the USB holes with glue that the german company did'nt do

[PNGuinn]

Re: Why the fcuk...

If what you say is true, and I'm not for a moment suggesting it's not, (although by all the normal rules of sanity it OUGHT to be utter bx) it seems to me that some steel manufacturer has a mighty good case suing someone for providing a control system totally unfit for purpose.

Pass the Popcorn.

[Tom 13]

Re: some steel manufacturer has a mighty good case suing someone

Except the control system manufacturer probably has another model or variation on the installed model that would have provided the necessary functionality and the steel manufacturer opted to purchase the cheaper version.

It all comes down to the risk analysis and bean counters. That's completely at the discretion of the steel manufacturer.

[DrXym]

Re: Why the fcuk...

"Does the control software allow this to happen?"

Because the control software isn't designed to operate on an open network.

It's meant to run on a closed network because there are far too many ways to interfere with it's operation. e.g. PLC A might be listening to temperature readings from PLC B which are sent using a standard protocol like profibus. So an attacker could screw up what A does by jamming false readings into the network. Or countless other attacks.

It's not even necessary that this interference is malicious. Imagine if someone was streaming Netflix over the same network and interfering with safety controls meant to protect someone from injury.

So it's meant to be closed for certification and safety reasons. The most likely reason it would not be (legitimately) closed is if there is something bridging the network to another e.g. a PC might have two network cards so it can capture data from the inner network to provide to the outer one, e.g. performance metrics. This is a very hard problem to solve and probably requires the PC to be protected by a high restrictive firewall that provides least privilege access to the data it is supposed to be exposing. It's easy to see how that might be screwed up.

[Grikath]

Re: Why the fcuk...

why the fcuk not? Given the setup and inherent workings of steelworks the controlling machinery is already heavily networked and operated/monitored from a remote control room. It takes but one beancounter insisting that the machine that actually handles the controls also is quite capable of doing the in-company email and "intranet" so a separate box is Unnecessary Expenditure and you are quite literally down with your pants on fire.

Been there, done the Facepalming, and yet It Was Done, and not as an isolated incident either.. Production security to Beancounters is personnel not walking off with a packet of soup from the Management restaurant, not the very real possibility of a whole plant borking because the "intranet" is compromised and crashes all production systems.. They simply can't be bothered until it inevitably happens.

[PNGuinn]

Re: Why the fcuk...

"They simply can't be bothered even WHEN it inevitably happens"

There - corrected it for you.

Beencounters are generally NOT there to save the company money or even keep it afloat. They're there to massage their own pathetically self inflated opinions of their own importance and ability. For which they simply MUST grow their own empires.

Generally endowed with a limited ability to count beans in isolation, they usually have no inkling that there are many different kinds of beans, nor how these beans interact.

Good management will keep 'em locked - preferably chained - up in Beencounter Central, supervise how and what they are counting and very carefully analyse the "results".

Running a business takes business accumen and management skill. Skills beencounters generally don't have.

When the BCs start running the company its time to bale out. The result is inevitable.

[Anonymous Coward]

Breaking News

The same group of hackers now claims responsibility for removing all the commas from the last paragraph of this article.

[ukgnome]

Re: Breaking News

> More importantly, why the fuck does a furnace NEED to be connected at all????

It's not fun been this hot and lonely, that's why

[Anonymous Coward]

Re: Breaking News

It's not fun been this hot and lonely, that's why

It's not fun BEING this hot and lonely, that's why

Come on man, this is basic English.......

[jake]

Gawd/ess.

When will manglement finally understand that there is a reason cognizant network engineers demand an air-gap between TehIntraWebTubes and SCADA?

Hopefully never ... as a consultant, I make large amounts of money cleaning up ;-)

[Anonymous Coward]

Re: Gawd/ess.

Air gap or not, what about the PLC programming tools (which are, sadly, typically Windows based)?

Some of the time you want the progamming tools connected to the corporate LAN, for perfectly valid reaons. These boxes may not look like PCs and may not have all the "PC security" tools IT departments like to use with Windows, but they'll be just as vulnerable. At other times the same boxes are necessarily connected to the automation LAN.

Infected PLC programming tools is one of the ways that Stuxnet propagated. It's sad that people still think airgaps are an answer. Getting rid of the Windows dependency in places where Windows doesn't belong is a much more helpful strategy.

[DrXym]

Re: Gawd/ess.

Stuxnet wasn't a drive by attack. It was specifically crafted to exploit a specific target. If the target was running Linux then it's likely that the attack would achieve the same end by different means.

[Anonymous Coward]

Re: Gawd/ess.

"Stuxnet wasn't a drive by attack"

OK

"It was specifically crafted to exploit a specific target"

And it seems to have done it quite well.

"If the target was running Linux then it's likely that the attack would achieve the same end by different means".

Maybe, though others may not necessarily agree, especially if the Linux was sufficiently cut down and hardened, to reflect the dedicated use of a programming panel rather than a general purpose desktop?

But who said anything about Linux anyway? Why not (random example), QNX, or maybe another lightweight but functional-enough OS?

[Anonymous Coward]

Re: I too know nothing about industrial systems

But I know that Windows is shit!! Fuck relevancy!

[Destroy All Monsters]

Re: I too know nothing about industrial systems

"If the target was running Linux then it's likely that the attack would achieve the same end by different means".

More realistically, people who are so insecure about computing that they run Windows For Critical Systems (WINFOCS) are already a bit sickly from the get-go.

[jake]

@AC (Was: Re: Gawd/ess.)

"Some of the time you want the progamming tools connected to the corporate LAN, for perfectly valid reaons."

What part of "air gap" do you not understand?

Stuxnet demonstrated that humans and SneakerNet prove that most corporate/national security have absolutely zero clue about security.

As a side-note, my programmable logic controllers are quite adequately managed by one of the BSDs (and sometimes Slackware). Microsoft products aren't used in these here parts.

[Anonymous Coward]

Re: @AC (Was: Gawd/ess.)

"Microsoft products aren't used in these here parts."

Lucky you. Whose PLCs are you using that enable you to eliminate the MS dependency typically found in the programming tools?

[jake]

@AC: Was: Re: @AC (Was: Gawd/ess.))

"Whose PLCs are you using that enable you to eliminate the MS dependency typically found in the programming tools?"

Uh ... all of them?

Your lack of systems, computer and networking literacy doth not make 1D10Ts marketing nuts & bolts as "exclusive" a reality.

HTH, Happy Horrordays.

[Admiral Grace Hopper]

South Korean Nuclear Plant

When I heard the news this morning about schematics being dragged out of a South Korean nuclear power facility and heard the assurances that the control systems were not hackable my thoughts turned to air gaps and firewalls. Leaving any system controlling a sufficiently large amount of energy open to attack seems somewhat careless.

[ilmari]

As a C guy wrangled into automation, I can onky concur with what people have said before. When I first encountered it, it was a bit like finding a tribe of cavemen in central park.

Siemens came out with a new version if the software I have to use. Haven't tried it yet, but I heard it now takes less than a day to convince it to jnstall on windows other than XP SP1 32bit. The actual install, obce it starts, still takes 6 hours, as the installer consists mostly of a wrapper around a few thousand .bat scripts. Woe if you accept the install directory proposed by the installer, because the installer can't actually install properly if destination directory contains spaces or any other "funny" chars.

The installer itself is distributed in half a dozen .zip files. Not split inti individual components, you have to recreate the proper directory structure and unzip each i ti the correct directory.

Their "cheap" stuff is actually less bad, with

both linux and windiws versions, and $150 piece of hardware with inputs and outputs, ethernet port and built in webserver that apparently does something useful. But, despite being able to run a webserver, you're still limited to the equivalent of 400 lines of code (let's say python level of density).

[jake]

@ilmari

Me, I just buy the parts & talk to them one-on-one via a serial MUX (and modems, where needed) using shell scripting. It's worked for me for thirty five years. Corporate "necessary management software" has always been a joke ...

[Anonymous Coward]

Order hierarchy

Could there not be a weight applied to information and instruction?

One way read-only info from sensors to some logging program, lowest level.

Control commands within expected envelope, normal.

Control commands outside the normal operating envelope require manual conformation.

Automation seems to start with an "everything" connection then back off, close down, filter as problems arise.

In a way it is the same with Email, why should I be able to except anything but a certified base document of any one format. Get Adobe to say "this type of PDF is safe with a base viewer" and requires a 2MB viewer, right now there is no standard limit to the complexity and the reader is larger than some operating systems.

Safety critical IT needs to be seen as a field apart, where bells and whistles are not required don't accept them.

[Doctor Syntax]

Separate LANs

At my last gig (several years ago, given that I'm retired) there were several separate LANs. The nature of the business was secure processing of personal data on behalf of clients. LANs for production systems were kept separate from the office and development systems. If we needed to inspect live data for any reason (to find out how the clients had failed to pass well-formed XML this time) we had to use a specific PC in a secure area under supervision.

It's what was needed to be done there and it's what needs to be done more generally. If data has to be passed across networks then implement some secure bridge that only passes carefully sanitised data, e.g. CSV rather than spreadsheets, checked against attempted buffer overruns etc.

It may seem inconvenient to have air gaps. However having a furnace full of solid steel, yoru customers' credit card details stolen or all your servers' contents dumped online brings home the true meaning of inconvenience.

[DrXym]

Re: Separate LANs

Typically there is no access at all to a network controlling machinery. The only plausible reason I can think they didn't is if they had a PC sitting on 2 networks to publish a report or visualize what the hardware was doing.

The typical way to protect that PC would be to stick a firewall in front of it, define a bunch of explicit services on a port and filter out everything else - much like what might happen with a payment system or authentication server. So basically nothing but the request reaches the machine and nothing but the response leaves the machine. I'm guessing that random factories don't realise they're as much a target to malicious hackers as an ebusiness is which is why they occasionally turn into victims of attacks.

[Tom 13]

Re: nothing but the request reaches the machine

A reasonable solution from a technical standpoint, but...

In walks the bean counter's IT Security cousin. He's got his check list of "Best Practices" and sees that your carefully thought out system isn't getting it's MS updates. So now the machine has to be connected to the internet on port 80 and ...

Because I've met this IT Security person, and some worse actually. Thankfully I don't work in SCADA or other critical infrastructure work, but it still sends chills down the spine.

[Anonymous Coward]

Re: nothing but the request reaches the machine

"now the machine has to be connected to the internet on port 80"

Don't rate that cousin much. Surely the MS-approved route would be to use the site's Systems Management Server as a local source for tried tested and proven patches (tee hee). Or is it Systems Center Configuration Manager these days?

Whatever it's called this week, that would at least eliminate the need for a direct port80+friends Interweb connection from the critical systems.

May leave a few equally interesting avenues still open though.

[chris 17]

Re: Separate LANs

All bets are off if the separate LANS are just VLANS on the same infrastructure.

You'd want to be using fibre to the host (not as expensive as it sounds) and ensure complete physical separation. I've worked in so called secure locations where those in the know (as they've been there longest) will argue these systems are air gapped despite sharing infrastructure somewhere in the path which they deem as acceptable as they can't see it.

[All names Taken]

SCADA

With interconnectivity going worldwide the supervisory control part of scada can also go worldwide?

[kunjiri]

com/dcom and fiewwalls

People who have insight into computer/network security seldom get a say in the design and deployment of industrial controllers. However, the end users are pretty sure about what they want: a windows system, preferably xp, with lots of proprietary software including a Visual studio. Though the firewall by default blocks all traffic, most of the systems are tweaked just to facilitate data exchange with a controller..you know that com-dcom stuff may get murky at time.... In fact, when I realised the gravity of this, sitting in a mill with with the feed flying past each rolls in a few meters per second rate, oh my god...that mill could be literally hacked with a piece of proprietary s/w, and with just physical access to a plant wide network. Nowadys, even open source opc implementations too may facilitate that. As many pointed out, we have to move to other platforms as early as possible...or to the opc ua stuff...and force the vendors to shun their proprietary protocols for open royalty free versions...

[Anonymous Coward]

Re: com/dcom and fiewwalls

"People who have insight into computer/network security seldom get a say in the design and deployment of industrial controllers."

It's a long time ago, but my recollection is that the MAP/TOP stuff included various security capabilities, largely on the network (not system) side of things. This was in the era of PC/XT, VMS V3.7, and System V on 68k (back in the days when fd=open() had 2 parameters or 3 depending on whose UNIX it was). Doing networking right was neither cheap nor widespread in those days, but some people were thinking about it.

"the end users are pretty sure about what they want: a windows system, preferably xp, with lots of proprietary software including a Visual studio"

Let's take that as a given (not sure we should, but for simplicity, for now)

Why does that mean that Joe Public needs a connection from their desktop PC to go **out to the plant floor** for their data? The software technology exists, and has done for decades, for all the data they need to be accessed securely and safely in a managed image of the shop floor data, which itself is connected to the shop floor network by a securely managed firewall/gateway (physical or conceptual). Thus Joe Public's desktop PC **cannot** connect direct to the shop floor.

Obviously doing it this way has up front costs which appear to be a couple of quid more than the usual anarchy. But anarchy eventually has a cost too.

[Steven Jones]

It could be worse

Possibly just as well that the Germans have decided to close down their nuclear power plants if they can't keep critical control systems safe from hackers. Although I would hope that things would at least fail safely, even if not cheaply.

nb. I initially found a few elements of this story not entirely plausible, but as it seems to be official then so must it be.

[Wilseus]

Re: It could be worse

Possibly just as well that the Germans have decided to close down their nuclear power plants if they can't keep critical control systems safe from hackers. Although I would hope that things would at least fail safely, even if not cheaply.

As I understand it, and I am *not* a nuclear scientist, most sensible nuclear reactors are supposed to fail safely. One of the reasons Chernobyl went wrong so spectacularly is that it had a positive void coefficient, e.g. it needed constant intervention to stop it going out of control, rather than negative void coefficient where it'll just shut down under the same conditions.

[Anonymous Coward]

Re: It could be worse

The other, rather more exciting reason, was that they decided to test the idea of using the residual steam to power the pumps during a simulated failure, and they ran out of water...or that's how I think I understand the film that was made.

And the reason that this got through was that the people in charge were largely unqualified party nomenklatura.

Modern fighter aircraft are unstable and depend on computers to function properly, but the software isn't written by people who got their jobs through who they were at school with.

[Wilseus]

Re: It could be worse

The other, rather more exciting reason, was that they decided to test the idea of using the residual steam to power the pumps during a simulated failure, and they ran out of water...or that's how I think I understand the film that was made.

Yes, I think that's right. But if the reactor had had a negative void coefficient, the lack of coolant would have caused the nuclear reactions to cease, not spiral out of control - in such a design the presence of coolant actually *increases* the nuclear reactions, and similarly its absence causes the reactions to slow down which is obviously is a lot safer. With the Chernobyl design, the opposite is true.

[Tom 13]

Re: are supposed to fail safely.

Yes, they are. But that only accounts for all of the foreseen cases. Sometimes the miscreants or life are more inventive than the safety engineers.

For instance, TMI was a very close thing and from an unexpected source with no hackers involved. If the hydrogen bubble in the reactor had be just a bit larger, it would have exposed enough of the core so the water wouldn't cool the reactor. IIRC the moderating rods had already bent from the higher than expected temperatures and so would not have fallen as designed. Ironically, the critical point in the whole scenario happened about two hours into the accident (essentially before any news about it broke) and once past the critical point the danger fell off rapidly.

[Anonymous Coward]

Re: are supposed to fail safely.

Good point, and an explanation of why nobody in their right minds will be trying out thorium reactors in the developed world. Every undeveloped technology is considered safe by its proponents right up until the things they haven't thought of or known about start to happen. (This is also why I remain deeply sceptical about the "hydrogen economy"; a hydrogen Buncefield hardly bears thinking about, yet Buncefield happened many years into the exploitation of oil.)

[Anonymous Coward]

Re: are supposed to fail safely.

"Every undeveloped technology is considered safe by its proponents right up until the things they haven't thought of or known about start to happen."

Or the things the proponents were told about but chose to ignore because they were working on modern principles such as "faith based engineering".

[Alan Brown]

Re: are supposed to fail safely.

"a hydrogen Buncefield hardly bears thinking about"

Unlike petrol/diesel fumes, hydrogen is lighter than air. You should be more worried about LPG and CNG filling stations which in many countries are sited so closely together that one going up may set off the others (anything less than 2 miles separation is too close)

Personally I don't see a "hydrogen economy" every taking off. The raw gas is simply too reactive to be handled safely and reliably long-term in automotive environments, especially if the equipment is subjected to 10 years of the kind of abuse average drivers inflict on current cars.

[Alan Brown]

Re: It could be worse

"One of the reasons Chernobyl went wrong so spectacularly is that it had a positive void coefficient"

And another one was that the operators had shut down virtually all of the safety mechanisms in order to perform an unauthorised experiment.

That's without even going into the non-nuclear stuff such as what happens when graphite sitting at 800C+ gets exposed to oxygen (Sellafield, Chernobyl), or several tons of molten Sodium gets dumped in a basement (Monju) or plain old water is heated up to 1100C or so (Fukushima)

It wasn't very long ago that most reactors required external energy to SCRAM because the control rods had to be lifted into position instead of being passively dropped into place. Even a balance beam would have negated that difficulty but apparently that was too hard for the designers to come up with.

Virtually every single safety device in industrial processes today exists because someone didn't think of what might happen if it didn't exist (or in some cases is now mandatory after beancounters decided it was too expensive). It's not helped by H&S inspections coming from within the industries in question and people not daring to think about the "what if" scenarios.

https://en.wikipedia.org/wiki/Flixborough_disaster was a wakeup call, but not enough of one.

[Erik4872]

The Internet of Things will save us all!

I wonder what's going to happen when everyone's furnace, toaster, refrigerator and lights are connected to the Internet.

Stuff like this is crazy. I don't work in automation, but i do work in an environment that requires a network with no Internet access for some key functions. Part of the reason for this is that, like the SCADA gear everyone's talking about, we deal with wacky proprietary crap that absolutely fails to work when you start turning even basic Windows security on. The crap vendors won't (or can't in some cases) fix the problem - most of them are either the only supplier or one of two in the world. The problem is that all the stuff you can't secure *needs* to be air gapped. It's a major pain to deal with and requires nasty workarounds like sneakernetting files around, but it has to be done. This goes doubly so for us because some of these devices are actual end user machines with people sitting in front of them.

Everyone forgets that these Internet-connected devices, even though they communicate through standard protocols, have that server software implementing the protocols sitting somewhere. In the case of IoT things, that could be a cheap system-on-chip, or a Windows 2000 Server SP3 embedded computer with no upgrade capability.

[Yet Another Anonymous coward]

Invade Poland

With no evidence, the FBI have blamed Poland for the attack and intend to invade.

[Anonymous Coward]

Re: Invade Poland

No, sorry, you misunderstand. Anne Applebaum is married to a Pole, Poland can do no wrong.

It's Germany. Now, who could have a grudge against the Germans? Russia, Hungary, Bulgaria, Romania, the Czechs, the Slovaks...insert many other European countries here...David Cameron. The question is who doesn't have a grudge against the Germans.

By a process of elimination, it must be the Danes, because they have just laid claim to the North Pole and the US will be out to get them.

[Destroy All Monsters]

Re: Invade Poland

Danes are just the hat on top of Germany, so it's not them.

If steampunk had been a thing:

"Yes Mr. Göring. Poland. Well, I hear your conglomerates have a lot of steel & ironworks controlled by this newfangled "networking" thing? Really a shame if something should happen to them. Thank you now, my chauffeur will take me to Tempelhof. Ta!"

[Stevie]

Bah!

I'm sure it sounded like a good idea at the time, and I know Young People like to have Chips With Everything, but could someone tell me why in Bessemer's name a blast furnace had to be connected to any network, let alone the inter network?

Even net-connected lightbulbs make more sense that this, and they make no sense whatsoever.

[Anonymous Coward]

Welcome to the Industrial Internet ...

... and the Internet of Things (IoT) which is about interconnecting everything. Gartner says this will reduce industrial production cost by 30 percent, and also will allow much more flexibility ("lot of one") ...

Realistically, you can't keep industrial processes totally isolated any longer - one of the aspects is that you could run into grave risks of human error (Chernobyl was already mentioned in this discussion - that disaster could have been prevented easily if there had been suitable remote supervision from a central monitoring center).

However, the key is to keep those connections secure. This also requires very high robustness of the IT infrastructure versus malware and external attacks. Windows and Linux have thousands of known vulnerabilities (plus a unknown number of unknown vulnerabilities) and hence do not qualify for critical environments. Virtualization software is also risky, VMware does have hundreds of known vulnerabilities, same goes for the major Unixes. Critical processes do require platforms with much stronger protection against external attacks (and yes, there are some).

Of course, protocols containing vulnerabilities are another big risk. Anybody still believing Open Source is very secure because thousands of highly qualified useful idiots do work night and day without any pay to find and fix security bugs ? Heartbleed was a good wakeup call ...

It's extremely hard to build secure systems based on vulnerable platforms and cheap software that does run on any laptop. Critical processes do need a much more robust foundation - top management should understand this, and should keep the bean counters at bay.

[Destroy All Monsters]

Re: Welcome to the Industrial Internet ...

Some really fat brush usage going on here.

Windows and Linux have thousands of known vulnerabilities (plus a unknown number of unknown vulnerabilities) and hence do not qualify for critical environments.

You are confusing application-level vulnerabilities with OS-level vulnerabilies.

Running Linux may not be the perfect idea for a critical environment (this depends on the requirements), but "unknown vulnerabilities" is only one of them in the most demanding settings, as long as you keep it minimal and do not run random network-using applications (including sundry browsers) on top. Sure, there may be a packet-of-instant-p0wnage out there, but how high is that probability really?

VMware does have hundreds of known vulnerabilities, same goes for the major Unixes.

Break out of the sandbox, will you?

[jake]

Re: Welcome to the Industrial Internet ...

"Realistically, you can't keep industrial processes totally isolated any longer"

Mine are. You CANNOT get into my Winery/Brewery's computer systems from outside. Air-gaps and proper training of staff work.

I won't take a contract if a corporation doesn't agree with me about staff training.

[Anonymous Coward]

Re: Welcome to the Industrial Internet ...

Chernobyl would not have been prevented. It was the supervisory people that carried out the stupid experiment, and once things had started to go wrong it was already much too late. In the same way with Fukushima, it was, according to some sources) face-saving bureaucrats in the Japanese government that prevented the rapid response (helicopter borne emergency pumps) that could have prevented a disaster.

In the opposite example, at the end of WW2, Wenck interpreted his orders rather creatively so as to rescue as much of the 9th Army as possible from the Russians.

There should always be a firegap between bureaucrats and the sharp end, manned by the people who actually know how things work.

[Alan Brown]

Re: Welcome to the Industrial Internet ...

"It was, according to some sources) face-saving bureaucrats in the Japanese government that prevented the rapid response (helicopter borne emergency pumps) that could have prevented a disaster."

Those face saving bureaucrats prevented disaster being averted on several occasions. It was only when the senior engineer in charge on the site said "fuck this shit" and started doing the right thing instead of "just following orders under protest" that Fukishima's potential for Real Nasty Stuff went away - and even then he made a couple of bad calls, such as retaining vented hydrogen in order to allow its radioactivity to die down before release. He was qualified to sort things out. His superiors were not - but they were giving the orders.

Incidentally, collossal fuckups by face-saving (and faceless) bureaucrats in the japanese govt are hardly a new thing. All but 4 survivors of JAL 123 died because the japanese govt refused to ask for assistance and blocked USA search'n'rescue personnel from going onsite, despite them having located the crash within minutes and being overhead with rescue teams ~10 hours before any japanese response teams could be mobilised - see https://en.wikipedia.org/wiki/Japan_Airlines_Flight_123