What to do about inherent security flaws in critical infrastructure? The latest threat security research into operational technology (OT) and industrial systems identified a bunch of issues — 56 to be exact — that criminals could use to launch cyberattacks against critical infrastructure. But many of them are unfixable, due to insecure protocols and architectural designs. And this highlights a …
Its a choice for the manglement really
1. Air gap the critical systems from anything internet related (glueing up the USB ports too), then only applying updates from a secure scanned laptop
2. Leave everything as is , as its much more easy to download the production numbers from the line to your desktop PC than to dial the line supervisor and say "how many widgets made today"..... after opening the urgent e.mail from 'accounts' that has a PDF.exe file attatched.............
I know which choice our manglers would pick....
From what I have seen, corporations in manufacturing look to pay as little as possible to the engineers. They are like persona non grata, they can't complain and unions like to channel the anger of the floor workers at engineers - often telling them you make so little, because engineers take the most of the budget! And look they just sit there and do nothing! Of course, for a fat wad of cash under the table, they'll never say that actually corporation does that on purpose. It's better than workers behave like crabs in the bucket and not see what's really going on.
That being said, it often goes like that: "Steve, you did some programming right? Would you be able to fix this machine here? It seems to be discarding too many components that look just fine! The code is on that computer in the corner, unfortunately we lost documentation, but it's just Siemens S7, so my dog could do that but it's too fkin lazy. Just do it over the weekend and take Thursday and Friday off. Thanks mate!"
Many of these protocols run in air-gapped environments which limits the utility of many of these exploits to requiring somebody to be physically on-site in environments full of deadly hazards. If necessary, all of these protocols can be wrapped in infrastructure that secures them all the way down to at the wire level. Also keep in mind that simplicity can be a physical security / safety feature. If a connection between two devices suffers an authentication failure and that failure, say, causes an explosion at a chemical plant, was the addition of authentication to a system that could only be exploited at the wire level by cutting into conduits really a smart trade-off?
Modbus "has no security" in the same sense that "JSON has no security". Both are at their heart just data formats. Transport security is something provided by the network layer.
Modbus/RTU started out as an RS-232/422 point to point serial protocol. There was no security because none was required. Transmission over leased line phone communications depended on the phone lines being secured by the provider.
Modbus/TCP was created by just taking Modbus/RTU and tunnelling it over TCP. Since Modbus is just a data format, it doesn't really matter how the message gets there.
The logical step to add security to Modbus/TCP is to just tunnel it over SSL or something similar. The Modbus messages won't care about that any more than the web page you are reading cares about whether it arrived by HTTP or HTTPS.
All that's really needed is for industry to agree upon what port to use and how to handle certificates. Modbus itself requires no changes.
Of course this will do little in practical terms for security, since most of the real world security exploits seem to revolve around finding a Windows PC somewhere on the industrial network and exploiting that through bog standard Windows vulnerabilities.
there were things like MMFS and MAP/TOP. Not just from Modicon and friends, but in principle supported by many players in the industry. Maybe around the late 1980s?
Built on a full 7 layer OSI stack too, which meant that things like security and authentication and common data representation formats were largely designed in rather than added on as a band-aid/afterthought (without much thought).
Siemens AP had a "lite" version which used only four of the seven layers.
But no, there was no need for the OSI stuff, because IP v6 and associated vapourware was going to sort all the problems that needed sorting.
Except, 30+ years later, they're still not really sorted, and the lowest common denominator RFC addicts still don't see any need to do the job properly.
"most of the real world security exploits seem to revolve around finding a Windows PC somewhere on the industrial network and exploiting that through bog standard Windows vulnerabilities."
Ain't that the truth.
And thats part of the problem. Arseholes above see an network admin sitting there all day "Doing nothing" so they then look to cut costs and IT is always the first to be poked. The amount of marketing emails we see come in that go to managers "You can save thousands with the cloudy" "cloudy cloud cloudy cloud, savings". They then fall for the bullshit, fork out money for a consultant who just wants to sell you shit or knows a mate who can sell you shit. They point at IT as they want rid so you can point out they are talking bollocks.
Eventually you're "made redundant" only to hear a few months later the new, cost effective system, had a massive breach and "Lessons have been learned".
Sick of it.
It is more insidious than that.
IT seems to be one of those areas where everyone who has a 'puter at home or a smartphone suddenly thinks they are an expert when IT disagrees or objects with what they want.
Couple that with a senior management who hold similar views and the outcome is deep, sticky, brown and very very smelly.
Oh - and THEN they call IT to come clean it up and wipe their backsides for them.
We have a hybrid setup, part cloud, part onsite. They want to move a lot more to the cloud now. They were cloud before I started and assume it was assured it would "save money". Hybrid did (full cloud never will). One of the savings was flexible licensing with Office 365. Not using a license, remove it and you don't pay for it so you can just pay for ones being used.
Now we're tied in and can't get out, that is now changing, changing to the old setup that was in house, so not saving money anymore. We now have to say how many users we have and then pay for those licenses for the whole year.
Arseholes.
"One of the savings was flexible licensing with Office 365. Not using a license, remove it and you don't pay for it so you can just pay for ones being used."
Absolutely. The "month by month" may be slightly more expensive, but you can add and remove licences at will as staff join and leave. Very useful for interns and temporary staff.
In the mid 1990s I was sent to repair a cardboard box maker. The cough logic controller consisted of plug-in modules the size of a house brick with various switches and controls - pretty difficult to compromise I'd say.
Seriously, PLCs have always been a disaster waiting to happen.
Have they?
I'd trust a proper dedicted PLC from a manufacturer who has been in the business decades than any of these abortions built around Rasperry Pis or Arduinos etc and programmed by people who THINK they know what they are doing.
The problem comes when connectivity gets involved and security is sacrificed for "convenience" aka LAZY buggers who want to do everything via their effing phones. And when IT/security says no they "squeem and squeem, stamp their feet and throw their toys out the pram" until they get their way.
>> I'd trust a proper dedicted PLC from a manufacturer who has been in the business decades
Neither Siemens nor Honeywell seem to have any grasp of secure solutions. AD and Windows hardening is something that neither system can easily cope with, GPO exceptions galore.
Last year an updated Honeywell EBI building integration system was procured and one requirement for monitoring stations was Microsoft Silverlight - EOL last autumn.
>> ...than any of these abortions built around Rasperry Pis or Arduinos etc and programmed by people who THINK they know what they are doing.
Big companies employ only the best and brightest? I think you're misguided.
Please keep in mind Theodore Sturgeon's law.
I don't think you quite grasped the arugment I was making.
I also think that you miss the fact that, particularly with Pis, that they have (almost by definition) a whole raft of additional layers of software on Pi compared to (say) a Mitsubishi FX series. (such as the FX5). All of which can impact the overall reliability of it as a industrial controller.
As for "Big companies employ only the best and brightest? I think you're misguided".
Where did I state that?
I know small companies that (or think they do) employ the "best and brightest" who have a total lack of basic common sense. Don't even know the basics of using the correct rating fuse in a mains lead and think nothing of connecting a device, which has WiFi connectivity, to the internet (and controlled via an account on an external system) to turn the light inside it on and off using their personal phones. And then use that in an internal process.
Instead of just taking the bloody lamp out.
Who said anything about Arduinos etc? I certainly didn't.
It seems the point I was making was totally lost. The kit I was referring was completely self contained and intrinsically secure.Yes, it was old and slow (but so is making cardboard boxes) and it was common for these things to run 24/7. As soon as anything is software controlled you have introduced a weak spot. There are a (vanishingly small) number of PLCs that make this better - writes to the program area are disabled by a mechanical switch on the PLC itself. If there is any outside contact apart from just reading settings there is still the problem of malicious data being injected.
On top of this you get ridiculous passwords used (especially when HEX is required), such as:
0001, 0002, etc, on a series of identical machines.
Then there's the fun ones:
CAFE
ABCD
DEAD
D0D0
Yes. I've seen all of these :(
I agree with you on that, although in an emergency I've used an Arduino to replace a small (obsolete) logic module 'reconfigured' by a forklift driver.
Also I admit to being about 5 years out of date, but at that time I regarded most PLCs I came across as merely the best of a bad bunch...
...and most importantly, the way they are used!
Interesting factoid. The ATmega328 has a built-in watchdog that can be set to perform a reboot if the code locks up.
"Also I admit to being about 5 years out of date, but at that time I regarded most PLCs I came across as merely the best of a bad bunch".
Like most things, there are The Good, The Bad and The Bloody Atrocious. There is equipment I'm still having to keep going with PLCs 30+ years old in them (Specher + Schuh SESTEP 290).
And when first encountering them - especially if from more typical computing/microprocessor background - they are one of hell of culture shock. Their roots in mechanical relays make the way they work seem alien.
The "program scan cycle" takes some getting used to.
"Interesting factoid. The ATmega328 has a built-in watchdog that can be set to perform a reboot if the code locks up."
I think a lot of microcontrollers designed from the ground up as embedded controllers have had for donkeys years.
Relay logic and digital logic modules however were very limited in functionality and were very, very difficult to debug when it came to finding problems in the machine. There's also a lot of things they simply couldn't do. There's a good reason they fell out of favour even for very simple applications.
A lot of the reason you can have a plethora of inexpensive high quality goods in abundance these days is because of advanced industrial controls.
The current Modbus protocol ... does not have authentication.
I recall working on a control system for a large industrial system back in the early 1980s. Forty years ago there was no such thing as industrial networking [*].
The Modicon programmable logic controllers were a marvel, they replaced a wall full of relays and switches with a smallish box programmed with a custom programming unit. (Inside was an AMD (!) 2901 micro-controller and a bit of RAM and a UART). It had industrial-strength electronic input-output, and was very well regarded.
A colleague was tasked with getting the serial-port on a PDP-11 running RSX-11/M running and talking the Modbus protocol to the Modicon. In Fortran.
So the notion of "air gap" was intrinsic to the design. Arpanet was just becoming Internet; commercial access was still narrowly restricted; the notion of connecting a factory to such a network was ... a fantasy.
[*] The Honeywell TDC 2000 system had a rudimentary interface to external systems; serial port if I recall correctly. Proprietary, of course. Our PDP-11 used the TDC-2000 interface to read the state of the plant and wrote commands to both the PLC and TDC-2000 to operate it.
There was nothing more satisfying than watching my software run up a 200 tonne load of wood chips and caustic soda to several hundred PSI and 200 degrees C with superheated steam at 1500 PSI --- and then stop to let the mix cook. In the Good Old Days....
The AMD2900 series were bit slice processors, not micro-controllers. Each was a 4 bit vertical "slice" through a processor. You would create an actual processor by combining multiple bit slice processors together with some glue chips to create a CPU.
Other PLC makers used them as well, including Allen-Bradley in their early PLC/2 series (which also had magnetic core memory).
Early PLCs used bit slice processors because the microprocessors of the day were too slow. As microprocessors became faster they displaced the bit slice processors in that market.
Before PLCs existed, mini computers were extensively used in some areas of industrial control. What the PLC added was an easy to use application specific programming language (ladder logic in most cases) and packaging in a format which allowed the CPU to be located with the machine and all its associated wiring instead of in a separate control room. This made it possible to use programmable logic in a wide range of industrial applications.
Seems to me the ITU, IEC and ISO should be falling over themselves to mandate basic standards for (I)OT security design.
Without that, the industry will save money by ignoring the problem. The ancient principle of caveat emptor (let the buyer beware) applies, right alongside the well-known "there's one born every minute", especially among technology consumers.
They have done. Its called IEC 62443
Some is still in progress but the component and system level is pretty mature. Of course in the states you have things like NERC CIP, and the power and water systems have there own standards, but 62443 is a pretty good starting off point for OT security
What people are missing is that each OT installation is, essentially, a separate case. When I worked for Continental Telephone (decades ago - before the internet, the PC, the mini computer) we got some phone records from exchanges that were still running Strowger switches, a technology developed in the 1890s and still in widespread use in the 1950s. Some small telcos we bought still used manual switchboards and phones (this was when the telco owned the phone). Fortunately, in those cases, there was a market for them as antiques which paid for part of the upgrades but replacing all our existing switches and phones would bankrupt the company.
Much of the OT infrastructure is in the same boat. Their security was focused on physical breaking and entering because their were no hackers and the phrase "breaking into a network" would have been meaningless to them. This is compounded by the original local ownership and local installation of the OT software of their choice.
The Incebe-cert article referenced in the story has a rather glaring error in its description of Modbus/TCP.
"Change from master/slave to client/server. According to the new specification, Modbus devices will now no longer be the master and the slave, rather it will change to a more computer language and therefore they are called client and server. The client would correspond to the traditional master and the server would be the final device, previously called slave."
The Modbus/TCP spec already uses client/server rather than master/slave and has done so from the beginning. I just checked my copy of the spec (2006 edition) and it says client/sever.
The reason that client/server was adopted as the terminology was to make it compliant with the terminology used by TCP, which in turn came from the analogy of a restaurant waiter and diners (server and clients) used in the IT industry to reflect the network configuration used there, where one server would have many clients. In an industrial network one client would have many servers (hence the master/slave analogy used)
Since the intention was to simply adopt office IT technologies wholesale instead of re-inventing them, the client/server terminology came with it.
The Modbus organization themselves have published a security specification which is referenced in the inciebe-cert article. Inciebe-cert seem to understand normal computer security, but they're clearly not that familiar with Modbus itself.
So to recap
About ten years ago it was highlighted that these systems have laughable security if any at all....mostly the latter.
Just now, another report shows that the industry has done nothing to remediate said systems or come up with new protocols and designs that are secure by default.
You were told ten years ago and you have done nothing since and are now whinging it is too difficult because there are too many things to fix, most of which can't be and yet you still installed the same old crap systems that were know to be insecure.
If you had started 10 years ago you could be shipping proper secure systems now and it would be a start, and mitigate existing ancient systems by air gapping them or firewalling them off, wht something when you can stick your head in the sand and do nothing and get paid for it.
I still remember a conversation I had with a conveyor system supplier about patching their Windows servers controlling the system, their answer was they don't support patching and we wuold need a dev system to test on, because everyone has a dev warehouse, we were also their first customer to request 2016 (in 2019) and they didn't know if it would work or not. They also have to run their software as a logged on user on a console session or it wont work.
These systems, companies and developers have fallen in to an archaic mentality and don't see the problem. If i was hacker I would be targeting them for ransomware because I bet there are running unpatched systems and have very poor security if any just like their systems.
"About ten years ago it was highlighted that these systems have laughable security if any at all....mostly the latter."
Many decades ago, actually. Try connecting to the gear that monitors The Beam at SLAC, for example. Or the controls for the Stanford Dish. Or San Francisco's Hetch Hetchy water supply. Or rather, don't bother. You can't. Grad students wanted to hook 'em up to the 'net back in the late '70s or early '80s. The sane among us put the kibosh on their plans.
Commercial interests of today, however, are truly insane. We tugged on their capes, and were shrugged off. We tapped 'em on the shoulder & were elbowed away. We pulled on their shirts, and were thrust aside. Some even kissed their boots, and were trodden upon. Our message was always the same: "Please, PLEASE, **PLEASE!!** don't connect SCADA kit to publicly available networking systems!"
But did they listen? No. They did not. The idiots.
On the bright side, those of us with a clue are making a pretty penny in our retirement, cleaning up the resulting mess :-)
Yes, I know, I've posted this or similar before (most recently 12 days ago).
Most of it could be protected using a computer the size of Arduino mini or something similar, which only has one job: Either VPN or STUNNEL or any other way making the connection secure.
It can be done, is just isn't done until something big happens. "I told you so" doesn't work well in such situations.
Modbus maybe, but a lot of industrial protocols are very timing sensitive so adding an encryption layer would create issues.
At the end of the day Modbus is an edge protocol and it is questionable whether add encryption will make things more secure unless you are trying to access your I/O remotely (which would be pretty stupid). It is similar to suggesting that all I2C busses have encryption on them
Which I2C bus is connected to the internet, and reachable there? Which industrial timing sensitive protocol is internet connected?
And even if they use TCP/IP, anything timing sensitive will run over dedicated lines, and the internet connection is just a side effect. You you VPN/STUNNEL the internet side, but obviously not the LAN.
You somehow mix things up where my VPN/STUNNEL suggestion cannot be applied anyway, by design, and does not need to.
I don't need a data security expert to tell me that MODBUS is not secure. Its not designed to be. Its security is based on physical -- plant -- security in that getting to the connections to intercept or mimic the protocol is likely to be difficult and potentially dangerous. Fortunately the majority of systems are designed with the assumption that components can fail, you don't just rely on a safety critical command executing because things go wrong in the real world -- sensors fail, actuators get stuck, stuff happens. PLCs can be compromised but not in the same way that you'd take over a PC running Windows -- they're often running a program that looks a bit like a very primitive version of TinyBASIC with not a whole lot of wriggle room for creative destruction.
(Stuxnet is quoted as a successful attack but this relied on two things -- one was physical access to the network, the other a failure to bake in safeguards into the individual units. As a rule you don't allow machines to accept commands that will allow them to damage themselves. Its easy to overlook but I don't think the Iranians will get caught by this a second time.)
MODBUS tends to be for slow speed plant control. There are other industrial networking protocols that are a lot higher performance. EtherCAT, for example, is widely used. It is, though, rather difficult to interfere with without the system recognizing a problem and going to a 'safe' state (raising the alarm in the process). In addition there are protocol enhancements specifically designed to improve machine safety. As a rule we only need to be concerned with interference with the controller, the application running the system, not the system itself. The security researchers may find the way things are done a bit alien but its worth looking at closely because it might explain why IoT protocols aren't exactly taking the world by storm.
Yes industrial plants (in most of the world) have hard wired interlocks/protections and will fail safe, but that can still be disruptive in the short term. Turn off the electricity to a water treatment plant and folks go thirsty pretty quick (hours/days). Luckily there are also contingencies for this including manual overides so I'm also not really worried. Your domestic smartmeter however - if anyone cracks that we are doomed....
This post has been deleted by its author
OT security is a challenge.
Firstly most security processes are based on IT systems where the priority is to protect data. OT systems availability is the priority. So for example if we design security based around certificates, and that certificate expires what should a system do? In the IT world they may deny access, in the OT world that would cause huge issues. There are other issues such as the fact OT systems are designed to have a long working life (25+ years), run with minimum interaction and are generally bespoke and finely tuned, so any changes have to be carefully planned. The last one means patching systems is not as easy as just uploading new software, there has to be considerable testing, planned downtime. Add in that your system maybe be older than some engineers and that just adds to the challenge.
Some suggestions such as authentication don't fit well with OT systems. M2M is a challenge because it requires non-human authentication. Also adding security without deep thought has its own issues. You could just be adding a layer that an attacker can do a denial of service attack. Unlike IT systems, the attacker is often happy just to bring your system down.
This brings another point. I said availability is a priority in OT systems, but in fact Safety is the overriding concern. There is an overlap and ballance between safety and security. Getting both right is the challenge.
The other thing to remember is that OT systems are often very performance sensitive. When you are controlling billion $ of equipment with very fast control cycles, adding jitter on the line can cause mayhem. So running industrial protocols over say VPN is just not possible. Many of these protocols such as Ethercat are basically running at Ethernet wired speed. Adding an encryption layer would cause chaos.
As someone suggested air gapping a system makes sense, but of course in the world of remote diagnostics customers want to be able to see how their system is doing. There are other architectures that are almost as secure. A DMZ zone while not totally secure can be secure if the connection to the outside world is well controlled. One criticism of the report is by examining each protocol in isolation, it misses the mitigation techniques in the system context.
Some have suggested that security has not improved in OT systems in the last 10 years. That is just not true. Standards such as IEC 62443 and the industry acceptance of them has really help improve OT security, but people are not going to rip out legacy systems to add security, and shoehorning security into existing systems is very hard, so the change is not going to be met overnight, but most systems designed today are expected to be security aware.
Quote (Reid Wightman): "...they may load a malicious payload ... into the controller..... Only, now we have no way of telling whether they did it....."
Really? No possibility of creating a fingerprint of the software......and checking regularly that the latest software image HAS THE SAME FINGERPRINT??
.....or is this idea just too advanced?
Automotive ECUs do have signed software update mechanisms and further security mechanisms.
Nevertheless, most CAN traffic is not authenticated these days (except for some special signals). An attacker could splice a hostile control unit into a CAN bus in order to mess with the signals.
Is it a problem ? Probably not, as the attacker could in this case simply mount a remote controlled handgrenade into the car and have the same bang for lesser bucks.
The car makers use crypto in order to protect their investment and make modifications harder.
Of course, without code signing, you could have the movie-plot idea of stealth, hostile code in the brake ECU doing seriously nasty stuff. Without the police putting the ECU under the microscope later. Realistic ?
Finally, authenticating all CAN messages would easily require 3x the bandwidth (can message is 64 bit, but authentication would require something like additional 64+128 bit) . Higher BW would probably require much more expensive wiring/cabling, which is why it is only done for special messages while 99% of CAN messages are not protected.
Note that even with code signing, a complete swap-out of the brake/ESP control unit is probably possible in most cars today. The "hostile" brake control unit could have a radio receiver function for "disable brake".
In this movie plot, a car could "fly out of the curve" after receiving the signal from the baddies.
Again, who needs to kill somebody covertly ? And will the police not find the modified ECU ?
In the automotive world, CANbus is by default plaintext and not authenticated. If an attacker has access to your car's wiring, he can do a lot of damage.
BUT - the attacker could likewise cut your brake piping, damage the brakeplates, losen the screws of your wheels etc. According to some horrible DDR stories, the Stasi did exactly this in order to punish dissidents.
The "fix" to such issues is to
A) have proper physical security around your car and in/around your industrial plant. Everything from intel/security service, police down to plant security personnel. Dangerous people are kept away from your car by capable security experts(the real 007s) of your government.
B) Hide such bus systems from the outside world by means of PROPER firewalls. If remote access is needed, use SSH, stunnel, TLS, secure IP tunneling, SE Linux, seL4 etc. This will cost some money to set up.
All this handwringing revolves around the idea that everything must be secure without any additional cost, except for "patching".
Finally, Iran has a serious internal intelligence+security failure, which allowed enemy spies to physically access their PLC networks. I fail to see how more ciphering of PLCs would have saved them from this threat. Iran state security is weak, that is the core problem.
Of course one cannot assume that a refinery(or similar sized plants) with thousands of employees is totally free of bad apples. Compartmentalize the plant with physical access locks, have plenty of cameras and most importantly, have a plant-internal intel+security service which will find out funny stuff.
Run your employees through government intel databases to weed out the obvious criminals. Liaise with government on threats against your plant.
Never assume an "intranet computer" is always friendly.
All of these security measures require seasoned IT and security experts, it requires documentation and maintenance of the various measures. It requires managers who know what they are doing. And it requires a budget, something the beancounters obviously hate.
Physical security is an underated, but critical part of OT security. once someone has unhindered physical access to your systems you will be compromised, it is just a matter of time and effort.
It is also the area not under vendor control, so you have limited ability to do anything about it.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
