Legion of demons found in ancient auto medical supply dispensing cabinets Consider this a reminder that end-of-life software doesn't get patches: researchers have turned up more than 1,400 vulnerabilities in a widespread automatic medical supply dispensing system from CareFusion, because old units are still running Windows XP. The computer-controlled dispensing cabinets are installed in hospitals …
I haven't disassembled the particular kit in question, but going from analogous systems, the most practical software upgrade would probably involve dumping the old PC and installing a new one with up to date software. It's much less time consuming than having a serviceman on site struggling with ancient and wheezy PC hardware.
However, while there may be a PC involved, there may also be things like label printers and bar code readers. If those kit vendors didn't produce upgraded drivers for Windows 10 (after all, why would you be installing Windows 7 this late in its life cycle?), then you have to replace those as well, and they can be a lot more expensive than a PC. The new model printers and bar code readers may not fit in the space originally meant for the old ones, etc.
Automated pill dispensers have to be networked to the central management system, or they don't really work because they can't update the current patient medication orders, order new drugs, etc. And going back to manual pill dispensing isn't a realistic option, because of modern standards designed to reduce dispensing errors (which when done manually are a measurable cause of death for patients). So all the commenters who are going to declare that the solution is to "air gap" them aren't thinking the problem through. If you're going to do that, you may as well be claiming that the solution to security for email, file, and web servers is to air gap those as well.
Hospitals and nursing homes are automating, just as factories have had to adapt to automation. They need to plan equipment life cycles, including mid-life upgrades.
Factory automation vendors for example are pretty good at this when it comes to hardware. Their software however tends to be nightmare for pretty much the same reasons as we've seen here. They tend to have lots of dependencies on multiple third party proprietary software vendors who have wildly varying product life cycles that don't match up with those of their customers.
A lot of the problem in both cases is the mindset involved in the original equipment design of buying COTS hardware and proprietary software components that are here today and gone tomorrow. It's possible to design systems that are intended to have long lives and go through mid-life upgrades with little hassle, but so much software and hardware engineering is outsourced these days to parties who have no long term stake in it that it's amazing that it works as long as it does.
@thames
There's a difference between air-gapping individual units and segmenting your network so that someone who might open an email containing an alleged invoice or someone servicing the HVAC installation can't affect each other let alone critical systems such as these.
It requires thought and planning. It probably requires more expenditure on H/W as functionality which would otherwise be provided centrally need to be provided separately on each segment. But if you take security seriously it has to be done.
And yes, I have worked in a situation where admin offices and production had separate networks and even different parts of production were subject to extra security. Security was taken very seriously, it was the first word in the business's name and they meant it.
"
I guess it depends on the device, but my Symbol barcode scanner is USB, and you can choose whether it appears as a HID Keyboard or a Serial Port, both of which have pretty standard support.
"
That's nowadays, yes.
At the time the systems in question were installed, USB was uncommon and the devices probably interface via the parallel port or perhaps even SCSI or some other protocol that you will no longer see supported.
"At the time the systems in question were installed, USB was uncommon and the devices probably interface via the parallel port or perhaps even SCSI or some other protocol that you will no longer see supported."
I've never installed this system, but most scanners I've installed during rollouts for other dispensing systems have used either serial port or USB. Key thing though is getting the drivers to work can be an absolute PITA. The dispensing software I've seen is VERY sensitive to the driver version used on the scanners
There are many legacy systems that cannot be upgraded due to the cost of the hardware they drive, or are driven by. Many of these are in the medical and other essential services sectors.
If Bill Gates is such a philanthropist he should step in and either ensure free software support for the lifetime of the hardware or fork out the money to upgrade the hardware such that it works with later versions of Windows. I am aware that he's no longer in control at MS, but from a PR point of view taking such an attitude would be seen positively.
Without that kind of safety net hardware designers are surely going to say that it is not worth the aggro to be compatible with Windows for the lifetime of the hardware, and may therefore opt for a more Open Source platform to run with. (Is this something that is happening in practice? Have big sales been lost because the medics have put their foot down and say software they are buying has to run under a non-proprietary operating system?)
The big problem is that most people, even those that should know better, just see Win XP and react rather than ask what version.
While I know nothing about the systems in question, I would assume they would be using the XP embedded version customised to do the particular job. If that is the case what is all the fuss about, other than having the units on a dedicated air gapped network that some idiot insisted be connected to the internet.
These systems are running XP embedded. I know, I was trained on them, ooh, about 6 years ago. AFAIK these models are still being sold.
The bad news is that CareFusion (actually part of the BD group) have an almost comical suicidally backwards approach to technology, in particular computing. At least they had then and I haven't heard of any major strides forwards on this front. And they have a lot of them in place and frankly if they work, then a hospital will shy of replacing them and if CareFusion repair them, they take care to put them back pretty much just as they were.
Option 1, upgrade all your kit to Windows7.
Write new device drivers and do all the automated and user testing to show that there are no bugs / inconsistencies / undocumented behaviour.
Before you get halfway through this, start again with Windows 8.
Repeat for 8,1 then Windows 10.
Remember to evaluate every month's set of patches before approving them for all your customers.
Option 2, stick with WindowsXP that has been running for 10years and firewall it from the nasty evil network.
Upgrading often is not an option and the device drivers (often very specific to hardware and OS in these custom systems) need a rewrite - which the manufacturers are not going to do because they would much rather sell you a new system. The manufacturers like to keep shipping the old design as long as possible because the design costs are paid for and, when it finally becomes obvious that it's time to can it - EVERYONE has to buy a new system $$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$
"Embedded software that is long lived really shouldn't be running on software that has a finite life. Better to use something you have control, and (hopefully) have source of the operating system located nearby."
Nope. Hospitals are a regulated environment, which means anything that runs there MUST (by law) be approved by the government (otherwise, any suits as a result of faults default against them). The more custom the software, the longer and more costly the vetting process. And given the diverse pressures hospitals suffer from costs, staffing, building and equipment (hardware) maintenance, etc., it's rather a miracle so many are still operating.
"Nope. Hospitals are a regulated environment, which means anything that runs there MUST... be approved by the government... The more custom the software, the longer and more costly the vetting process."
Nope? You make an excellent case for the software vendors not relying on an OS whose EOL can be dictated by a 3rd party.
Nope. Hospitals are a regulated environment, which means anything that runs there MUST (by law) be approved by the government (otherwise, any suits as a result of faults default against them).
I rather don't think so (at least not here in the States, and I have some 15 years of SW development experience in the US medical field to back this up). Hospitals here only need to have FDA approved devices. The devices themselves can get approval with COTS software in them, or home-brew software in them, or any combination of the two. And in my personal experience, the mix doesn't matter to the maker of the device WRT how long it takes to get approved. So long as you follow the FDA approval process (and can prove you followed the process), you will get approved, and the hospital is perfectly free to pick and choose your product over any others competing for that floor space.
That's the fact, Jack!
So, at the of the day, it all comes down to costs, staffing, building and equipment (hardware) maintenance, etc., like you said. It has fuck-all to do with how the device was developed.
A large part of the problem are the regulatory frameworks involved and the grief in getting any medical software approved.
Fairly recently I was involved with upgrading a piece of medical software from Windows XP to Windows 7. The changes were fairly trivial, but getting the software revalidated for FDA approval was long, drawn out, and bloody expensive.
This is why so much medical gear runs on XP. Perversely, the validation and approval process that is actually supposed to ensure the software is safe is now having the reverse effect.
Easy to say.
The crux of the problem is that hospitals are a very costly business, and it is viewed as more important to have competent staff than it is to update computers. That means that the budget is seriously constrained and tools are used as long as they appear to function - EOL dates be damned.
Find me one hospital that is not understaffed with most departments not suffering from insufficient budget and you will have found a unicorn.
Unfortunately, the scum that take advantage of these institutions will provide a costly lesson to everyone - who survive it. One can only hope that some of those scum will find themselves in a hospital when another scum locks everything down for money, and that the scum on the hospital bed will live long enough to personally experience how unfair that is for all innocent people.
IT in hospitals is in for a rough and bloody ride, no doubt there.
The reason is that the product pre-release life cycle is incredibly long.
Medical products go through a very lengthy period of pre-release certification, and so it can easily be five years after development began before it even ships.
So even if you start at the bleeding edge, it's way behind by the time it first ships.
Which then butts up against another hard reality. That pre-release lifecycle is longer than the product lifecycle. IOW, the software is obsolete and broken beyond repair by the time it even gets deployed. It's a constant wild goose chase. You also see it in other industries where computer-controlled equipment is expected to run for decades yet the software within cannot be relied upon to last that long, mainly because technology marches on and eventually a game breaker emerges.
Well, why should drug dispensing machines be any saver than money dispensing machines...
One way to look at the problem is to conclude that the free market is not able to provide sustainable long-term solutions.
So what will it be - a medicinal-industrial complex akin to the military-industrial complex? A handful of companies that control the market - but are increasingly unable to deliver anything useful (Exhibit A: The F-35 desaster)?
Or a cluster of manufacturers attached to universities to combine R&D and actually building stuff, all run by the state in order to make sure they are still operating in a couple of decades?
@Olaf
As Thames pointed out the systems are networked to a central control so it's not a matter of air-gapping individual machines.
But the real problem is that it's been all too easy to install a single network, hang everything off it and depend on perimeter security. When you have PoS terminals compromised by laptops plugged into the network by HVAC technicians it should be crystal clear that that doesn't work. For years we've been told to program defensively; network designers should have been told to work on similar principles and connect clusters of machines that need to work together but isolate them from everything else.
<quote>Having 3-4 air-gaped physical networks in your premises is.</quote>
NOT if you do it from the very beginning.
My employer before we moved into our current building had such a set-up (EVERYTHING on a single LAN segment) which made the networking guys nervous.
That was the one piece of advice they gave to my PHB - separate the varied functions into their own completely physically separate networks. That advice was heeded, and we do not have the potential for a HVAC tech from our service company try to access anything else BUT those machines connected to the environment systems. The network guys sleep better at night.
You seems to imply that you know the future needs of your business as soon as you move.
Furthermore, a real PHB would never listen to the advise of the lowly network guys.
Our network guys are plagued with nightmares since we moved 3 years ago...
Don't these systems have a limited communications requirement? It seems a shame to throw them and their peripherals away when all that's needed is to limit the incoming link to genuine drug-related messages.
Is there any reason why a new interface for the XPs couldn't be provided so that they were no longer connected to the internet. For example, an RPi with a custom version of OpenWRT plus a message re-writer could sit between them and the internet and transcribe incoming data. Then the internet connection would be secure and incoming data would be screened so that only genuine prescription related messages were passed on and then only after being transcribed to a limited format which could not transmit malware.
"Is there any reason why a new interface for the XPs couldn't be provided so that they were no longer connected to the internet"
I believe these machines do full stock control for the drugs used, so have to communicate stock levels / use / anticipated demand to the parent dedicated server, which in turn need to communicate externally with various drug supplies. In the UK you'd route this through the NHS N3 network, but its still going to be exposed to the internet at some point
"
I believe these machines do full stock control for the drugs used, so have to communicate stock levels / use / anticipated demand to the parent dedicated server, which in turn need to communicate externally with various drug supplies. In the UK you'd route this through the NHS N3 network, but its still going to be exposed to the internet at some point
"
Which could still be done via an intermediate machine that if necessary runs a bespoke application that simply parses and sanitizes the incoming data before either blocking it or passing it on to the XP's, and does the same for data from the XP's to the Internet.
" a bespoke application"
there lies your problem: that won't happen. No-one would authorise or pay for an intermediate kludge program that sits between these systems and the drug suppliers own proprietary software. Too many things to go wrong. Besides which, it would destroy the logic of having these machines, where all the stock control is done at "point of sale". Putting another step into the system destroys that ability
there lies your problem: that won't happen. No-one would authorise or pay for an intermediate kludge program that sits between these systems and the drug suppliers own proprietary software. Too many things to go wrong. Besides which, it would destroy the logic of having these machines, where all the stock control is done at "point of sale". Putting another step into the system destroys that ability
Ah, erm. There is an "intermediate kludge" system that sits betweeen these (effectively POS, possibly Point Of Dispense) systems and the hospital patient records system. The theory is that there are two separate network segments, with the system running the "intermediate kludge" software acting as the gateway, effectively the router.
Sensible enough, until you combine this with a couple of issues - this system is rarely, if ever, patched and until a couple of years ago ran one of the least performing AV systems. The windows administrator and other maintenance user passwords for this system are, of course, hard coded.
The RPi connection to the internet could be a) secure b) tightly locked down c) self monitoring and d) report any apparent malfeasance to the NHS parent. With slight modification to its software the XP could use, for example, a limited vocabulary and a serial link for communication with the RPi, making it well-nigh impossible to hack.
The folks at Pi HQ can do custom limited editions. I'd have thought that a Pi on a PCi card ready to pop inside an XP box would find the 3,000 to 5,000 users in this and similar situations which would make a custom run viable. If I were a few years younger I'd be crowdfunding something on these lines tomorrow.
Wouldn't the hackers then just hack and pwn the Pi and then use that to hack and pwn the dispenser through the serial connection? It's just a difference in degree but not in kind: easy enough for a hacker to work through. Remember, some of the cleverest hacks involve unusual but still plausible (and thus valid) standard operating parameters that nonetheless cause havoc when taken as a gestalt.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
