UK's North Midlands hospitals IT outage, day 2: All surgery and appointments cancelled An ongoing network issue with provider Cisco has caused all outpatient appointments and elective surgery to be cancelled at two North Midlands hospitals in the past 24 plus hours, The Register can confirm. Both County Hospital and Royal Stoke University Hospital have had to shelve procedures since systems first went down on …
Probably in the same place as the budget for it - limbo.
These are hospitals, not luxury resorts. When even Google doesn't have a backup fiber link to its bit barn, you can hardly blame a lowly hospital for not having redundancy and multiple servers.
"Amazing to think that in war zones, bombed out hospitals keep operating."
Performing emergency surgeries in a bombed out hospital with the windows/walls half missing, and not worrying about getting paid for it, is not really a fair comparison to an NHS hospital in the UK.
For one thing, all the staff in that warzone hospital care about their patients, that's why they're still there.
I work in a hospital and we've got multiple redundant servers, network paths, power supplies etc. We test fail-overs regularly and we do test power outages. We've got paper backup systems and emergency PCs which can give access to "hot" information in case our main servers die.
After all, people's lives depend on it!
PS. I'm not saying we're perfect but we're at least a match for the banks and financial institutions I've previously worked for. The only thing we don't do is disaster recovery because if the hospital building is destroyed........
When I started development, the first rule of implementing an IT system was to also define a manual backup contingency procedure to put into effect when (not if) the IT system failed.
This process was then to be tested regularly and refined by putting any findings from the test into the procedure.
Heck, at my last job (food production), we had a 15 minute support window for critical outages, otherwise the customer was facing 6 figure losses. Surely health services have a higher priority than slaughtering animals?
couldnt agree more, until you factor in its NHS and crippled by <insert political party in number10> decades of under investment, sky high trust board fees to pay and healthcare, poor it probs bottom of the shitpile.
then all common sense and "best practises" fly out the window, much to the pain of their poor IT staff.
id rather be on the dole than try to provide IT in the NHS, a tiny empty pot to p into and no funds.
When I started selling computers (CPM and MSDOS), I would identify the manual system and then get that streamlined (under feasibility funding). It was often the case that once the manual system was sorted, it would be another few years before they decided that a computer was actually needed to help them, and that computer system initially copied the manual system, to aid implementation and fall-back.
Now it is a case of "We need a computer system! ... what was the problem?"
This post has been deleted by its author
Worth noting that when they mention "County Hospital", what they mean is "Stafford Hospital".
Yes, that Stafford Hospital.
The one with all the deaths circa 2010-2015.
They're pretty much permanently insolvent and the maternity unit is currently on the chopping block.
Budget? What budget? They can't afford nurses, much less computing - two monkeys with wrenches in the background would be positively decadent!
"The one with all the deaths circa 2010-2015"
That, and a big breakout of Legionnaire's Disease back in 1985 when it was still Stafford District General.
Back in the mid-80's they also had the an IT system that automatically assigned appointment slots and just kept going further and further forward as the waiting lists got longer and longer.
Eventually they were sending out appointment confirmation letters for appointments that were more than a year in advance and patients were turning up to hospital a year early thinking that it was just a typo on their letter!
I'd just like to say that, if Cisco could identify this specifically as a Cisco fault, they'd have put manpower and resources behind it immediately.
Yes,.. I'm an ex-Cisco contractor,... and I still "like" and "Respect" the company, and around 99% of the people,...
When faced with a mission critical problem such as this, especially in the Health Sector, they know that anything less than 100% commitment will generate
bad news in the media,...
So whatever the problem is,... they'll be paying serious attention.
"I'd just like to say that, if Cisco could identify this specifically as a Cisco fault, they'd have put manpower and resources behind it immediately."
Unfortunately I can't give any concrete or non-concrete details, or non-details, that may or may not exist, but I could name a well known very large high street retail chain who may or may not be in banking and who may or may not apparently or allegedly be able to counter this statement of efficiency about a large supplier that may or may not be Cisco or someone else.
I mean, how hard can it be to find a continually recurring kit crap-out and network fault and thus actually keep something simple like a retail branch reliably connected when your kit keeps falling over, it takes days to organise a simple off-and-on-again (but no permanent fix) and you're the world's biggest network equipment provider? Unbelieveably hard it seems.
A/C, obviously, for not alleging or aluding to any not-reasons about no-one at all, Guv.
The University Hospitals of North Midlands NHS Trust has displayed remarkable insouciance with regard to preparation for IT disasters it ought anticipate. Seemingly, the Trust is wholly reliant on an external supplier to keep its Internet and Intranet services running and instantly be able to fill the breach when problems arise.
Harking back to the days of paper records offers clues to how modern IT dependent hospitals can create useful backstops. Although paper records lacked the speed of access and flexibility of current systems they were robust. Fire/water were the worst hazards for records. It was pretty much inconceivable, except in times of warfare, that multiple hospitals would suffer this kind of damage on the same day.
Bear in mind that medical records, and supporting information, fall into two categories. First, those in current active use. Lists for planned admissions, cold surgery, and outpatient attendance would be known in advance, several days and more so. Second, records for people presumed still to be in contact with the hospital but not needed immediately.
From these considerations arise a number of modern technology and 'legacy' based methods for weathering an IT 'outage'. They are not mutually exclusive.
1. All records of people in touch with the hospital during, say, the past calendar year to be kept in an on-site Intranet backup server. Duplicates, along with not-current records would be held by off-site Internet connected services. This offers reasonable limited protection against off-site server/connection failures. Having a second on-site backup as well might be money very well spent if clinical consequences of 'outages' together with adverse publicity and justified criticism of management are to be avoided.
2. On regular basis (perhaps daily) temporarily transmit records, those anticipated to be required soon after, onto devices capable of standing alone should the Intranet fail. Records would be distributed to devices on wards/units associated with a planned admission or outpatient appointment. With this protection in place it makes sense to devolve wards, units, and departments into clusters of local Intranets able to function in absence of the primary network. Thereby, records can be accessed on the usual terminals by staff. As soon as a file keeping emergency ceases, updated records can be transferred back to their central on-site repository and onwards to off-site storage.
3. Automatically create abbreviated paper and/or microfiche records for use the next day (or so). Full records can be very thick paper files, sometimes multiple. Yet, only more recently documented referral letters, hospital contacts, and test result reports need be known for planned admissions and outpatient appointments. Clinicians would be asked to collaborate in setting limits on record content culling. The rest is mainly of interest for comprehensive periodic patient reviews and for medical-legal purposes, neither of which likely demanding urgent access.
The NHS needs to embrace a 'fail safe' ethos. Moreover, total reliance on external agencies asks for trouble.
All very true. But I understand when Brexit comes, organised by the people who worked very hard serially voting just before the end of March, it will unleash a potent particularly 'British' natural genius and self-reliance that will sort out all such niggling little problems and allow us to make the country stupendous.
And less regulated.
Shit, that's where I went wrong, I thought I was voting for the country to not be stupid, turns out I accidentally voted for the country to be not stupendous. Man, now I have to apologise to all of those leave voters...
I agree.
The problem is that to do that costs money. Money that the beancounters will likely see as wasted. After all, you need hardware and software for the infrastructure, which incurs maintenance, electrical power, purchase and licencing costs, and you may need staff to operate it, which, again, would be an extra cost.
For something that in an ideal world would rarely, if ever, be used.
Personally, I think they *should* have that kind of resilience built in, because in an emergency, people could die without the info held in those files, but they probably won't, due to money. After all, they've outsourced this stuff to save money (it's debatable whether it does, but that's another story), so may not be willing, or able, to bring this stuff back in house.
Not only is there the cost involved in doing so, but the suppliers may not be able to offer that function, and may be unwilling to give the NHS the necessary access to do so.
We are hitting that particular thing. We handle a lot of equipment loans where I work, and previously we operated a custom designed system that enabled us to track who had borrowed what. It also enabled some equipment to be pre-booked, and when the equipment was pre-booked, would enforce a requirement for a risk assessment to be approved before the booking could go ahead . It also maintained the company's inventory of equipment that was not available for loaning or booking.
Sadly, the only part of that system still functional is the inventory management as two of the development team left (and where not replaced), and the remaining members (of whom I am one) didn't really have time to spend on the development in our day to day jobs.
The down side is that we have bought in systems for managing the bookings, and loaning. Two, seperate, commercial systems, and neither will share data either with each other, or with our inventory system. And each does require their own inventory.
I imagine many other hospitals have had similar problems, and not all been properly prepared as per your excellent advice, but they find a way to manage, muddle through for the duration and we don't hear about it. What gets me about this case is the defeatist, give up and all go home approach.
Many of the patients coming in know what they are there for, who they are seeing and what for, many appointments are exactly the same checks etc as last time. And, for example, try to contact the patients and ask them to bring their appointment letters. Even these days there are lots of paper records in the system . Make an effort, it might be slower than usual and maybe some things will have to be cancelled, but getting through even 50% of the workload is a huge advance on nil.
Money that the beancounters will likely see as wasted
The real problem is that not enough beancounters (nor MPs for that matter) are inconvenienced by these events. If we had a very ill Chief Accountant who's op was continually rescheduled cancelled I suspect we may see more funds becoming available.
Same with some MPs - if we had a wheelchair-bound Minister of Transport then the Government would be throwing cash at developing accessible public transport.
One can dream I guess...
Ah, remember that back of the car conversation on that documentary a few years ago ? Jim Hacker to Sir Humphrey: How come there's a motorway to Oxford but not to Cambridge ? Humphrey in reply: It's been a long time since we had a transport minister who went to Cambridge !
And if Labour hadn't burdened the NHS with such costly PFI contracts the hospitals may be able to afford some of these measures. As it is many hospitals are running a huge deficit.
It always helps to be fully aware of the facts before prophesising
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
