back to article Labyrinth of 371 legacy systems hindered hospital's IT meltdown recovery

Last summer's datacenter outage at one of the UK's largest hospitals took two months to completely rectify because of the complexity associated with 371 legacy IT systems, a new report has found. Guy's and St Thomas' NHS Foundation Trust suffered an IT outage at the peak of last summer's heatwave, when temperatures hit 40°C ( …

  1. Anonymous Coward
    Anonymous Coward

    Incompetent IT leaders in GSTT

    The leaders of IT in Guys have had years to sort these problems and have stubbornly refused to modernise, repeating the same mistakes time and time again. They aren't short of funds as they've knowingly overpaid on contracts multiple times (tens of millions and more).

    They have brushed these problems under the carpet for many years and the entire set of IT leaders are not fit for purpose. They make everything overly complicated, refuse to change and that's how they ended up with ATOS as a provider. They were the only ones willing to sign up for this mess.

    1. RPF

      Re: Incompetent IT leaders in GSTT

      Bet they're far from alone, especially in the NHS world.

      1. Anonymous Coward
        Anonymous Coward

        Re: Incompetent IT leaders in GSTT

        I don't know GSTT personally but I do know NHS IT. I used to find it amusing back in the day when I was a junior dev. when nothing could ever be pinned on one person because it was an "institutional failure" because to me it seemed so obvious.

        Now I'm older, not necessarily wiser and a bit further up the greasy pole and I can see for myself the cats cradle of systems, dependencies, ownerships, empires, silos, devolved decison making and "business" interests that are trying to keep us all alive and all I can do is say "glad I'm not sick" (see Bob Hope, Spies Like Us).

        1. ecofeco Silver badge

          Re: Incompetent IT leaders in GSTT

          Cats cradle of "holy shit" is right.

          Same everywhere, really. "Is there a hard way to do that? Then lets do that!"

    2. ecofeco Silver badge

      Re: Incompetent IT leaders in GSTT

      IT leaders? You do know they do not control the money nor work process right?

      That said, yes, I've seen my fair share of IT manglement as well.

    3. Julz

      Re: Incompetent IT leaders in GSTT

      Fixed it for you:

      NHS have brushed problems under the carpet for many years and the entire set of leaders are not fit for purpose. They make everything overly complicated, refuse to change and that's how they end up with multiple agency providers. They willingly sign up for this mess for the money.

    4. herberts ghost

      Re: Incompetent IT leaders in GSTT

      In most cases the cause of this type of outage is caused managerial incompetance. They chose short term savings at the expense of customer service. Failure to invest in system upgrade including power and air conditioning backup. What kind of moron puts the backup in the same city/power-grid as the primary?

      These folks will often blame the legacy systems that they inherited. But mainframes (for instace) are 1000 times faster than they were in the 70s.

      Children will say rewrite everything. The track record of conversions is terrible. The SW works but the specs are lost.

      You were cheap/incompetant time to find another job.

      1. richardcox13

        Re: Incompetent IT leaders in GSTT

        > In most cases the cause of this type of outage is caused managerial incompetance.

        Largely lead by political incompetence.

        "Cut 33% of managers" made positive headlines, but they cut too many useful ones. Since then productivity - based on positive patient outcomes - per staff member has fallen steadily. Basically there is no ability to improve the way things are done (font line staff are too busy being frontline staff to fix things).

        And then the massive drop in capital expenditure meant there is no budget for anything than "what's the cheapest" despite that being the more expensive option in the medium to long term.

  2. Anonymous Coward
    Anonymous Coward

    Reading between the lines:

    "We'd got rid of most of the people who actually knew how the systems all interacted with each other to save money".

    1. BoldMan

      Funny how this reminds me of the RBS debacle a few years back...

    2. ecofeco Silver badge

      "But all magic boxes are the same, right? Besides, what do non-executive plebes know anyway?"

  3. elsergiovolador Silver badge

    IR35

    So no updates since IR35 changes were introduced in public sector in 2017?

    Wonder why...

  4. andy 103
    FAIL

    Tip of the iceberg figures

    "The trust incurred £1.4 million ($1.7 million) in out-of-plan spending on technology services to respond to the incident."

    Notwithstanding the £??? of planned spending beforehand to ostensibly make sure this type of thing "could never" possibly happen.

    I dread to think how much in total was spent on this completely avoidable fuck up. On any view, it's likely to be a lot more than £1.4 million.

  5. Anonymous Coward
    Anonymous Coward

    Is this "legacy" used as a synonym for "existing" ?

    1. Roland6 Silver badge

      Should also ask whether this “system” is being used as a synonym for “server” that isn’t running at least Window's Server 2019…

      So the complexity is people don’t know what application systems are running on which physical system/server and the interdependencies between application system and between physical infrastructure systems.

    2. CrazyOldCatMan Silver badge

      Is this "legacy" used as a synonym for "existing" ?

      In our case, 'legacy' means old crap (much of it coded in-house) that won't run on a modern version of any OS, we can't get permission to p2v the damn thing so it continues to use a physical cluster in the server room, uses an old version of SQL server [1] and, every time we say that we are going to turn it off, we get howls of anguish from the two people who use it, claiming that it's business-essential [2]. We've managed to reduce those systems over the years (Cyber Essentials is a *great* tool for enforcing compliance) but we still have some.

      [1] And the system owners refuse to let us update it because 'it might go wrong'. Even when we offered to leave the old system in place and clone it and upgrade the clone. Apparently, they are too busy to test the new one..

      [2] We ran the stats - it gets used once or twice a month and costs us north of £25k per year in license fees for the middleware. Which *we* have to pay rather than the system owners. The data it actully provides doesn't appear to be used in any of the business processes that the system owners use..

      1. VeNT

        Bonus points of it's an 16 bit application running on virtual 32 bit hardware

    3. Anonymous Coward
      Anonymous Coward

      In my day job I correct people from using the team legacy systems to the incumbent system many many times. I'll allow legacy if there's a new system implemented and transition in progress but unless that's happening, it's not legacy yet.

      People who don't want to fund maintenance on something love calling it legacy and are then surprised when it breaks and 'the legacy system' was still so important!

      I feel sorry for the person in the 50deg cold aisle in the datacenter wondering whats 'least important' in this mess that could be shut off to keep the most critical stuff going. Clearly they never reached a decision on that!

  6. MarcoV

    solution

    Rebrand the interwoven legacy systems as micro services?

    1. ecofeco Silver badge

      Re: solution

      I would like to subscribe to your newsletter.

  7. Anonymous Coward
    Anonymous Coward

    IT is a cost to be minimised

    The NHS, like most government departments and business everywhere thinks of "IT" as a cost to be minimised when ever possible. It is not understood that the very spreadsheet that is being used to reduce the cost of "IT" is part of "IT" as is the computer the are using, the network, and the email they just sent a complain on about the cost of "IT".

    I keep using "IT" as it covers so many things that are actually distinct things, a bit like healthcare covers so many things, but you probably dont want the local Dr's reception doing any sort of surgery

    1. Ali Dodd

      Re: IT is a cost to be minimised

      And the idea that making cuts in IT will actually result in any sort of cost savings in the organisation. 9 times out of 10 you cut IT you will increase overall costs to the business (more likley 99% of the time really). Increasing funding to IT usually has the result of reducing costs elsewhere, not to say there are not efficiancies to be had and changes can be benficial but you save a few quid in IT to your peril as it is the blood and lungs of your organisation and aids every other department to do their job. Very rarely will it be treated anything like that however.

      1. andy 103
        Pint

        Re: IT is a cost to be minimised

        the idea that making cuts in IT will actually result in any sort of cost savings

        It makes me laugh whenever organisations - or even the Government themselves - come out with this crap. Because it's usually one breath after they've been going on about how we need to invest in technology and have a workforce of people who are savvy in IT for exactly the reason of: improved efficiency!

        I think what does result in cost saving is taking people who have no understanding of IT out of the equation. They seem to get paid handsomely for achieving very little. Much like I suspect a good number of people did at this hospital to produce a "plan" which meant what ended up happening could never possibly happen...

        Unfortunately if you don't have the knowledge and experience to understand how IT actually benefits people in a tangible way the result is an expensive clusterfuck. Which happens time after time after time in organisations which are otherwise well funded and full of so called experts.

      2. Potemkine! Silver badge

        Re: IT is a cost to be minimised

        Hear, hear!

        The problem is that too many times IT is still under the responsibility of the CFO, as if we were still in the 70s and IT used mainly by accounting. The first step would be for executive committees to place the CIO as the same level than the CFO. And too many times CFO can only think about reducing costs without understanding A) the gain in other departments B) the gain in term of having the final customers satisfied.

        1. ecofeco Silver badge

          Re: IT is a cost to be minimised

          The problem is that too many times IT is still under the responsibility of the CFO, as if we were still in the 70s and IT used mainly by accounting.

          I'm living that right now.

    2. Anonymous Coward
      Anonymous Coward

      Re: IT is a cost to be minimised

      Fairly recently I worked at a place where the PM's bonus was based upon delivering the project quickly and cheaply. Someone's idea of a way to save money. Except that it creates the perverse incentives of

      a) Not doing anything strategic that might help other projects

      2) Not finishing the project properly and leaving the users with a load of (often Excel) workarounds

      iii) Creating more work for the poor sods you were supposed to be helping. You know - the ones who pay your wages

      I get it that execs are scared of a £5m IT program, but then you can end up with 30 £500K projects

      1. yetanotheraoc Silver badge

        Re: IT is a cost to be minimised

        "Excel workarounds"

        Is it that way everywhere, or did you recently work where I currently work?

        1. Handlebars

          Re: IT is a cost to be minimised

          Excel? I once found an admin team in an NHS hospital using table grids in Word because they had no knowledge of Excel. Their manager told me I needed to fall in line with their 'process'. Yeah, no.

        2. Mike Pellatt

          Re: IT is a cost to be minimised

          Everywhere.

          Especially in the places where Excel is banned.

    3. Boris the Cockroach Silver badge
      Unhappy

      Re: IT is a cost to be minimised

      Its not just the NHS that thinks this, its everywhere

      But then a lot of people dont think "How much of our business depends on the computers working correctly?"

      which is how I can get chewn out for asking to book a machine tech to come in and swap out a failing HDD at a planned time, and get the answer "It has not failed yet, so we're not paying for 2 hrs of the tech's time swapping out a working HDD"............... "So we wait for an unplanned outage right when we need to use the machine urgently... then have to wait 4-6 days before the tech can come in?"

      And people wonder why theres a bit of plywood in my office that says "Stress reliving tool... please punch freely"

      (notes for the questing: the machines have a lot of vibration when running.... the HDDs are mounted on carriers designed to isolate them.... it doesn't always work.... and yes we're testing SSDs at the moment....)

  8. ColinPa

    It's not a cost - it is a money saver

    I remember visiting a foreign bank a few months after a major outage. After the event the CTO brought in an enlightened manager (and some of his old colleagues).

    Instead of being a cost - he sold the IT systems as saving money. "If you are not part of the solution - you are part of the problem" and several senior managers decided it was a good time to retire.

    They got rid of paper and used pdfs They managed to shrink the pipeline so things could be approved in hours not days (or weeks). Managers now had time to manager, not shuffle paper ( nor signing things).

    Due to take overs they had several bespoke systems doing almost the same thing. They picked the best one (eg for loans) , and migrated all the bespoke loan systems to one. There was a 6 month bulge in spending, but then they got rid of old systems, and saved money in licensing costs. Some systems they virtualised, and so got rid of old hardware and made better usage of the existing resources.

    He charged back costs to the departments, so now each department had to pay it's cost rather than have IT centrally funded. The heads of departments had to go to the CTO and discuss the cost of running their department. After this meeting the heads of departments were more than happy to let go of bespoke, expensive systems, and use the "off the shelf" system provided by the enlightened manager. When they saw progress, the heads of departments were even more keen. I think he was promised 1% of the savings, and retired a few years later a rich man. Getting rid of idle old, hot machines saved them a lot of money in electrical power and aircon.

  9. Sparkus

    ATOS again?

    hmmmmmmmm

  10. Rich 2 Silver badge

    104°F

    Are you obliged to add completely bollocks units into your articles now that the Reg has gone all American?

    1. FirstTangoInParis Bronze badge

      Re: 104°F

      Surely all metrics should be referenced to El Reg standards? In this case 2 Hilton. All budget figures should be in Pogbas. There’s a cross over there somewhere. Maybe sponsor the patient or hospital stats on highly visible screens by some IT provider, who then will take the fall for such disasters in the article?

  11. Anonymous Coward
    Anonymous Coward

    NHS IT

    Sorry IT guys but paper notes and hard copies of scans were given up far too easily. IT failures were and are a regular occurrence in the biggest NHS Trust in the UK. The Trust's response was a helpline with a wait similar to the Inland Revenue's. WannaCry affected this Trust particularly badly back in 2017. Systems were down/suboptimal for many weeks. "You don't need a chest x-ray to diagnose pneumonia" was the local Medical Director's glib response.

  12. midgepad

    The mess in the bucket.

    From one class of users' perspective, the records centre around one patient, the one in front of us now.

    Plain text and picture files work nicely, and can all go in one folder.

    Automation is another matter, with assistance to writing prescriptions for instance, but even there, sending messages, and dropping a line of text into a file, may not be inferior.

    Data for analysis might need to be centralised, but that can be done separately, or programs can pick out one record at a time and ask it questions.

    Someone wants to add a new data type to the bucket, fine, provide the open sourced viewer for it.

  13. Anonymous Coward
    Anonymous Coward

    Is 50c really all it took to cause this much damage?

    Most kit should be able to survive that surely?

    1. Roland6 Silver badge

      Most kit is only spec'd for :

      Operating: 0°C ~ 45°C

      Storage: -25°C ~ 70°C

      From work I did when UK ambient temperatures went over 40C, I suspect temperatures of components within cases within cabinets was significantly higher.

      I suspect with a datacentre ambient temperature of 50C, given the tolerances within spinning disk drives, thermal expansion would be significant.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like