back to article Sysadmins fail to fix NHS IT snafu, HUNDREDS of appointments cancelled

IT staff at the UK's largest health board have failed to fix a major system error that's cutting health workers off from clinical and administrative services. NHS Greater Glasgow and Clyde (GGC) had to cancel 288 outpatient appointments yesterday along with four planned inpatient procedures, 23 day cases and 40 chemotherapy …

COMMENTS

This topic is closed for new posts.
  1. IT Hack

    Patients

    So what are they telling their patients? Patience?

    1. Anonymous Coward
      Anonymous Coward

      "We apologise for any inconvenience"

      ...or premature death

    2. Brewster's Angle Grinder Silver badge
      IT Angle

      Re: Patients

      How will a Gilbert and Sullivan operetta help? Or is that what they're playing while on hold? If the latter, it sounds like they really need the whole Ring Cycle?

  2. hplasm
    Meh

    "network problems"

    Ah- that old chestnut.

    1. Phil O'Sophical Silver badge

      Re: "network problems"

      Probably a connectivity issue, as in "it's a public holiday in India today, and we can't reach any of our outsourced IT staff"

    2. Anonymous Coward
      Mushroom

      Re: "network problems"

      Yes, and we can all thank Bill for that error message (no matter how many times it was used.)

  3. Kevin Johnston

    I told them....

    once you give up quills and parchment it will all go wrong but no they went ahead and had the 19th century regardless

  4. Halfmad

    GG&C is a muddled mess of old health boards shoved together, they have a lot of management, team leaders etc.

    I'm just curious if this is down to lack of change management with a staff member doing something silly, however as it's IT this would almost certainly be covered up with "transient network issues".

    1. Shinobi87

      speaking from experience

      everything is fully changed managed up there

  5. Anonymous Coward
    Anonymous Coward

    A "network" problem?

    Betcha it isn't ... if by network you mean transport layer downwards.

  6. JQW

    So...

    So, where are these SysAdmins based? Bangalore or Kuala Lumpur?

    1. Anonymous Coward
      Anonymous Coward

      Re: So...

      Not sure about the sysadmins but we email Atos for our tech support.

      1. Andy Gates

        Re: So...

        Atos? Atos would say the system was perfectly healthy even if it was dead.

    2. Shinobi87

      Re: So...

      also in Glasgow, I believe some of the 1st line helpdesk was outsourced at one point :)

  7. smudge
    Holmes

    Failed on 01 Oct?

    Ten quid says there's a calendar somewhere with 31 days in September.

    (Can we have an "only half-joking" icon please?)

  8. Shane8
    Joke

    Have they tried...

    [chris o'dowd'] Turning it off and on again ? [/chris o'dowd']

  9. Justin Stringfellow

    so presumably..

    the medical staff still went to work that day, and the patients still turned up, with appointment letters in hand, and were turned away because "computer says no"?

    WTF, why not send them in? What have I missed?

    1. CCCP

      Re: so presumably..

      Exactly!

      Pen + Paper anyone? Or do they use Star Trek-style EMHs all over the place?

    2. David Pollard

      Re: so presumably..

      "What have I missed?"

      Maybe a large proportion of their patient records are now computerised?

  10. Anonymous Coward
    Anonymous Coward

    Clinical apps seem to be back up as of 5 minutes ago but email and file access is still stuffed.

    1. Anonymous Coward
      Anonymous Coward

      So ... AD problems then?

      1. Juillen 1

        There really are a shed load of problems that can happen on the network.. AD just complicates it further up.

        1. This post has been deleted by its author

      2. Andy Gates

        Probably not, that would lock everyone out. My guess is the PAS (Patient Admin System - backbone of the whole dealio) interfaces are buggered in a not-easily-fixed way.

      3. This post has been deleted by its author

      4. Anonymous Coward
        Anonymous Coward

        AD

        Some Chaos Monkey deleting OUs??

      5. Anonymous Coward
        Anonymous Coward

        DNS

        Sources close to the troubled starlet claim her DNS is in rehab.

        Imagine if you will, a 20,000+ seat organisation with a DNS zone up the wazoo.

      6. hplasm
        Meh

        AD problems

        So it seems.

  11. Dr Who

    Every time there's an issue in the cloud, there are plenty of people taking the "told you so" line. This appears to be an example of the reverse. Running your own data centre is all very nice and you can certainly touch everything, but it's very expensive indeed to provide the sort of N+1 or 1+1 redundancy that is standard fare in colo or managed data centres. I'm not taking sides here, just trying to balance the argument a bit.

    1. Peter Gathercole Silver badge

      @Dr Who

      Ah, but if the problem is in-house, with your own employees, then you can at least bang some heads together and threaten to kick people out of the door.

      If the problem is either in the Cloud, or with some outsourcing outfit, all you can do is threaten to not renew your contract in 4 years, by which time everybody will have forgotten!

      Of course, if the contract was negotiated properly, you may get some financial recompense, but I don't trust any of out public organisations to be that good at getting such clauses into a contract....

  12. Fletchulence

    Outages

    These do happen. For me it's a bit rich to accuse of system of failing the patient because one the basis of one days downtime. I'd need to know what benefits it created the days it was running. Then I have an idea of whether this is a story or not.

    If, on the other hand, they can't quantify the benefits of the system, just turn the bloody thing off of course.

    1. Peter Gathercole Silver badge

      Re: Outages @Fletchulence

      Sure, outages happen. But good design should make such an outage relatively short. Keeping critical systems up even when things are failing should be part of the design and change process.

      One day's down time for a regional health authority equates to hundreds of thousands of pounds of wasted resource, even more if you take into account the time of the people who attended their out-patients appointments without being seen.

      I sometimes remember when I used to talk to a power engineer in a regional electricity distribution company, where five nines is a way of life, not an aspiration.

    2. Anonymous Coward
      Anonymous Coward

      Re: Outages

      Absolutely outages do work, but it's how fast you recover from a major outage and what the cause of that outage is that's important. Users will accept, grudgingly that there will be outages, but they won't accept one which drags on with little or no information coming from IT, which is usually the case.

      As someone who's been involved in investigating incidents like this within the NHS it's likely that half truths are already flying around and users are getting half of the story whilst IT management try to cover up a complete balls up. Let's hope that's not the case, for a change.

    3. Anonymous Coward
      Anonymous Coward

      Re: Outages

      It ties all the clinical Applications together as well as the respective data, clinicians can then logon enter a users Unique ID, view information, tests etc which have been collected from various applications in different departments, etc NHSGG&C is a damn site better integrated system wise than Englands NHS

      1. Anonymous Coward
        Anonymous Coward

        Re: Outages

        Ties them together but doesn't stop you individually logging into each system to continue doing your work and the system also shouldn't pull others down when it breaks..

  13. cuddlyjumper
    Holmes

    Ealing powers

    Lateinos and Romiith, anyone?

    The Sherlock icon... for those in the know...

  14. Anonymous Coward
    Anonymous Coward

    I think it's more than Glasgow.

    NHSmail isn't allowing attachment downloads and Choose and Book (lovingly called Booze and Chuck) isn't working for private healthcare companies.

  15. RISC OS

    This is what happens...

    ...if your sister brother's one man IT company is given the contract so that you can bank some of the costs by him giving you backhanders

    1. hplasm

      Re: This is what happens...

      If you're BillG's brother, perhaps...

  16. RISC OS

    "they had come up with a way to let chemotherapy patients be seen"

    "they had come up with a way to let chemotherapy patients be seen",

    like before there was something called IT?

    What exactly did the IT team do, pick up the patients in their own car and drive them to see the doctor?

    1. Anonymous Coward
      Anonymous Coward

      Re: "they had come up with a way to let chemotherapy patients be seen"

      I think you're missing the point, boards in Scotland are increasing going towards paperless or at least paperlight working, meaning there's nothing to refer to . for example if you're going to give someone radiation treatment it's quite important to do so in the correct location not with the patient pointing and saying "around here" and with the correct dose etc.

  17. Anonymous Coward
    Anonymous Coward

    "So...AD Problems then?"

    Without going into any further detail, I can confirm that this is where the problem lies...

  18. zooooooom

    Err hold on - surely the patients know when their appointments are - all the docs have to do is sit in their office and wait for em...

  19. Anonymous Coward
    Anonymous Coward

    ...

    .. Blue screen of actual death???

    1. Anonymous Coward
      Coffee/keyboard

      Re: ...

      That was my keyboard.... you bastard.

  20. raving angry loony

    Same old, same old

    So a poorly planned system that's had a thousand political fingers screwing up the design, and now that it's not working it's again the fault of the sysadmins. No matter how unmaintainable the system might actually be. Same old, same old in other words. Nobody ever looks at or blames the wonks who got promoted for delivering an unmaintainable system of course.

    1. Anonymous Coward
      Anonymous Coward

      Re: Same old, same old

      This system has been developed locally by the board, it's got nothing to do with politicians. Every health board is doing the same, developing something similar but as usual there's no one system that fits all due to regional demands being different, but they'll all still interconnect when up and running where required.

      Political interference in the NHS is rife but it's not the cause of this, although I did find it funny that the muppet Scottish Health Minister said that boards would review their backups etc. I think you'll find most do that very regularly anyway.. usual minister talking about something he clearly knows nothing about.

      This will boil down to one of two things:

      1. Incompetent management or staff member.

      2. Lack of oversight of change management.

  21. CABVolunteer
    WTF?

    Doesn't sound like a major problem or they've got a decent fallback system in place

    "NHS Greater Glasgow and Clyde (GGC) had to cancel 288 outpatient appointments yesterday along with four planned inpatient procedures, 23 day cases and 40 chemotherapy treatments when IT systems crashed."

    With such a huge scale of operations across Glasgow, the quoted loss of activity is tiny. I'll bet the usual daily DNAs plus cancellations due to staff/theatre non-availability is ten times greater.....

    1. Mark Wylie

      Re: Doesn't sound like a major problem or they've got a decent fallback system in place

      Bang on the mark. There are very few parts of the organisation that have a critical reliance on IT, mostly outpatient clinics where records, test results and referrals from GP's are pulled from various internal and external sytems via the Clinical Portal.

      1. Anonymous Coward
        Anonymous Coward

        Re: Doesn't sound like a major problem or they've got a decent fallback system in place

        So....is it the Citrix platform which fell over?

  22. Magnus_Pym

    Cuts?

    It is possible that the upper management were tasked with making savings and chose to get rid of the costly and troublesome AKWTAD (actually know what they were doing) staff.

    1. Anonymous Coward
      Anonymous Coward

      Re: Cuts?

      you mean all the contractors, the NHS would fall apart without them

  23. Clyde

    it's not a disaster

    Local radio had one doctor on - he saw all his patients today. Phoned up the GPs who faxed over patient notes. Colleagues - some did, some didn't.

    Son of a friend works in IT there. He says where patients showed up, they could send janitors (or whoever) to get their files. They just couldn't know in advance who would show up.

    And my money's on the AD login.

This topic is closed for new posts.

Other stories you might like