back to article Major IT outage forces UK emergency call handlers to use 'pen and paper'

Tech services provider Advanced has taken part of its infrastructure offline as it tries to contain a suspected security incident, with a range of hosted applications not available to health customers, including NHS 111 emergency services. The outage began yesterday morning and isn't expected to be resolved until early next …

  1. EnviableOne

    Adastra is used by many Ambulance services and Out of Hours doctors too...

    not what the NHS needs right now...

    1. VoiceOfTruth Silver badge

      Maybe a case of too many eggs in one basket. One (possible) compromise = everything affected.

      1. elsergiovolador Silver badge

        Also IR35 meaning severely restricted access to talent...

      2. sad_loser

        Failover?

        If this is mission critical how come there isn’t a backup with failover/ fallback?

        1. Handlebars

          Re: Failover?

          For 999, fallback used to be whiteboards in the control room. Don't know if that's still the case.

  2. Anonymous Coward
    Anonymous Coward

    Out of hours doctors? They did exist at one time but no longer. It's all 111 after the surgery closes. Isn't it MODALITY?

    1. midgepad

      111

      Passes on a bit more when there are failures.

      To fewer, tireder, doctors, with more calls on them jn hours and out.

      And a government that for 12 years has been trying to damage the health service. Do try not to elect them again.

      1. Peter2 Silver badge

        Re: 111

        Another point of view is simply that there is a finite amount of money available. The NHS now costs ~7.5% of GDP a year and it has long passed the point that you can keep increasing the amount available to it every year by percentage points of GDP every year is sustainable.

        What the NHS is willing to do depends on how much money it has, and so if you continue to increase the amount of money then they will continue to find things to spend the money upon and it has passed the point that this is sustainable.

        Personally, I feel that the NHS needs at least 3 distinct budgets. One for reactive maintenance (A&E?) one for long term maintenance (quality of life issues such as hip replacements) and another for what amounts to end of life care so you don't end up with one part of the NHS draining all of the funds from the other parts of the organisation.

  3. adam 40 Silver badge
    FAIL

    BRING BACK PARIS!

    The post is required, and must contain letters.

    1. The Oncoming Scorn Silver badge
      WTF?

      Re: BRING BACK PARIS!

      He's right......

      Now that there's a gaping hole in the icons available See - Ic.....Ohhh!

      Can we have a TARDIS/DNA/Pterry one's?

      1. cyberdemon Silver badge
        Unhappy

        Re: BRING BACK PARIS!

        Wtf, what happened to Paris?

        1. Fruit and Nutcase Silver badge

          Re: BRING BACK PARIS!

          I'll be cancelling my subscription

  4. NATTtrash
    Headmaster

    To doctor or not to doctor...

    Publication Pulse Today said yesterday a letter from NHS England's regional team warned General Practitioners – doctors – in London ....

    Sorry, but "doctors" are people who have PhDs, and I suppose we all are not looking forward to being treated by a doctor in <fill in your personal favorite non-medical discipline>.

    With that definition, and considering the El Reg commentards, we have some here who would be qualified to give you that prostate check up...

    Then again, "title creep" seems to be wide spread. Even dictionaries seem confused...

    physician

    noun [ C ]

    mainly US or formal /fɪˈzɪʃ.ən/

    (UK usually doctor)

    a medical doctor, especially one who has general skill and is not a surgeon

    The latter addressed correctly as "mister" and not "doctor" of course...

    1. ITMA Silver badge

      Re: To doctor or not to doctor...

      "(UK usually doctor)

      a medical doctor, especially one who has general skill and is not a surgeon

      The latter addressed correctly as 'mister' and not 'doctor' of course..."

      Sorry but don't try and impose your Americanisms here in the UK especially as this is a UK based item.

      Plus UK surgeons can often also be referred to as "doctor" depending on their experience and seniority. You are confusing them with the much more senior "hosptial consultants" who are called "Mr/Mrs/Ms" to differentiate their higher seniority from more "lowly" doctors.

      https://www.bma.org.uk/advice-and-support/international-doctors/life-and-work-in-the-uk/toolkit-for-doctors-new-to-the-uk/doctors-titles-explained

      1. Fruit and Nutcase Silver badge
        Facepalm

        Re: To doctor or not to doctor...

        Before qualifying, medics wait for the day they can be addressed as Doctor.

        Once attained, they can't wait to be addressed again as Mr/Miss/Whatever, when, with experience, further qualifications and a bit of luck, they become a Consultant

        1. ITMA Silver badge
          Devil

          Re: To doctor or not to doctor...

          Absolutely...

          Once consultants, especially surgical consultants, they can then behave like Sir Lanceleott Spratt:

          https://www.youtube.com/watch?v=oVWjAeAa52o

          And while were on the subject of "titles", why is that some in the USA seem to get very upset about who can and can't call themselves "engineers" again all based on bits of paper and/or membership of a professional body. Despite some not actually being able to physically make anything themselves.

          And yet in the USA you call train drivers "engineers". We prefer to call them more accurately "train drivers".

          1. Fruit and Nutcase Silver badge
            Joke

            Re: To doctor or not to doctor...

            Our Members of Parliament expect to be addressed as "Honourable"...

            Icon - >

          2. J.G.Harston Silver badge

            Re: To doctor or not to doctor...

            I was surprised when my dentist Mrs X retired her replacement was a Dr Y.

        2. midgepad

          Re: To doctor or not to doctor...

          Fellowship, (in/) the FRCS, not a consultancy.

          It is a bit outdated.

          Probably several sociologists could earn their doctorates of philosophy (PhD) by contributing new understanding of why.

          Meanwhile, doctorates in Philosophy followed doctors doing medicine and when natural philosophers got going theiy aspired to doctorates.

          Philology, etymology, that's where to look.

      2. NATTtrash

        Re: To doctor or not to doctor...

        I'm sorry too, but no American passport or location here, nor meant in the comment. Just Bologna Process, which is Italy ;)

        But having said that it is also fair to say that the UK isn't the only one. That is the thing with human beings. Always looking to have something that distinguishes them from others. That is for example why some of my German colleagues (not that passport too before somebody thinks so ;) find that their biz cards are too small to fit all those prof. habil. dr. dr. dr. med. on there. Which is certainly not Bologna too. And since Bologna is in Italy, I am still surprised to see that there (not that passport too) people award me dott.ssa...

        1. ITMA Silver badge

          Re: To doctor or not to doctor...

          Don't worry about it.

          One of our UK "eccentricities" LOL

          And and as Fruit and Nutcase quite rightly pointed out

          "Our Members of Parliament expect to be addressed as 'Honourable'..." and certain MPs on the Privy Council go by the title Right Honourable.

          More like "the wheezers and dodgers" (though such a comparison is likely to insult the real Wheezers and Dodgers: https://en.wikipedia.org/wiki/Department_of_Miscellaneous_Weapons_Development).

          And as for "honourable" - if they shake your hand, be sure to check your wallet and watch after...

  5. Anonymous Coward
    Anonymous Coward

    Talking about this

    With a friend of mine from a European country, who's still in the emergency services (as I was, about five lives ago).

    He was complaining about the fact that nowadays all the logistics and despatch stuff is to some extent or another "cloud-based", which relies on luxuries such as internet access or a functioning power grid, neither of which is a given in the event of a major issue.

    Curiously, I understand the French system, while it kind of sucks day-to-day because it's so Spartan in some ways, is designed to be extremely resilient (much of the emergency response in France is either paramilitary in nature or down right military, as the gendarmerie and the Paris and Marseille fire battalions).

  6. TimMaher Silver badge
    Flame

    Per Ardua....

    .... Adastra.

    111 has become even worse than it was when it started. It is now completely useless.

    Even 999 has deteriorated with some especially crap number pushing in some circumstances.

    I had to phone both of them a couple of weeks back and it was really hard work.

    I’m not talking about resources or waiting times here, I’m talking about the procedures.

    Not good even before an outage.

    1. ITMA Silver badge

      Re: Per Ardua....

      We've had a fleet of new Fiat ambulances which a significant number of ambulance crews can't fit in properly to drive!

      https://www.bbc.co.uk/news/uk-england-cambridgeshire-61847310

    2. Fred Daggy Silver badge
      Pint

      Re: Per Ardua....

      I find the number 0118 999 881 999 119 725 ... 3 quite snappy. Never needed it in real life but worked well in an emergency drill.

  7. Anonymous Coward
    Anonymous Coward

    It's advanced

    To be fair one of their own people probably tripped over a plug

    1. Anonymous Coward
      Anonymous Coward

      Re: It's advanced

      Hope they didn't hurt themselves too badly, it's a bugger of a job getting an ambulance ethse days

  8. mIVQU#~(p,

    Why are these systems exposed to the internet.

    1. john.w

      Because it is rather fundamental to the service they are providing? The bigger the intranet the more it becomes to look like the internet.

    2. storner
      Boffin

      They need not be. Only takes one bad click on an email and your internal network is exposed.

      Most cyber incidents these days happen that way - attacking from the outside is a lot more work unless you have *really* poor security.

  9. john.w

    Miscriants carrying out an act of War?

    A visit from those based just outside Hereford would be possible response for this type of attack.

  10. Tony.
    Holmes

    Hackers.. .

    we await on this being blamed on "state sponsored military grade cyber-attack"... which usually just means some PFY with more access than they should have clicked on 'invoice.pdf.exe" they got in an email....

  11. ChipsforBreakfast
    FAIL

    Fail to plan, Plan to fail...

    We know systems will be compromised. It's fast becoming inevitable, no matter how comprehensive the security or how determined the management are when all is said and done defenders need to be lucky all the time, attackers only need to be lucky once. We need to start accepting that when we're designing, specifying and commissioning systems for critical infrastructure.

    We need to look at enforcing diversity - multiple systems, multiple suppliers, multiple architectures, multiple access paths. Yes, that will introduce problems with interoperability but those shouldn't be insurmountable with careful design. We need to be mandating that level of diversity across the entire critical infrastructure - no one supplier, system or datacenter should be permitted to operate in more than a quarter of any given segment of critical infrastructure.

    Monocultures are notoriously fragile (windows, I'm looking at you!) - they need to be avoided when dealing with life-critical services. Only by promoting diversity and actively discouraging monoculture can you both promote innovation and enhance the resilience of the system as a whole.

  12. Binraider Silver badge

    Question. Would running on pen/paper all the time be cheaper and more efficient?

    IT can not solve every process. Questioning where IT adds value is something often forgotten in system design.

    1. midgepad

      The answer is known*

      Dragging it all into one big system and trying to use it to program people less so, though

      * no

  13. IanRS

    High experienced?

    I cannot help but think that if your incident response team is highly experienced then you should be fixing a few issues before they become incidents.

    1. tomuk

      Re: High experienced?

      Critical Systems should be behind the HSCN network which is not public facing and you need a HSCN line to access.

      If this wasn't the case why?

      Also, I have never known any audit of the requirements for an HSCN line and have seen the form fraudulently filled out to get one. This means there is bound to be so many weak points to get onto this network.

      Was when I was in education but we struggled as a school to get Advanced to meet our internal security expectations. They were eventually met. Again you are still down to the weakest member of staff working on your service.

  14. Rob Dyke

    Guaranteed availability lolz

    I checked out the gCloud for Advanced's products and oh my

    To start "Data is synchronously replicated between the two centres using Microsoft and web technologies including Hyper-V Server virtualisation and SQL Availability Groups. With the technology enabling a real time replica to be created at an alternate location, it provides operational continuity in the unlikely event of a partial or complete failure of either centre."

    Then "Data is then in turn backed-up using Dell DR4100 Disk Backup Appliances, de-duped and then replicated in the alternate data centre. A full weekly backup is taken and a daily differential backup, each one being automatically verified with any inconsistencies automatically and immediately flagged to our team of database administrators."

    And just in case you are not yet convinced "A combination of the data centres’ significant infrastructures and our active/active operating model therefore delivers a flexible, robust and proven approach to disaster avoidance and resilience and provides a high degree of business continuity"

    gCloud

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like