back to article 'Major incident' at Capita data centre: Multiple services still knackered

A major outage at a Capita data centre has knocked out multiple services for customers – including a number of councils' online services – for the last 36 hours. Some of the sites affected include the NHS Business Services Authority, which apologised on its website for the continuing disruption and said it hoped its systems …

    1. Dan 55 Silver badge

      Who here didn't know Capita is indeed a single point of failure?

  1. Aristotles slow and dimwitted horse Silver badge

    The realities of Capita IT terminology...

    No single point of failure == Many points of failure.

  2. Anonymous Coward
    Anonymous Coward

    You are all wrong ! it says so here; http://storage.capita-software.co.uk/cmsstorage/capita/files/0e/0eb0f967-7265-4dab-afa5-d6db3f9ffbd3.pdf

    Backup data centre in Laindon, diesel generators ( tested twice a year) , UPSes, etc. so it can't be broken can it ?

    1. John Crisp

      "Backup data centre in Laindon"

      That explains it. The backup gear had all been nicked.......

    2. easytoby

      Unless they hare just plain lying...

  3. Anonymous Coward
    FAIL

    Am I missing something,,,

    "He added Capita has a virtualisation platform which hosts at least 30 clients and many internal Capita systems, "

    The beauty of VM's is you can spin them up in your BAU / DR site....oh wait.

  4. Mike-H

    What if these services had a DR option and the customer didn't take it? There's so much focus on cost these days.

  5. Anonymous Coward
    Anonymous Coward

    I just wonder

    How many of my colleagues find out whats going on by reading the news on el Reg rather than Capita Connections?

  6. Anonymous Coward
    Anonymous Coward

    This is indeed a tragic day for a leading British company. Still at least the weather is perfect!

    1. Anonymous Coward
      Anonymous Coward

      Still at least the weather is perfect!

      For meatsacks not inside a building, yes. But an interesting thought is that summer is now a time of real grid instability, because all that essentially unplanned solar PV dumped on the grid causes huge instability. Varying output (both predictable and not), asynchronous supply, lack of system inertia, all of these cause network and transmission problems. The hippies may b e rejoicing when there's a "no coal" day, but the system operators are sweating, I can assure you.

      And those network stability problems don't need to be absolute failures - just sufficient to push a particular line or substation out of tolerance and trip a breaker, and Bingo! Then you get the knock on effects. I can't say that had any bearing on Crapita's problems, but its a big deal that worries the network operators.

      1. Anonymous Coward
        Anonymous Coward

        "lack of system inertia ... a big deal that worries the network operators."

        Doesn't seem to worry anybody in the UK power industry enough to actually *do* much about it (e.g. invest in robustness). Competing privatised stovepipes is not an obvious way to encourage proper joined up thinking and consideration of the bigger picture - but who knew that?

        Anyway, it's 2017. System inertia, for example, doesn't just come from large lumps of rotating mass. It can come from "synthetic inertia" based on modern high performance power electronics, which achieve the same result as the rotating mass but do it more flexibly, via digital control mechanisms.

        Companies like ABB have, not surprisingly, been doing this at grid scale for a few years now. [In principle GEC might have had a go too, if they hadn't gone bust almost two decades ago, having made a strategic decision to put money in the bank rather than to invest in products and people and technologies.]

        See e.g. this handy summary of synthetic inertia in general:

        http://www.ee.co.za/article/synthetic-inertia-grids-high-renewable-energy-content.html

        and/or for some rather more detailed analysis with a specific focus on wind, there's e.g.

        http://elforsk.se/Rapporter/?download=report&rid=13_02_

        The UK have largely been ignoring these options, preferring to whinge ("insufficient inertia") rather than invest. It's so much more profitable to continue relying on 1960s miracles of engineering such as Dinorwig's fast response pumped storage, and to build relatively quick response diesel generator farms around the country. But other options are available, though some may require people to "think different" and worse still some of the other options may have a short term negative effect on corporate financial results. And apparently that's not allowed.

        [more in a moment]

        1. Anonymous Coward
          Anonymous Coward

          Re: "lack of system inertia ... a big deal that worries the network operators."

          [continued]

          Then again, maybe this (from 2016) is a better late than never sign of better things to come in the UK:

          http://uk.reuters.com/article/national-grid-battery-idUKL8N1B72XQ

          "Aug 26 EDF Renewables, Vattenfall and Eon were among seven companies which won four-year contracts with Britain's National Grid to supply super fast balancing services, National Grid, said on Friday.

          The contracts are the first Britain's power grid operator has awarded to battery storage technology, and were worth at total of 66 million pounds.

          National Grid needs to balance electricity supply and demand on the grid on a second-by-second basis to make sure the system runs efficiently.

          A total of 201 megawatts (MW) of capacity -- roughly the same amount as produced by a small power station -- was secured from seven companies at eight different sites, with the earliest contract starting in October 2017 and the latest in March 2018.

          The amount each company was awarded depended on the amount of capacity offered and how long it would be available for.

          [continues]".

    2. Anonymous Coward
      Anonymous Coward

      Leading? leading on the way down to hell, you mean?

    3. Anonymous Coward
      Anonymous Coward

      "a leading British company" - who's that then?

  7. GingerOne

    How are a company as big as Capita relient on ONE datacentre? Even forgetting their myriad of other failings surely this is reason enough for all of their customers to jump ship and for no one ever to employ their services again.

    I just cannot beleive this. Literally day 1, week 1, IT basics - make it fucking resillient!

    1. Anonymous Coward
      Anonymous Coward

      "make it flipping resillient!"

      Why would the people in charge want to make it resilient? It'll eat into those people's bonuses, surely?

      Until the impact of failure directly hits the pockets of the people in charge, and has a bigger impact than the cost of failure when it happens, those people have no motivation to build resilient systems.

      This isn't the 1990s any more you know, when IT people built systems resiliently **because it was the right thing to do for the customer**, and if you were good as a designer a system that provided critical functions in a degraded mode in the presence of partial failures wasn't always that much more expensive (in $$$$) than a basic setup straight from the box-shifters stocklist.

      Those days are long gone. When did you last read a news item relating to (e.g.) Tandem NonStop, or other high availability technology or techniques? Devops, yes. Kodi, yes. Drones, yes. Resilient systems? Pointers welcome.

    2. fruitoftheloon

      @gingerone

      They aren't....

    3. handleoclast
      Devil

      Re: make it fucking resillient!

      They did make it resilient. Well, the important parts.

      If the guys at the top get fired for incompetence (as they truly deserve) they still get a golden parachute. Big money either way. That's true resilience for you.

  8. Anonymous Coward
    FAIL

    Uh-oh!

    "Good afternoon, my name is Steve in Mumbai. I see that the fault you have reported is complete loss of data centre and failure of DR. I am here to help you with your complete loss of data centre and failure of DR. May I ask you first, have you tried turning your computer off an on again?"

  9. Anonymous Coward
    Anonymous Coward

    Presumably Pay360 customers know the system will be down for 5 (and a bit) days each year.

  10. GingerOne

    Is my place of work an anomoly? We don't have DR because we have a resillient always-on sytem with our own private cloud. I just don't understand why the beancounters in these places don't understand. Yes, good IT costs money, but guess what - it's worth it when shit goes wrong.

    If we lost a datacentre it would be a big worry for the infrastructure team and the rest of us in IT because our resilliency would be affected but the general userbase would carry on working as normal, non the wiser to any problems.

    1. easytoby

      It's an anomaly in comparison to NHS and many public sector and large charity situations. Here the knowledge in the customer organisation to specify and enforce appropriate contracts is missing. Also missing in many cases is the leadership strength to demand proper action on 'difficult' situations.

      1. Terry 6 Silver badge

        Part of the problem is that the bean counters demanding the (illusory) cost savings that lead to outsourcing all sort of services also refuse to pay for/retain the staff that can keep control of it. i.e. You don't just get rid of the school meals service, the cleaners or the payroll etc. you also get rid of the staff from those departments who know what is needed, and how it should be run. In fact, since the options for front-line staff savings are often not that great those supervisory and middle manager staff are the jam on the toast that helps to make the outsourcing costs seem to add up. And middle managers are always seen as a fair target, whereas the top brass on huge salaries always seem to survive.

        (And no, I'm not a middle manager, but I've seen how they and senior front-line staff can make so much difference.)

  11. Anonymous Coward
    Anonymous Coward

    Be prepared

    We keep a spare shilling for the meter nearby. Pah, DR, who needs it?

  12. This post has been deleted by its author

  13. Anonymous Coward
    Anonymous Coward

    Shareholders haven't grasped this yet

    Capita share price up 4.3% today (they were down yesterday as they went ex-div).

    1. Anonymous Coward
      Anonymous Coward

      Re: Shareholders haven't grasped this yet

      Haha! That's nothing, the share price dropped ~50% last year when the profit warning was issued, and never really recovered.

      Two of the directors just happened to have dumped a shit ton of shares the very same day!

  14. PeteCarr
    Facepalm

    No thanks!

    Just had a sales call from S3-Capita trying to flog infrastructure and hosting services. Asked the salesman "Has your data-centre come back online yet?" He laughed uncomfortably, then paused, I interjected, "That'll be a no then, and thanks but no thanks."

    1. Doctor Syntax Silver badge

      Re: No thanks!

      "thanks but no thanks."

      You thanked a (presumably) cold-calling salesman?

  15. Shareholder

    System failure

    Sys failure caused by incompetant directors, caused by a fourth rate HR section that can only select staff by looking at a bit of paper - not on ability. See what can happen!! Have read enough reports showing bad choices. 90% should be removed immediately, before customers leave.

    1. Inventor of the Marmite Laser Silver badge

      looking at a bit of paper -

      that and using LinkedIn

  16. Terry 6 Silver badge

    It is an eternal mystery

    Capita/G4S/whoever can hit the headlines for all the wrong reasons. Do all the potential clients run away from them as fast as they can possibly go? Or do they continue to line up and buy more?

    What would you expect to happen - and what does happen.

    It seems as though when you get big enough no amount of incompetence and failure can be enough to bring you down.

    1. Anonymous Coward
      Anonymous Coward

      Re: It is an eternal mystery

      "when you get big enough no amount of incompetence and failure can be enough to bring you down"

      The concept of too big to fail was pioneered by the banks with great success. I think that other sectors saw the financial crisis, and said "we'd like a piece of that". So Crapita have made themselves a de-facto part of the public sector and too large to be allowed to fail. But not just them. You might argue that there's alternatives to Google, and that Facebook is an unnecessary frippery. But would the US government really let those huge and convenient spying machines collapse if push came to shove?

      As another poster comments, the public sector customers ought to be able to nail Andy Parker's scrotum to a gate post, but won't because they are poor at agreeing contracts, poor at interpreting contracts, and worse at holding big suppliers to account. In fairness, the OP didn't mention Fat Andy's knackersack, but the general drift was there.

    2. Anonymous Coward
      Anonymous Coward

      Re: It is an eternal mystery

      well let's wait an see... they have the whole of the bank holiday weekend to cobble together some kind of solution... if they're not back by Tuesday surely someone will start to ask some serious questions about the outsourcing culture that we've adopted via stealth campaigns over many years... this could become a very hot political potato.

      1. Anonymous Coward
        Anonymous Coward

        Re: It is an eternal mystery

        surely someone will start to ask some serious questions about the outsourcing culture

        What's that?

        Rocking the boat that's floated by the extreme capital investment leverage bought hy putting customers' balls 5 cm over the asphalt at 110 miles/h?

        Not going to happen if people with share options can pretend to be the one company which exploits IT with efficiency that cannot be found anywhere else on the planet.

  17. fruitoftheloon

    the other data centre...

    I left Capita 10 yrs, ago, we had a v v important internal system in West Malling and a 'warm' Dr standby in the other data centre.

    We did a real fail-over test (ironically) on my last day, it worked fine...

    I wonder if some of those afflicted by this fsck up haven't been paying for a warm/hot DR, if not, TOUGH SHIT!!!

    1. Anonymous Coward
      Anonymous Coward

      Re: the other data centre...

      Are those that are trying to fix this even in this country?

  18. Anonymous Coward
    Anonymous Coward

    Just like Pigs...

    ... Capita parts don't fly!

    The anonymous customer gave Capita undue credit when he said "They have probably had to fly parts in from out of the country as the infrastructure is so old."

    Were parts needed for this outage (seems unlikely) then I can categorically say that Capita will use the cheapest means possible to ship them - usually next day courier as immediate couriers are considered too expensive and needs 2 manager approvals. This itself causes untold delays because 1) managers are rarely available 2) bonuses could take a hit so extreme reluctance to authorise persists.

    Also, why should they worry when they're not the ones hurting with system outages when so often the pain is carried by their customers? Generally the take is that if the customer was stupid enough to take out a contract without service penalties then there is no need for them to pull their finger out. When parts are needed the first question (before what part do we need?) is "Are there service penalties?"

  19. amanfromMars 1 Silver badge

    The Revolution will be Virtualised

    Clouds Hosting Advanced Operating Systems in Chaos and Melting Down. Well, well, well ...... Who'd have a'thunk it ...... a Cyber FCUKishima in Dumb Servering Systems.

    And to think that such is only the Start of the Beginning of All that is Planned. Or would you like to think and disagree?

    1. Scroticus Canis
      Happy

      Re: The Revolution will be Virtualised

      I almost understood that. Damn this spliff must be good or Are you back on the meds again?

  20. Anonymous Coward
    Anonymous Coward

    The way to a grand upgrade of the DC's hardware appeared to be not very hard-to-find... and the shareholders would finally welcome this long-awaited opportunity to invest in the stability of their own future income...

    Just a power fault, not the value service infrastructure (-:

    What do you think would be the lower bid and how long will it stay on bottom after *this*?

  21. petetp

    Crapita eats the shit sandwich again.

    How does this company manage to stay in business?

    1. Destroy All Monsters Silver badge

      Just order more sandwiches?

    2. Vic

      Crapita eats the shit sandwich again.

      As the old saying goes, "The more bread you've got, the less you taste the filling"...

      Vic.

  22. cantankerous swineherd Silver badge

    has this got anything to do with the British Hairways clusterfuck?

    1. Destroy All Monsters Silver badge

      Multiple clusterfucks incoming

      Apparently they are not related and BA has denied that any hack occured.

      “Uh, we had a slight computer malfunction, but uh… everything’s perfectly all right now. We’re fine. We’re all fine here now, thank you.” [winces] “Uh, how are you?”

  23. Scroticus Canis
    Facepalm

    Customer service and data are not Mission Critical

    The new Crapita motto.

  24. handleoclast
    Meh

    British Airways

    BA has suffered a "major IT systems failure" that is affecting its global operations.

    Coincidence or another Crapita customer?

    This one is resulting in catastrophic disruption. Lots of delays and cancellations. On a holiday weekend, one of their busiest times. Gonna be a lot of compensation paid out to very unhappy passengers.

    If (it's a big if, I'm guessing here) BA's IT was outsourced to Crapita, BA is going to demand major compensation from Crapita. Council claims for compensation would be trivial compared to this. So if that's the case, and you have shares in Crapita, now would be a very good time to sell them.

    Again, let me emphasise, I'm guessing. Could be no more than coincidence.

    1. Anonymous Coward
      Anonymous Coward

      Re: British Airways

      I feel sorry for the passengers but not Bastard Airways, serves them right for outsourcing to India :

      "The GMB union says this meltdown could have been avoided if BA hadn't made hundreds of IT staff redundant and outsourced their jobs to India at the end of last year."

      Source:

      http://www.bbc.co.uk/news/uk-40069865

    2. Tail Up

      Re: British Airways

      Coincidence. And there's even another one: at the same time a couple of fighter jets were lifted from a Scottish base to keep IT up there :-)

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Biting the hand that feeds IT © 1998–2020