back to article Capita data centres hit by buttload of outages

Hard-pressed outsourcing titan Capita wrestled with almost sixty separate outages at its data centres in the space of just half a year, The Register can exclusively reveal. An internal log of the problems dating between June and December 2017, seen by us, detailed the extent of the downtime, with as many as 44 incidents …

  1. Tom Paine

    Statement of the bleedin' obvious

    Is a lot. A lot a lot. Yes they probably have a ton of DCs and systems, but the support org is supposed to scale also g with the estate - especially if as seems likely there's a massive array of disparate legacy systems that require their own dedicated support teams.

    1. m0rt

      Re: Statement of the bleedin' obvious

      'Lewis branded the company as "too complex", said it was "driven by a short-term focus", and "lacks operational discipline and financial flexibility".'

      No shit. Seems this is the fashion for most companies since the original .com boom/bust.

      Remember five year plans? I do. Last one I came across in a business I worked for was '97.

  2. SVV

    60 is not a high number

    No, wait til they collapse - THAT will involve lots of REAL high numbers.

    1. Tom 38
      Angel

      Re: 60 is not a high number

      "60 is not a high number, it was way above that until the DC that records our outages went down last month"

  3. wolfetone Silver badge

    I can see their point

    60 isn't a high number.

    61 is a high number.

    But 60 isn't high.

    1. AndrueC Silver badge
      Joke

      Re: I can see their point

      60 isn't a high number.

      61 is a high number.

      62 is right out.

    2. phuzz Silver badge
      Trollface

      Re: I can see their point

      I think in the US they consider 420 to be a high number.

  4. Anonymous Coward
    Anonymous Coward

    If these are really data center hardware/facilities and not application related

    then 60 is phenomenally high. I have been in IT 30 years and never had a year getting near 10 let alone 60.

    1. Anonymous Coward
      Anonymous Coward

      Re: If these are really data center hardware/facilities and not application related

      You should see the number of Infrastructure P1/P2 faults (non application) at EY - it's easily that a month.

  5. ForthIsNotDead

    Data centres...

    A water company that I'm currently working for is currently "learning" the "challenges" of having ALL OF YOUR FUCKING DATA on the *OTHER SIDE* of a broken data link.

    Ho hum.

    There's a sort of quiet hush in the office today, and lots of distant phones ringing (a lot) down the corridor.

    And it's not in the customer billing department, if you know what I mean.

    Me? Can't access my shared drives so I'm browsing El Reg on my (personal) laptop over a mobile data connection.

    1. JudeKay (Written by Reg staff)

      Re: Data centres...

      Our newsroom email is news@theregister.co.uk. Just sayin'

      - A friendly Vulture

      1. choleric

        Re: Data centres...

        If you publish your address like that you'll get a flood of emails.

        1. JudeKay (Written by Reg staff)

          Re: Re: Data centres...

          Glad you've fathomed what we're after. We're just fishing for replies about your outages and the glassbowl antics that made them happen... Seriously, though - we can protect your identity.

          1. Crisp

            Re: Data centres...

            Oh gods! Water load of puns!

          2. Anonymous Coward
            Anonymous Coward

            Re: Data centres...

            "Seriously, though - we can protect your identity."

            Can confirm, lovely people

  6. Lysenko

    No surprise...

    The Maldives Basketball Association members running outfits like Capita hear the term "redundancy" and automatically equate it with something you need to get rid of.

    1. macjules

      Re: No surprise...

      We generally refer to Capita management as the City University Netball Team.

      1. Sam Therapy
        Coat

        Re: No surprise...

        No doubt they meet on Tuesdays and end their get togethers with the traditional "See you next..."

        Coat... getting...

  7. Missing Semicolon Silver badge
    FAIL

    Capita is "too complex"?

    Then you, sir, are another highly-paid exec who frankly isn't up to the job.

    Jonathan Lewis:

    If you can't do the detail for the organisation you lead, then go find a smaller one and leave the job to a real manager who actually knows how to run a multi-million pound business.

    1. tiggity Silver badge

      Re: Capita is "too complex"?

      But most of these CEOs dont know / care how to manage a business. It's all short termism as their bonuses are typically tied to short term targets.

      Until there's a delayed gratification approach - in that long term improvement is rewarded, then things won't change (& no chance of that - can you see a CEO waiting 5 years for a bonus based on long term genuine value added improvements to a company)

      1. lglethal Silver badge
        Go

        Re: Capita is "too complex"?

        Funny I read that "too complex" comment completely the other way - in that he recognised that the company had not been following the KISS principle (I doubt I need to enunciate that, but here it is anyway - Keep It Simple Stupid).

        If Crapita finally has a CEO coming out and saying "hey, we have made things way more complex then they need to be and we need to fix that." then I'd say that they might actually have found a CEO who finally understands the problems of the business...

        Still I completely agree with Tiggity that all CEO bonuses should be delayed by 5-10 years to ensure some long term thinking.

        1. Anonymous Coward
          Anonymous Coward

          Re: Capita is "too complex"?

          Still I completely agree with Tiggity that all CEO bonuses should be delayed by 5-10 years to ensure some long term thinking.

          If all the bonus is delayed for that timescale, then crap managers benefit from remedial actions by competent successors (if any, natch), and at the individual level, chances are that your bonus this year comes from the last job you did, or the one before that. Effectively, that would encourage MORE seagull managers, not less - why work hard, stay in the same job for bloody years, when the rewards are the same for flitting around doing not much, and never staying long enough to take either decisions or flak?

          What would be much more effective than managing the bonus structure would be retrospective bonus clawback going back up to ten years. So if Crapita, Carillion or whoever end up in the shit, the Insolvency Service should go back and publicly recover money from those senior managers who ran the business as the problems were built up. Even if they've squirrelled it away or spent it, at least they'd be publicly humiliated, but in most cases they will have extensive pension promises that could be reduced to if they haven't got the cash. If those rules had been in place a decade ago, Fred the Shred and his pocket-picking cronies could have been held partially to account for the mismanagement of RBS, instead of retiring on a million quid a year pension paid for by the bank we all had to bail out.

      2. low_resolution_foxxes

        Re: Capita is "too complex"?

        Not always true. The average idiot sales team selling buttons will always run in 3 month sales cycles, there are other industries out there that work to very long lead times.

        Large petrochemical plants can take 3-8 years to get through the design spec/permit/construction phase. Many UK offshore wind turbine sites (round 3) have taken 10 years of development to get this far and will take another 5 years to complete.

        Tesla has spent 10 years designing the Model S and getting the Gigafactory built (nearly there).

    2. csecguy44

      Re: Capita is "too complex"?

      There appears to be no said or implied requirement for a "head/manager of IT" to actually have any technical expertise at all. Surely at least a bit of "good understanding" would go a long way.

      Despite the above, I often see people on charge to have absolute 0 knowledge at all. When I'm asked if new Windows 10 computers will work with the existing Cisco switches, my alarms are all starting to go off.

  8. Duffaboy

    Bagdad bob is back I employment

    "There are no allied tanks in Bagdad " "and there are definitely no outages either"

  9. adam payne

    A Capita spokeswoman said: "Capita has a cloud-first strategy and it would be incorrect to attribute migration to anything other than our strategy."

    Sorry but no one is going to believe that.

    1. Commswonk

      Sorry but no one is going to believe that.

      "Sorry but no one is going to understand that" might have been closer to the mark.

  10. Anonymous Coward
    Anonymous Coward

    Well we are not going to be hiring any new staff with a recruitment freeze that is currently active are we.

    1. Anonymous Coward
      Anonymous Coward

      I daresay the solution to the problem will be more offshoring, not less. So no UK recruitment, but lots of new opportunities for cheap and crap drones in the offshore markets.

  11. Anonymous Coward
    Anonymous Coward

    Its all Relative

    As someone who used to work in Capita, 60 isn't high for them!

    I remember a time when it was over 20 a month.

    The problem is the senior people only seem to listen to the MD's and they are over paid and Teflon coated ensuring none of their decisions which mislead customers and screw over staff ever come to land on them. The few good MD's have mainly left, which only leaves those which either enjoy banging their heads against walls or those who have managed to evade every rock thrown at them.

    Capita is the architect of its own issues, always looking to make money out of itself by charging the internal businesses rates which are well in excess of those externally, which in turn just pushes up the prices for its real customers. Even in the DC when urgent maintenance is needed, the people there know that but their changes get rejected by the internal businesses so they are always on the back foot and getting kicked for P1's which they tried to prevent, and have to work with the worst spec systems around.

    The datacentre is apparently currently running a refresh programme for its networks but again it went to the cheapest supplier not one which could give better visibility into issues, or better protection for the huge amount of our data they hold, and which has probably already been leaked out without the 1990's tech realising it.

    If Mr Lewis wants the real story I would sit down with the people on the front lines and the customers, it will hurt ALOT but its the only way to understand the businesses and sort out the complexity, which is only their to shield the people who should have been booted out years ago!

    1. Anonymous Coward
      Anonymous Coward

      Re: Its all Relative

      I enjoy my job - Capita has been a good employer, as in they are like the absent dad that has little thought for his offspring, but the maintenance direct debit is too much aggro to cancel...

      but I too echo your sentiments... having worked for the company for nearly 2 decades, the man at the top needs to talk to, not the frothing masses, but to ground level staff that have some exposure to the pointy end of process... the lack of resourcing and cutbacks to make short term targets is painful and self destructive...

      I would really love to just enjoy a civillsed chat with Mr Lewis... and that's not meant as a blood thirsty threat!!

    2. This post has been deleted by its author

  12. Wolfclaw

    What's an outage, 7 years with my current contract and not a hour down :P

  13. Jim McCafferty
    Happy

    Pot and Kettle

    The article suffered an outage in the first sentence...

  14. Anonymous Coward
    Anonymous Coward

    This isn't unique to Capita...

    ... lots of short-term thinking out there amongst it's competitors too.

    I used to work for an outsourcer in the UK with a similar managerial outlook. The last two years of my employment became a recurring trend - hauling a managed environment out of the coals because an outdated bit of kit had fallen over, or some idiot beancounter had laid off the entire team responsible for management of a customer solution and it merrily plowed on for a couple of years on it's own before nosediving. It was 2-3 of these incidents every month without fail.

    Flagging these issues up to the management was met with silence, no plan at remediation and no appetite to fix the issue. If it involved spending money or someone taking responsibility for something, they didn't want to know.

  15. Anonymous Coward
    Anonymous Coward

    Connected?

    Would they be using the same cloud provider?

    http://www.bbc.co.uk/news/technology-42839462

  16. Anonymous Coward
    Anonymous Coward

    I can't really understand why people are surprised

    the customer buys all the kit at the beginning of the contract.

    They are gouged for the duration of the contract term.

    There are not explicit contract provisions to keep anything maintained. There is no incentive for provider to maintain proactively. Indeed provider can't use centrally managed resources as you can't bill for all the new and shiny stuff for all the new prospect customers.

    Hence wall of silence from internal management as there are only incentives to hide the problem and carry on while customer is too disconnected from the actual solution to realise they should ask...

    This is normal, culturally embedded in both customer senior management (being blind to what happens 5 years after signing up) and to the outsourcers senior management only wanting to rake in margins for a job they have no intention of doing.

  17. Anonymous Coward
    Anonymous Coward

    Cloud Storage Strategy

    A bit rich to have a cloud storage strategy when you have launched your own cloud system "Capita Private Cloud", so essentially the same old strategy of putting it in their Data Centre, but calling it cloud!

    I wonder if someone will leak the outages for their Networks business, i bet that would make interesting reading!

    1. Angry IT Monkey
      Coat

      Re: Cloud Storage Strategy

      "I wonder if someone will leak the outages for their Networks business, i bet that would make interesting reading!"

      Maybe they keep trying to but the connection isn't stable enough to get it out!

  18. Anonymous Coward
    Anonymous Coward

    This may explain why they keep sending me threatening letters to pay my TV licence, which I did again in January. They do seem to be the only people in the country that have my new address as flat c when it is actually flat 3.

    1. Anonymous Coward
      Anonymous Coward

      Send a 3heque, payable to 3rapita ...

    2. Anonymous Coward
      Anonymous Coward

      "the only people in the country that have my new address as flat c when it is actually flat 3."

      A 'friend' had been trying to sign up for HP's Instant Ink subscription or a few months.

      The address is in a block of 1970s flats with around 40 flats all with the same postcode.

      Friend had tried signing up online, several times, each attempt had mysteriously failed with a really hellpful error message e.g. "An error occurred, pleas try again later".

      Eventually the friend gave up and called the support phone line and the helpful (yes, honestly) person talked through the process for half an hour or so.

      One of the various problems was that whatever address database was being used didn't want to allow flat numbers *at all* for this postcode, although every other address database for every other online application allows (or even requires) the entry of flat numbers.

      Maybe HP Ink have outsourced their order processing to Crapita. Or vice versa. What a combination that would be.

  19. CheesyTheClown

    Entertaining read

    Mainframe tech is specifically designed for handling these problems. Using “everything is an object” and “share nothing” tech, it should be possible to run these systems for decades without an outage. Using big-ass custom cloud platforms is a quick way to end up screwed.

    This is why companies like Google, AWS and Microsoft are dumping IaaS and containers in favor or more reliable FaaS architectures. Yes, there were problems from the 60s to the 90s, but mainframes have generally always been far more reliable that the crap most vendors are passing as PaaS today.

    Acid compliant share nothing record storage systems as well as FaaS as well as well designed load balancers and non-SAN object storage can offer almost zero-downtime (way better than five nines) platforms.

    A vendor like this has absolutely no possible excuse for service outages. If they can’t do it themselves, they should call IBM and get it done right. It’ll cost a lot, but nowhere near as much as losing business due to letting IT people be involved in information systems.

    1. Craigie

      Re: Entertaining read

      'call IBM and get it done right' - lol wat?

  20. MMR

    Ummm

    Here is an idea. Why Capita don't outsource their IT?

  21. Anonymous Coward
    Anonymous Coward

    Saw a Capita building safety helmet at work.

    Working in a hospital, I was not encouraged to notice a Capita safety helmet on one of our building projects recently.

    What is it going to take down when it fails?

  22. Anonymous Coward
    Anonymous Coward

    Crapita.. cry me a river..

  23. Diogenes8080

    This thread may require resurrection.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like