back to article Power cut interrupts UK.gov cloud service supplier

A local power cut has interrupted the services of UK government cloud provider Memset. Memset, an SMB supplier on G-Cloud, saw power to its data centre in Reading severed at 8.39am on Tuesday with the company claiming to have resolved the problem half an hour later. Both primary and secondary power suppliers to Memset's …

  1. Voland's right hand Silver badge

    The outage continue till ~ 11-ish

    There was fallout until around 11-ish. The biggest outage memset has had during the 10+ years I have been using them.

    It is also the first time ever for their system to show BS - it was showing VMs as LIVE while they clearly were not.

    1. andyheat

      Re: The outage continue till ~ 11-ish

      I think that only relates to the setup status - I.e. Pending, live, on-hold, expired.

      Their is another "server status" indicator on the server detail page itself.

      However I wouldn't mind betting at least some of their monitoring servers are in Reading...

  2. Phil Endecott

    Anyone else read that as Mumsnet?

    1. Will Godfrey Silver badge
      Happy

      So it's not just me then {phew}

      1. m0rt

        Same here.

  3. Dave Pickles

    Dad's Army Diesel

    I wonder if there is a case for data centres with backup generation to join the National Grid Short Term Operating Reserve (STOR) scheme - nicknamed "dad's army diesel" by El Reg a while ago. There would be some extra cost for switchgear and synchronisers, but the generators would get to run more often and there would be a financial incentive to keep them in a better state of maintenance.

    1. MK_E

      Re: Dad's Army Diesel

      Not to mention that running for STOR on a semi-regular basis will let them spot problems with the generators before they actually need them for real. If they fail during STOR, it's nowhere near the disaster that failing during an actual power cut would be.

  4. Anonymous Coward
    Anonymous Coward

    I am going to assume this is the same outage that affected hundreds of customers including myself with name.co.uk / simply hosting AKA poundhost this morning?

    1. andyheat

      I thought exactly the same, but after making this known on Twitter Simply/Poundhost's commercial manager phoned me to insist they own their own data centre and it couldn't possibly be the same problem.

      Although it was suspiciously the same circumstances, power outage followed by generator failure... hmm.

      1. Paul 164

        Our SimplyHosting server fell offline at 8.35am - around the same time as this company's servers went offline... Too much of a coincidence for me.

      2. Anonymous Coward
        Anonymous Coward

        The video datacentre tour on Simply's website is the Everest facility, so that answers that one.

        I was there this morning, no customers seemed to be allowed in until they'd essentially fixed everything and almost everything was responding again which seemed a little strange. Also no explanation has been given to customers for that yet.

  5. John Smith 19 Gold badge
    Unhappy

    Somebody not testing their DR precautions?

    I think so.

    1. Anonymous Coward
      Anonymous Coward

      Re: Somebody not testing their DR precautions?

      The thing with nice diesels is that they do need to run every so often or you invite all sorts of problems. A fun deceptive one is sediment formation in tanks and filters - you get a nice start, think it's all OK and cut over power, only to find it die a little later when its fuel line clogs up..

      1. David Shaw

        Re: Somebody not testing their DR precautions?

        Yes, I saw this happen 'live' at a 'telco' MIx trunk access node when we tested the mightily impressive twin Rolls Royce diesels;

        "Yes, all OK power is . . .oops" as multiple 565megabits WDM fibres went rather too dark, and (most) traffic was re-routed via Edinburgh

        the 'worser' thing was on the cable laying ship when the repeater went overboard - 'but I thought *you* had connected the fibre'

  6. frank ly

    Side by side

    "Both primary and secondary power suppliers to Memset's facility failed."

    Were they laid in the same trench? It does save installation costs.

    [Many years ago, a certain organisation had dual redundant internet connections from two separate suppliers (very sensible) that arrived from opposite sides of the site and went into sensibly designed network switches, etc. They both failed one day.

    The ISP suppliers had both contracted the same cable laying outfit who put the cables through the same conduit, when the route went over a bridge, and a large truck crashed into the conduit. Trust no one.]

    1. Anonymous Coward
      Anonymous Coward

      Re: Side by side

      A decade or so ago I was visiting Basingstoke and North Hampshire hospital while work was going on in the car park and indeed in many places around the main hospital building. It was in the winter, and late afternoon, so not very bright outside. It's a quite busy regional hospital.

      There was a bit of a bang from somewhere, and the lights went off, as did everything else that was mains powered. People in the know waited. And waited. The backup didn't come on. No mains power anywhere in the main building.

      Gradually, people in the know started being shit scared. Understandably. I wasn't just scared (despite being just a visitor passing through) but I was effing angry too. Not only had some inadequately-supervised cowboy in a backhoe been allowed to dig through the main incoming supply, the generator backup's cable was (as hinted a few posts ago) in the same trench and consequently also taken out.

      You'd maybe think it couldn't get any worse, but work was going on in the main hospital corridor and there was a full height "temporary" (as in, "many weeks") partition along most of the corridor's length. The emergency lighting (with local battery power) along the corridor was lit but useless because some idiot had allowed the temporary partition to be built in front of the emergency lighting and done nothing about providing temporary emergency lighting while the real thing was obscured.

      So there are people rushing about the hospital trying to respond to a "should never happen" condition, and the main corridor is (by now) in *total* darkness apart from a few people with hand-held battery lighting.

      Absolutely appalling. Amazingly, I don't remember it being in the local or national press at the time.

      In slight contrast, a few miles north of there, and rather more recently, Royal Berks (Reading) hospital partially lost their electricity and declared a major incident, closing A+E. Well you would wouldn't you, especially if you hadn't properly tried your DR plan recently:

      http://www.telegraph.co.uk/news/nhs/11776054/Royal-Berkshire-Hospital-AandE-shut-after-power-cut.html (31 Jul 2015)

  7. Anonymous Coward
    Anonymous Coward

    Poundhost / simplyhosting too

    Possibly there's some shared data centre as my dedicated server went out at exactly the same time. The twitter feed was good though, they've acknowledged a problem with backup generators not kicking in, the USA based different data centre server status page - not so good though showed everything apparently running normally until long time after the twitter.

    There's also likely to be intermittent interruptions so they can test the power back up and I think it's good that they've (pound host) have kept the customer informed when they are trying to turn it on and off again in time honoured helpdesk fashion.

    1. Paul 164

      Re: Poundhost / simplyhosting too

      Our server remained offline until 5.25pm. We were not kept up-to-date and were fed copy-and-paste Twitter replies to any questions asked of them.

      The situation was so bad that the boss made me move everything from the server we have with them to another company we have servers with.

      Bye bye SimplyHosting!

  8. Ken Smith

    IBM in Hursley ran their backup generators once a week when I worked there. The noise near my office was really irritating but the testing was worth it.

    I have had servers at Memset for years and their service has always been top notch. This is the first outage of any consequence.

    1. Anonymous Coward
      Anonymous Coward

      Re: hursley diesel generators.

      Those are actually the pumps for the fire suppression system, not electrical generators for the DPMR.

      Hursley has redundant power from the grid (from opposite sides of the site)

      Anon for reasons.

  9. tin 2

    I love how generator failure becomes "Issues transitioning on to generator power"

    1. Anonymous Coward
      Anonymous Coward

      Given the levels of redundancy in modern data centres, a problem transitioning to generators is more likely to take everything out than a generator failing.

  10. Anonymous Coward
    Anonymous Coward

    Not only power issues

    We were quite badly affected until around 12ish - not everything came back online cleanly - and they also had a network issue that you havent reported - I did mail into your newsdesk ;)

  11. Anonymous Coward
    Anonymous Coward

    "Everest managing director Ed Butler told The Reg a "small number" of clients in Reading had been affected but would not comment on Memset."

    From what I heard the whole site lost power, I guess the site has a "small number" of clients given that they all would have been offline.

  12. Peter Christy

    In complex systems, its often difficult to spot "single point of failures". Many years ago, I used to work for a major broadcaster. The videotape machines back then required compressed air for the bearings in the video heads. We had three compressors, and the whole area could run on any two. Half the area could run on one. I walked in one morning to find the whole area shutdown due to a compressed air failure! Why? Because the compressors were water-cooled, and workmen digging up the road outside had breached the water main!

    I worked for a while at remote transmitters. One of the main transmitters covering the north of England had three 27-litre V6 marine diesels as standby generators. The oil and water was kept up to temperature by electric heaters, so that if the mains failed, we could start the generators almost instantly. Even so, we ran one up for an hour each week in rotation, so that we KNEW they would work when needed.

  13. Anonymous Coward
    Anonymous Coward

    Power cuts interrups UK...

    If we have a cold winter, that could extend across more than just the web.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like