back to article Brit cloud slinger iomart goes TITSUP, knackers Virgin Trains, Parentpay

Brit cloud provider Iomart is having an outage-ridden day - but its service-outage-hit customers are arguably having a far worse one. Reports first began to surface on Twitter at 12:26 this afternoon with users of Iomart subsidary Melbourne IT noting some problems. @MelbourneHost @iomart Problems?? — Nutty About Hosting (@ …

  1. Blockchain commentard

    Easyspace has the same issues - coincidence?

    1. Charlie Clark Silver badge

      Telegram was also down in EMEA for about an hour about 09:45 GMT.

      I wish these companies would run monitoring and status services on other systems.

    2. This post has been deleted by its author

    3. Anonymous Coward
      Anonymous Coward

      Same company so no, not a coincidence

    4. Trev 2

      They are using an identical tweet about the problem and they mention data centres in the same locations, so presumably they are the same company these days. I know Iomart went round buying up lots of hosting providers a few years ago, but don't know who bought who in this case.

      On a side note, it's nice to see all the bumf from Easyspace about reliability including "Cross Connection to a number of Tier 1 carriers" - so it seems that didn't work too well. Suggests that maybe it's a config (aka upgrade just before Easter went badly wrong) issue rather than a hardware break?

      1. Anonymous Coward
        Anonymous Coward

        I'm an Easyspace customer and yes, they're the same company. Luckily, I'm in Maidenhead.

        "Suggests that maybe it's a config (aka upgrade just before Easter went badly wrong) issue rather than a hardware break?"

        You may be more right than you know. There is planned maintenance at Maidenhead over the weekend. The maintenance shouldn't have downtime - but I suppose if someone accidentally applied the update to both failover partners at the same time, with the potential for a failed upgrade to content with, that could bring down a data centre quite easily...

    5. Anonymous Coward
      Anonymous Coward

      Easyspace are a subsidiary of Iomart, and almost all their kit is(or certainly was) in Glasgow, one of the 2 DCs that got toally cut off...

      1. TRT Silver badge

        As is hostingUK.

  2. Nate Amsden

    sort of reminds me

    Was at a ATT conference maybe 9 years ago. They had one bit where they talked about multiple provider network redundancy. Then they showed a slide of a (real)huge fiber cut that had probably a dozen or more carriers in the same pipe, I want to say the pipe or conduit was 1 to 2 feet wide buried in the ground.

    Certainly was interesting to think about wrt datacenters and normal requirements that there be at least 2 fiber connection points on different sides of the building. How far might those fibers go before they converge? (If they converge) have never gotten a confident answer for that.

    1. Anonymous Coward
      Anonymous Coward

      Re: sort of reminds me

      The issue is not the cut, but the lack of cross-provider connectivity.

      97% of all UK provider networks interconnect is at the LINX. While it is now geographically distributed, it is still only around London.

      In the event of a double cut, if the networks were interconnected elsewhere as well as the London area one could buy temporary transit and stitch its network via a tunnel for a couple of hours.

      Unfortunately that is not the case.

      1. rh587

        Re: sort of reminds me

        According to the record on PeeringDB, they do peer at IXManchester (albeit only a 10G connection, piffling compared to dual 80G at LINX).

        Given that they have a presence in Equinix Manchester Williams/Kilburn (MA1), they could presumably pick up some transit from a provider there as well.

        Seems more likely (as per Alister's experience in Derby) that the fibres coming out of the affected DCs are all in the same duct for some distance, meaning that even with redundant/diverse links that (eventually) split off to go North/South, if you physically ding them between the DC and the split, you'll take out the lot...

    2. Alister

      Re: sort of reminds me

      Yes, it's a question I've pondered before. A datacentre that we use in Derby boasts of multiple redundant fibre links, but whilst that may be the case at their site, all the fibres go through the same telephone exchange in Colyear Street, and that building only has one ductway and cable room, so at that point all the fibres are in the same duct.

    3. WolfFan Silver badge

      Re: sort of reminds me

      Then they showed a slide of a (real)huge fiber cut that had probably a dozen or more carriers in the same pipe, I want to say the pipe or conduit was 1 to 2 feet wide buried in the ground.

      Was that of the Really Big Cut(tm) near the Chesapeake Bay Bridge-Tunnel? The one caused by a gentleman with a very big backhoe and a very small brain? If it was, then the conduit was 18" in diameter and was made of reinforced concrete. It was Class IV or V, I think. See http://www.ncp-inc.com/price1.html for how much the stuff costs per foot. There were lots and lots of signs saying don't dig here, but he knew better.

    4. Anonymous Coward
      Anonymous Coward

      Re: sort of reminds me

      @Nate Amsden; Obligatory XKCD.

      But if you don't like that you might find this one more relevant here.

  3. LadyK

    Seriously don't envy the guys having to repair those fibre cables whilst having CEO of whichever shit company screaming down the phone to hurry up.

    From memory a 10 metre length of 360pr fibre used to take us around 12 hours to replace after the locals mistook it for copper but this was around 25 years ago and we did have Intercity125's whistling past every now and then due to the reluctance of British Rail to stop trains allowing us to work without interruption.

    1. werdsmith Silver badge

      Nobody is going to let any screaming CEO talk directly to the splice boys.

    2. Doctor Syntax Silver badge

      "Seriously don't envy the guys having to repair those fibre cables whilst having CEO of whichever shit company screaming down the phone to hurry up."

      Step one of procedure. Take out all manglement phones if they're not already out.

  4. Alister
    Thumb Up

    *Teetering Infrastructure Turns Services Utterly Pants

    Thumbs Up for that one

  5. Alex 72

    customer 1st

    Maybe throw gamma or bt some money to lease another line i get that some of the traffic cant go on the public internet unencrypted but encrypt it and pay for the overhead I doubt there isn't at least provider who could lease them enough bandwidth to get tier 1 services back up. It would be expensive on this level of notice but so is publicity like this

    1. Anonymous Coward
      Anonymous Coward

      Re: customer 1st

      Sure, just throw some money and you'll be back up and running in no time.

      Just ignore all the minor issues; fibre runs, internal cabling, routing changes, firewalling, load balancers....just pay some cash and it magically all goes away instantly.

      1. Ben Tasker

        Re: customer 1st

        They just need to plug in a BTHomeHub, obviously....

  6. Anonymous Coward
    Anonymous Coward

    In before someone else.

    " Can you give us the number for your compensation line, we're losing absolutely thousands with this downtime

    — Unique Fundraising (@UniqueFundraisi) March 29, 2018"

    BAU / DR plan?

  7. tin 2

    They have only 2 redundant links?

    For a company (apparently) that size?!

  8. m0rt

    "have no fear - the Web Wizards are on the case! "

    Devops gone too far in this case, methinks.

    1. John Brown (no body) Silver badge

      It appears Virgin Trains and Virgin Media might be more closely related than just licensing a brand name!

      1. TRT Silver badge

        They had to be sacrificed to the gods of the network in order to ensure the return of bountiful bandwidth.

  9. Aristotles slow and dimwitted horse

    Erm...

    Sorry, just had to do a double take at a few of those tweets... having to sit an 'A' level on an empty stomach due to an internet outage? How the f**k does that work then?

    If you have reached the age of 17 or 18 and you are unable to feed yourself by ANY means due to an internet outage, then you really do have more problems than an internet outage.

    1. Brewster's Angle Grinder Silver badge

      Re: Erm...

      Because you were bussed (or driven) into school from one of the surrounding villages and either don't have an ATM card, left it at home (because, like, it's a school, not a mall) or don't have the time between exams to do the forty minutes there-and-back through the torrential rain to the nearest ATM.

      Although, on second thoughts, "ANY means" includes mugging a third former or shoplifting. Okay, as you were.

      1. werdsmith Silver badge

        Re: Erm...

        It's Easter holidays. The schools are shut for 2 weeks.

        1. Doctor Syntax Silver badge

          Re: Erm...

          "It's Easter holidays. The schools are shut for 2 weeks."

          From Friday. At least, that's the situation with the grandkids' schools.

  10. Ken Moorhouse Silver badge

    Re: Two simulataneous breaks...

    Or the first break was ignored.

  11. anothersortofleave

    N + X hosting

    Where N + X = N = X = 0

    1. Anonymous Coward
      Anonymous Coward

      Re: N + X hosting

      Nice bit of redundancy there. If N + X = N then, by the usual laws of arithmetic, we have X = 0.

      So the dangerous no-redundancy form is N + X = N = 0.

      But then someone will say, we're OK, if X was zero they'd say so!

  12. Anonymous Coward
    Anonymous Coward

    The Cloud

    Somebody else's computer

    Somebody else's network

    Your problem.

  13. Anonymous Coward
    Anonymous Coward

    ARPANET Design Goal (circa very early 1970s)

    'A Brief History of the Internet' - "...ARPANET [the Great-Granddaddy of the Internet] ...capability to withstand losses of large portions of the underlying networks. ..."

    Hey kids. You're doing it wrong. Unless you believe that such outages are acceptable. But the evidence suggests that: No, such outages are not acceptable. Find somebody with long hair and a grey beard, and hire them to design your network.

  14. John Brown (no body) Silver badge
    Mushroom

    Cloud?

    Isn't the whole point of "cloud" that this doesn't happen? If it can happen, you're not "cloud", you're just a data centre or hosting provider.

    1. TRT Silver badge

      Re: Cloud?

      Every silver lining has one.

  15. Anonymous Coward
    Facepalm

    Iomart - the agnostic cloud hosting experts

    "Everything you need to know about iomart - the agnostic cloud hosting experts. Delivering cloud solutions to meet your exact strategic business objectives." ref

    "Mr. Angus MacSween has been the Chief Executive Officer at Iomart Group Plc of iomart hosting Limited since 2000" ref

    "Iomart to hike dividend by 90% after jump in profits" ref

    "Britain's Top 100 Entrepreneurs 2014: No. 51 - Angus MacSween, Iomart ref

  16. Justin Case

    I was a customer of Melbourne

    who sold out to Iomart.

    I remember looking at the new owners and thinking Oh No when I saw their EasySpace connection.

    EasySpace were the first ever domain name registrars and hosting service I used, before the turn of the century. I found their habit of charging you to release your domain name to another registrar quite apalling. I swore, once I'd coughed up the necessary Danegeld to extract all my domains from the sticky suffocating fingers of EasySpace, that I'd never have anything to do with them or any company associated with them, ever.

    Little was I to know, happily ensconsed with Mebourne, who provided a reliable, reasonably priced service, that I'd end up in their clutches once more.

    Looks like I'll be moving hosts again - looking for someone just like Melbourne. Someone who concentrates on getting the boring but essential basics right.

  17. Corwin_X

    I stand in awe of you guy's ability to create TITSUP backronyms!

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like