back to article What knocked out Brit cloud slinger Memset for the night? A busted fibre cable upstream of its data centre, apparently

UK cloud hosting outfit Memset has blamed yesterday's lengthy outage on some iffy fibre infrastructure. The company acknowledged that all was not well at 13:45 BST (12:45 UTC) on 21 June with a terse note among the sea of green on its status page to the effect that it was aware of some connectivity issues at its data centre in …

  1. tip pc Silver badge
    Flame

    Many moons ago a Fire in Manchester

    Knocked out my comms to sites in Kent & Hampshire.

    not sure if it was this or a different incident, it was some time ago

    https://www.theregister.com/2004/03/29/bt_fire_disrupts_emergency_services/

    when i had my monthly review i insisted they provide an outage statement, apparently the national ring our services in the south used traversed the pit that was alight, other services in and around Manchester remained operational throughout.

    This was in the days of horrendously expensive 2Mb/s frame relay connections.

    1. Daedalus

      Re: Many moons ago a Fire in Manchester

      Auf Englisch, bitte?

  2. TimMaher Silver badge
    FAIL

    Clapham?

    How does an outage, even a JCB through a cable, have an impact on Dunsfold, in deeply rural Surrey?

    1. Martin Gregorie

      Re: Clapham?

      Don't ever think that comms lines follow anything that even faintly resembles the shortest route from A to B. They don't.

      Back in 1980, when I was on contract at the BBC, our development team was in an office in Cavendish Square and the main BBC computer Centre was about 800m west of Shepherds Bush. We lost all contact with our mainframe for a couple of days when some berk with a digger took out the cable - at Acton!

      For those who don't know London, Acton and Shepherds Bush are both west of Cavendish Square - but Acton is about twice as far away from Cavendish Square as Shepherds Bush and almost directly in line with it, so that cable not only appeared to double back on itself, but was at least three times as long as it needed to be.

  3. Anonymous Coward
    Anonymous Coward

    The clue is right there…

    If your number 1 sales pitch is based on being the cheapest, I’d avoid you like the plague!

    1. Claverhouse Silver badge

      Re: The clue is right there…

      Don't see how this applies to broken cables outside their remit.

      1. Old Shoes

        Re: The clue is right there…

        Getting geographically separate routes is difficult and time consuming as there are often layers and layers of companies renting from one another.

        I have two internet connections. They are theoretically distinct as each is bought from one of the three largest companies in the market. Company A's service is provided on a leased fibre from Company C who then muxes it and backhauls it on their larger fibre to the next large city. Company B theoretically has their own microwave links and separate Northbound and Southbound copper backhaul. FOUR independent ways to get my data to the Internet!

        Here's the surprise: in a cost saving measure Company B has thrown away their Northbound and Southbound copper and now leases a single fibre from Company C. Yes, the very same Company C that provides my primary connection.

        I know this because some joker put a spade through the fibre they both use. And no, I checked, and I'm not one of the half dozen almost identical stories also in this comment section.

  4. Mike 137 Silver badge

    Resilience?

    ' “We identified faults with two separate lines which impacted connectivity ..."'

    There was a case over a decade back in Manchester, where a commercial ISP with banking customers had contracted for a primary and a backup upstream connection frrom separate providers to provide resilience. Unfortunately, when vandals dropped a wheely bin of burning petrol down a street manhole, they discovered that their two independent upstream providers had leased dark fibres that ran through the same duct.

    The very nature of the pyramid of contractors renders this kind of incident likely as the right questions are hard to ask, let alone answer, and we have to remember that the internet becomes intrinsically less and less resilient as we approach its its periphery (where we all are as users). For all the hype about the wonders of "digital", the underlying infrastructure is frighteningly fragile considering the reliance we are increasingly placing on it.

    1. DS999 Silver badge

      Re: Resilience?

      So why didn't they get maps of the routing of the connections so they could check for overlaps? It seems pretty obvious that two companies might overlap use of the same ducts especially as you get close to the destination.

    2. Anonymous Coward
      Anonymous Coward

      Re: Resilience?

      I seem to remember that case. The other issue that happens is that you get good diverse routes and then supplier 1 buys supplier 2 and reroutes everything without telling you, leaving you with no resilience.

  5. This post has been deleted by its author

  6. Martin Summers

    There was a 2 day Virgin outage last week that knocked out our fibre leased line in Birmingham. Someone went through a fibre cable on a building site in Canary Wharf. Apparently the cable wasn't ducted and no-one would let them on the site until 10am the next morning. I'd love to have seen that reported on.

  7. PandyH

    Only a couple of years ago a fibre cut took out a data centre in St Asaph, somewhere between Manchester and the DC. It was a shock because there were supposed to be 2 redundant routes coming into opposite sides of the building and travelling separate paths arriving on the opposite sides of Manchester.

    The DC owner (also an iomart company) found out that the fibre cables were separately routed… until the last couple of miles or so where they traversed the same farmer’s field who had decided to do some ploughing that night…

    Even more shocking was they had found this out years before the incident, had asked for it to be re-routed, and believed it had been done.

    https://hostinguk.net/blog/seriously/

  8. sad_loser

    NHS standards

    To host NHS data real-time applications one has to have resilience with two data centres on different physical networks, but as a previous post points out as the nature of networks does not rule out a single point of failure even then.

  9. PeteS46

    This reminds me of an incident back in the 1990's IIRC. I had arranged our 3 main sites to have Megastream2 links in a triangle and BT sales assured me that the links were diverse. But a fire in a tunnel between Manchester and Salford was upstream of our links and they failed. We were OOS for about a week, again IIRC.

    Not just us, either! Lots of businesses including banks were offline. So no cash machines, card payments..... (cue public unrest).

    The situation was so serious they drafted in plods from other areas to reassure the public. I had to issue them with local guide maps (thanks to the Tourism centre) so they didn't come across as gormless when interacting with the tourists and other plebs.

    Lesson learned: If your provider says the links they serve are diverse:

    * Don't believe a word they say!

    * Insist on them providing guarantees with significant penalties for failure!

    * BT/Openreach sales lie!

    aka Bernard

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like