London left buffering as Hyperoptic backup link refuses to boot

UK broadband provider Hyperoptic learned the importance of testing backup systems this week after the service went dark for customers in London. The service fell over on November 11, and despite users reporting that Hyperoptic's status page claimed there were no issues, there were indeed. man and scottish flag photo via …

[Anonymous Coward]

Ofcom's voluntary scheme

Funny how you can't volunteer to pay when it suits you.

[Anonymous Coward]

Re: Ofcom's voluntary scheme

I had a load of trouble with them. They agreed a date for install, which they missed, by several weeks. They pulled this exact "not eligible for compensation" shit.

The day that the broadband was due to go live I received a 5G router in the post. Completely unrequested and unneeded as I have an unlimited 5g data contract on my phone. It was instantly packaged back up and returned to them.

Months later they invoiced me "£170 - 5G router" insisting they'd never received it. Of course, by this time I'd (initially) lost/misplaced the proof of postage.

Queue a ton of legal emails back and forth (to state the obvious - you can't send someone an item unrequested and then bill them for it, this was a common scam in ye olde days, so there's specific legislation barring the practice). To cut a long story short, I eventually found the postage receipt to them and they relented (but not without significant and lengthy communications back and forth prior to that).

They effectively accused me of being a thief. Threatened to take me to court. Wasted my time over weeks emailing back and forth AND widely missed the original install date - absolutely zero offered in compensation. Not even a "here's 3 months free to say sorry". Just blanked.

Then, when I wrote a negative TrustPilot review .. they deliberately doxxed my TrustPilot account by using my full name in their reply (which is a unique name), which landed them with a GDRP complaint.

This level of customer service is particularly grating because the broadband itself is actually fairly reliable (this story notwithstanding).

[FrogsAndChips]

Re: Ofcom's voluntary scheme

Sorry abou that. I've been with them for years and never had to complain about the service, but this attitude is inexcusable.

[Like a badger]

Re: Ofcom's voluntary scheme

"They effectively accused me of being a thief. Threatened to take me to court. Wasted my time over weeks emailing back and forth AND widely missed the original install date - absolutely zero offered in compensation. Not even a "here's 3 months free to say sorry". Just blanked."

Have you escalated to the Communications Ombudsman service?

[Anonymous Coward]

Re: Ofcom's voluntary scheme

Not a good look, and will drive customer behaviour.

Did Hyperoptic bag any comp from Openreach as must have been a carrier grade fibre?

… and whose fault was the backup fuck-up?

[xyz]

Feeling smug...

just had 1Gb symmetrical fibre installed in my village house in Spain.

19.99€ a month.

[neilg]

Re: Feeling smug...

Are you sure?

Maybe it's a G-PON. Shared with world+dog.

[abend0c4]

Re: Feeling smug...

I've just withdrawn a post after failing to fact check it!

The OP could well be correct. XGS-PON is significantly more widespread in Spain and a lot of operators seems to offer symmetrical speeds up to 1Gbps - though higher speeds may be asymmetrical.

G-PON seems to be the default case for the mass market in the UK. As far as I'm aware, the majority of the Openreach FTTP network is G-PON with the gradual introduction of XGS-PON. Again, as far as I'm aware, Hyperoptic is similar. Corrections welcome.

My Portuguese connection is also G-PON and marketed as "1Gbps", but it's nominally 1Gbps down and 500Mbps up. The actual speeds are actually pretty close - around 920Mbps down and 520Mbps up.

Which isn't bad for an even smugger 15 euros a month.

[Doctor Syntax]

Maybe the backup fibre was in the same trench.

[Andy The Hat]

Maybe it was a couple of spare pairs in the same cable run ...

[Jellied Eel]

Maybe the backup fibre was in the same trench.

I'm guessing an IGP, so BGP probably won't get the blame this time. Or the usual problem when clients have totally resilient networks that automatically fail over in a few ms, with a seamless tranistion.. Which leads to that awkward question- "When did you last test it?" Which all too often lead to an answer of "Just after we installed & commissioned it".

[Anonymous Coward]

People don't want to pay and this is what you get. The end user buys the cheapest but often they have no way of knowing how resilent the service is, the vendor winning on cheap can't afford to build full resilence and the shareholders only understand the return today and never learn. It's a whole market & supply chain built on tiny knowledge between links & no view beyond a year.

[Jellied Eel]

People don't want to pay and this is what you get

Yup, but Hyperoptic is a service provider and wholesaler, so this is kind of core business..

"While our network is designed with built-in redundancy, we are also reviewing a secondary link that did not activate as expected, which should have limited customer impact."

They do network construction and given aggregation points are potentially expensive SPOFs, should have diverse connectivity back towards the core. No idea what kit they're using at the switch or optical level but using simple tricks like-

https://en.wikipedia.org/wiki/Link_aggregation

I've seen some vendors say LAG shouldn't be used on WAN connections because it's sensitive to latency variations across links, but around London, the difference between links should be within tolerances. Most decent switches can also run LAG or channel groups cross-chassis so the tin doesn't become a SPOF. For me, biggest advantage is no dependency on EGP or IGP routing. I have seen ports getting 'stuck' if they've not been used for a while, but think that was an old Cisco feature (ok, bug) and solvable by just having an ops procedure to bounce the backup ports on a schedule. But most of the tin I've seen for Ethernet aggregation has been pretty good for providing basic link monitoring.

[l8gravely]

Why the hell aren't they using BOTH links at once? That's the perfect test of your infrastructure, just push different clients/IP blocks down different paths, with failover when needed if one link goes down?

[Furious Reg reader John]

Yes, constantly using your "backup" links lets you know when they aren't working, rather than discovering they aren't working the second you fail over on to them.

[Yet Another Anonymous coward]

>Why the hell aren't they using BOTH links at once?

They didn't want to wear it out

[uccsoundman]

> Why the hell aren't they using BOTH links at once? That's the perfect test of your infrastructure, just push different clients/IP blocks down different paths, with failover when needed if one link goes down?

I can tell you that and I'm not even that smart. Lines are RENTED. If you have hot-hot redundancy when one line could easily support the traffic, then you are paying 2X what you should. Better to have the 2nd line in place but shut down. That costs less. You don't test it because you'd have to spin up that line and that costs money. Having a service go down may cost you some money in the future, but redundancy costs money NOW, and those who do not aggressively cut costs will themselves be made redundant. And really, how did this damage the company? They didn't lose a dollar of revenue. There was the costs of the consultants, but that comes out of a different account and doesn't show up on the quarterly earnings report.

[Anonymous Coward]

It didn't seem clear in the article where the cable was or maybe I didn't read carefully.

[Anonymous Coward]

Testing; a much underrated activity!

The other question is; did it take 24 hours to identify it was a cable? I have seen similar but not when in production and it represents core business. Was it a cable in the ground that had to be dug up or did they not have spares? I don't know what happens these days but when I was working in client on-prem systems we used to include spares in the price, unless the client asked not. More uusually they would ask to reduce the amount because the vendors would tell them not a problem with a 4 hour callout ... mmmm. Estimating the fix times & impact when making this decisions was another neglected area.

[greenwood-IT]

Backup circuit?

Sorry, am I missing something obvious here?

I install small business routers with primary fibre, and either DSL or mobile as a secondary backup circuit. The backup is always connected, but routing prioritises the primary, faster circuit. The backup circuit is tested every minute with a keep alive PING.

Surely the internet backbone doesn't require a dial-up circuit as a backup? If you've got fibre as a backup to a fibre, would you not have them both active and just load balance traffic? You then know that both circuits are working, and any failure will just reduce capacity rather than cause a total outage?

Hmmm, bloody experts :-)

[Anonymous Anti-ANC South African Coward]

It's not just one cowboy, they're all cowboys winging it.