Is everything OK over there, Britain? Have you tried turning the UK off and on again? ISPs, financial orgs fall over in Freaky Friday of outages

Today was more Friday the 13th than Friday the 31st in the UK, it seems. Not only is it Brexit Day, marking Blighty's withdrawal from Europe, but a bunch of services and internet connectivity broke. Here's a quick summary of what went wrong, according to Register readers writing in. If we've missed anything out, please let us …

[Anonymous Coward]

Your reader who has 500 users but no independent internet backup should get their management to invest in some decent resilience. We have 3 independent fibre providers and routes out of our building, two independent phone systems with two independent providers oh and an actual DR capability. No reason not too with comms being so cheap in 2020.

[IGotOut]

Until a major route from a separate carrier knackers, and you find all those independent routes are not as independent as you think.

I've had problems in different countries knock us offline.

[A Non e-mouse]

Back in the days of kilostream & megastream circuits, I discovered most north/south lines went through just one exchange in Birmingham.

[Chloe Cresswell]

That'd be Anchor?

[Anonymous Coward]

That'd be...

see icon?

[david flacks]

It wasn't just north/south lines. I remember the incident.

Woking-London routed via Brum as well so we lost the kilostreams to every one of our customers in the city. The joy of non-diverse routing for anywhere except the City Of London in those days.

1st time we actually used our dialback modems in anger and found there was a bug in the code that meant they negotiated 4800 baud connections rather than 9600.

[big_D]

Back in the days of kilostream and megastream, I discovered that BT couldn't lay a new megastream cable without digging through the old kilostream cable it was replacing!

I think the record was 10 outages in 1 day!

[A Non e-mouse]

A friend took over the running of a small shop. Their till software & credit card machines all rely on the cloud. I did say that they need to think of what happens when (not if!) their net connection goes down. I did try to explain that the cost of a 2nd net connection would be dwarfed by the loss of income if things went down.

Then things went down.

And they paniced.

And still won't invest in a second 'net conection.

[big_D]

Because lightning never strikes twice!

[paulf]

Back in my Saturday job days (late 1990s) the small retail establishment, where I was gainfully employed, didn't get a proper credit card machine until around 1996. Up until then we had to phone up a meat-bag to get a 6-digit number code to authorise each credit card transaction. Debit cards were still reasonably unusual with people paying by cheque+guarantee card.

We kept the clack-click mechanical card imprinter as a back up in case the phone line went down or the machine went wrong.

[Sherrie Ludwig]

Knuckle-buster to the rescue

Just last year I was in queue to purchase at a chain bookstore when their cc processing went down. They fiddled, tried for a bit but no hope. The manager on duty unearthed a knuckle-buster and credit card slips from a store room, but had no idea how to use it, like a young person today with a rotary dial phone. I showed them how to employ it, warned that it was only for cards with raised numbers, to check the expiration date visually, and get a signature and the phone number in case of problems. Then said that to process the charges, they would probably have to manually key in all charges off the slips, but check with head office first. Since my card was properly charged, I can only assume it all worked.

[Anonymous Coward]

> We have 3 independent fibre providers and routes out of our building

That may be possible in urban areas, but quite a few business are in more rural locations where suppliers and connections are literally thin on the ground.

[werdsmith]

My backup line is 4G. My main connection is via Virgin, I also have a deal with my neighbours on a different provider, they use my WiFi when they are down, and I can use theirs likewise.

[katrinab]

My local 4G mast is attached to the same street cabinet as my fttc and landline phone line.

[Dazed and Confused]

My local 4G mast is attached to the same street cabinet as my fttc and landline phone line.

My 4G backup goes from greased lightning to utter shite within minutes of the BT cabinet failing as everyone on the estate switches from their FTTC broadband to watching you tube on their phones.

[Anonymous Coward]

4G Masts do not connect via FTTC PCP cabinets, they are fed via a leased line directly back to the nearest exchange/POP. It might be located right next to your green FTTC cabinet because that's where the nearest BT ducting is, or where the nearest T-Node happened to be making life easier for all involved ;-)

(That, or the cabinet could have come AFTER the 4G mast now BT Openreach had to pull fibre for the mast).

[phuzz]

Of course, if they're running through the same ducting, then they're equally vulnerable to someone digging through them whilst trying to lay a new driveway (eg).

[big_D]

4G... I get around 300bps on my "4G" connection most of the time at work. If I'm very lucky, I might get Edge connectivity.

[eldakka]

Could always set up a microwave link between your site and one or two towns/exchanges over so that local outages due to local exchanges/cell towers going down could be bypassed.

But that aint cheap, you'd have to do a cost-benefit analysis on that.

[Anonymous Coward]

Re: more rural locations

15 or years ago I was involved in a project were some nut job had decided to have a datacentre inside a country house hotel (AKA vanity project). Getting a fibre in wasn't a problem. But when we asked for a second one coming on a different route BT sucked in a bit and said "Sure, that'll be £250,000" The management up stream lost interest at that point.

[Adam JC]

You can't really be *truly* carrier-resilient on VoIP, certainly not for inbound calls. The CLI has to terminate at a physical POP somewhere and if the carrier is Tier 1, like Gamma, and suffers an outage like that, you can't reroute calls - In-fact, you can't do jack shit except perhaps have a backup trunk with another provider, but even then your incoming calls are still fubar!

[Venerable and Fragrant Wind of Change]

The quote just says no SIP/VOIP service. It doesn't say there's no resilience - bearing in mind the range of perfectly valid fallbacks (for example, mobile phones, old-fashioned landlines) likely to be available.

[Irongut]

Netflix has been down for several hours.

[werdsmith]

That’s a blessing.

[John Brown (no body)]

Why? If you don't like it, you don't pay for it. On the other hand, people who are paying for probably have a different opinion to you.

[First Light]

Maybe she/he likes it TOO much.

[Anonymous Coward]

Excellent opportunity to "chill" without Netflix.

Unless you were using Netflix as the route to 'chilling', in which case I can understand some people getting a bit bothered about it (just not as hot and bothered as they were planning).

[Jou (Mxyzptlk)]

I am prepared.

I've got enough offline movies and series. Music to listen. Books to read. Games to play, including computer games with an offline installer.

And enough bed to sleep.

[Clunking Fist]

Re: I am prepared.

I have lawns to mow, house to paint, children to raise, exercise to take. Netflicks? That would be nice...

[Venerable and Fragrant Wind of Change]

Walking on eggshells

Um, do you really need to jump down the throat of someone who posts a harmless lighthearted quip?

[Doctor Syntax]

"We understand this was likely due to the end of the month demands such as payroll runs and the deadline for submitting tax returns."

There'll be another month end next month. And one the month after. Etc.

[Joe W]

And the transfer period for the German football(*} league also ended. It was likely all them damned krauts trying to pinch the British football players.

(*) soccer for those leftpondians

[A Non e-mouse]

This isn't the first time January 31st has happened.

[Anonymous Coward]

Yeah, but this one's speshal, its January 31rd, wen we get are cuntry bak!!!!1!!!

[Anonymous Coward]

@"Yeah, but this one's speshal, its January 31rd, wen we get are cuntry bak!!!!1!!!"

Those who are actually getting their country back are, in reality, a tiny fraction of the UK population.

The actual majority in the UK have actually lost their country given that they will have less control of what is done to them than before.

[Spanker]

(ex)Remainers are turning into those Japanese soldiers still found fighting WW2 in 1960 on some remote island. What a stupendous waste of your noisy time that was and you still don't plan on letting that tide in, eh?

[W.S.Gosset]

> given that they will have less control of what is done to them than before.

Actually, the EU expressly removed democracy, formally.

To be clear, your EU vote had essentially 0 power. Check out the EU's FAQs on the EU Parliament's/your MEP's legislative power.

[Pascal Monett]

Yeah. That's a pretty lousy excuse for not wanting to invest in some more processing power.

I know that, to scale your web functionality to the maximum of your customer base is a ridiculous notion, but you might want to make sure that you have baseline plus a comfortable margin at all times, and emergency supplemental capacity when the demand balloons.

I'm sure there are plenty of companies that are willing to help, it's just a question of writing a check . . Oh, right.

[Stubbly Dude]

There's this thing called auto-scaling that some large cloud companies offer apparently. It's terribly useful. Assuming that Tide is actually cloudy and not in someone's bloody garage...

[Anonymous Coward]

Stress testing and scalability.

We heard of them...

[the Jim bloke]

There'll be another month end next month. And one the month after. Etc.

but after Brexit, no one will have jobs...

[Andy The Hat]

30% increase in traffic causing it to fall over? That's either incredibly short-sighted network management (by the bean-counters) or complete cobblers ...

[redpawn]

Just call a plumber

"And again, the Internet is not something that you just dump something on. It's not a big truck. It's a series of tubes. And if you don't understand, those tubes can be filled and if they are filled, when you put your message in, it gets in line and it's going to be delayed by anyone that puts into that tube enormous amounts of material, enormous amounts of material."

[Anonymous Coward]

Re: Just call a plumber

The terrible thing is that the "tubes" analogy is kinda correct.

[cantankerous swineherd]

Re: Just call a plumber

Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway. - Tannenbaum

[Stuart Castle]

Re: Just call a plumber

You joke, but Amazon do apparently offer trucks as a method to transfer exabytes of data to AWS.. https://techcrunch.com/2016/11/30/amazon-will-truck-your-massive-piles-of-data-to-the-cloud-with-an-18-wheeler/

[Anonymous Coward]

Re: Just call a plumber

IIRC a lot of companies even offer to return the tapes/hdds.

[Doctor Syntax]

Re: Just call a sanitation engineer

Given the nature of much of that material.

[Stubbly Dude]

Re: Just call a plumber

So a bit like a gangbang where you are last in line because your bus was late...

[kschrock]

A historic day in every way.

God bless England, Her Majesty, and above all, the fine people of England, who finally gave the world what so many folks had been asking for, a second referendum on Brexit.

Well played Mr. Johnson.

[Anonymous Coward]

Re: A historic day in every way.

Yeah, but not by this generation.

[Intractable Potsherd]

Re: A historic day in every way.

'... finally gave the world what so many folks had been asking for, a second referendum on Brexit."

Ummm - there has only been *one* referendum on Brexit. If there had been a second one, fewer of us would be upset still (the difference between leave and stay was indistinguishable from noise - not a mandate to fuck the citizens of the country for the next two generations).

However, read another way, yes, so many folks around the world might have been asking for a the UK to leave the EU, especially EU citizens who were pissed off at the UK's dog-in-a-manger attitude to being a member ("we'll approve these rules, but we won't follow them...") English exceptionalism at its best - anyone thinking of making a trade deal with us, remember how fucking treacherous our decision-makers are.

[The Original Steve]

Re: A historic day in every way.

There's been two referendums on our membership of what we now call the EU. One a couple of years after we joined, and the other in 2016. Basically a generation between them.

And 1.4 million people in a population (not electorate!) of 65 million is only a rounding error if you didn't like the result.

Finally, whilst the the British often demanded exceptions and different treatment / opt-outs (Euro, Shengan etc) we're famous for sticking to the rules that were agreed. Unlike certain other countries I can think of... The nation of queuers will stick to all rules no matter now ridiculous they may seem. Not sure why us attempting to negotiate opt-outs and exceptions when the rules are made is a bad thing in your eyes.

[Intractable Potsherd]

Re: A historic day in every way.

No, there has only been one referendum on Brexit - 2016.

1.4 million in a population of 65 million *is* a rounding error, especially when about 3 million UK citizens living overseas were excluded from this very important decision. When is a UK citizen not a UK citizen? When it suits the government, of course.

Yeah, opt-outs like the Social Chapter which would have benefitted those people who voted to leave because they couldn't see the advantages of being members of the EU. The Euro was a bad idea, yes. Schengen not so much.

EU rules that were gold-plated by Westminster to be almost unworkable, *then* stuck to rigidly. When complaints were made, the response was "Well, it's the EU wot dun it, innit." Don't blame other countries for sticking to what was intended.

[Clunking Fist]

Re: A historic day in every way.

God bless all those natural labour voters who took one look at JC and said yeah nah. (And thanks to the good folk of Wales, too.)

[IGotOut]

I know it's a crazy idea...

...but could some of these be linked?

[Anonymous Coward]

Re: I know it's a crazy idea...

BT definitely had significant SIP issues on Friday - if Gamma use them as an upstream provider (ie full outage) or maybe just connectivity to BT is down because of BT. I don't use Gamma so don't know...

Anyone have any details of the BT issue? A major issue with BT connectivity could affect Nationwide if Nationwide use them as a supplier.

Speculation of course - we use BT SIP and while we had some issues, they were unrelated to the larger outage.

[Stubbly Dude]

Re: I know it's a crazy idea...

Well, they all happened on the same day, in the same country... beyond that I'm stumped..

[Anonymous Coward]

Turn the UK on and off again

Turning the UK on and off again won't work. It needs to be smacked upside the head.

Then again, the same can be said for my benighted (and knightless) Merica.

[Dr Paul Taylor]

Re: Turn the UK on and off again

Well, after four years of general fiddling with the switch, the UK has now been turned off.

It needs to be smacked upside the head.

It has probably needed that for a 100 years. Maybe it will get it.

As for turning it back on again, we'll have to see about that.

[Boris the Cockroach]

Re: Turn the UK on and off again

But dont worry, you've got a king again

After all... whatever he does , the senate will find him not guilty

[Anonymous Coward]

Re: Turn the UK on and off again

I think you'll find he's actually an deified Emperor - a Caesar in fact.

Hence the saying "Render unto Caesar" ;)

[Clunking Fist]

Re: Turn the UK on and off again

"the senate will find him not guilty" Sanction torture and illegally invade other countries on pretexts and turn them into shitholes (Bush II, Obama): okay. Make a phone call: beyond the pale.

Trump will be judged in the Court of Public Opinion in a few months time. So Democrats who have a proven track record of enriching themselves without delivering anything to the ordinary folk, are having a tanty cos their gravy train driver didn't get the nod.

First they said he was a Russian agent and that didn't work, now they are trying this. One day they'll pick a decent candidate and win an election! But they hate Bernie as he doesn't really play the gravy train game.

[Tom 7]

Re: Turn the UK on and off again

We turned it off by smashing the power switch!

[Stubbly Dude]

Re: Turn the UK on and off again

I think it's more of a 'turn it upside down and shake it' type of reboot that's needed now.

[Anonymous Coward]

Fibre

Wasn't there a major outage recently because the main AND backup fibre ran through the same physical junction box though separate everywhere else?

Said fibres getting cremated due to an underground substation fire.

[Anonymous Coward]

Re: Fibre

Are you thinking of this incident: BT denies cable fire was in [Manchester Guardian] A-bomb exchange?

[Anonymous Coward]

Welcome

to post-Brexit Little Britain.

[TRT]

Re: Welcome

Computer says.... No.

[el kabong]

Computer knows nothing

Whatever it says... it does not matter.

[Anonymous Coward]

Re: Computer knows nothing

Unless you're adversely affected by the computer's decision. The occurrences of which are only going to increase in future.

[Jou (Mxyzptlk)]

Re: Welcome

*cough* *cough*

[Anonymous Coward]

Re: Welcome

Yeah, good luck to you, guys.

The UK people will be missed but for sure the country's government won't !

[Anonymous Coward]

Gamma

My company has two diverse feeds to two separate Gamma SBCs. Both connections were affected yesterday. Not impressed. Once the dust has settled we'll be having a stern word with our account manager.

[Anonymous Coward]

Re: Gamma

I used to work for a small(ish) UK ISP in the early days, and despite having their own connectivity at their head office, nested in the corner of the room was a Plusnet ADSL circuit for emergency use.

I worked there for 9 months and we had to use that connection - more than once during several large outages... Go figure. (Anon, for very obvious reasons).

[macjules]

Online biz-friendly bank Tide.co

Worth noting that even though they call themselves a 'bank' they are not. They are simply a prepayment solution managed by Prepay Technologies Ltd, so not subject to FCA supervision. It is a Bulgarian operation with an offshore development centre in India but "headquartered" in London for the benefit of the VC investors..

[Strahd Ivarius]

Re: Online biz-friendly bank Tide.co

So the unexpected surge was due to Bulgarians buying .uk domains just before Brexit?

[Trollslayer]

Re: Online biz-friendly bank Tide.co

You beat me to it!

[Snake]

Re: Online biz-friendly bank Tide.co

In the States, "Tide" is a brand of laundry detergents.

I wonder if the Bulgarian's selection of "Tide" has any laundering associations... O.o

[John Brown (no body)]

Re: Online biz-friendly bank Tide.co

"In the States, "Tide" is a brand of laundry detergents."

Is was (or maybe still is) here in the UK too.

[Tom 7]

Re: Online biz-friendly bank Tide.co

In the UK "Tide" is description of the sewerage we are about to get.

[Richard Parkin]

War with Spain now :)

First day out and it’s WAR with Spain :)

Captain, art thou sleeping there below?

https://www.theguardian.com/politic...-talks-eu-to-back-spain-over-gibraltar-claims

[druck]

Re: War with Spain now :)

The Spanish will have to wait their turn, it's traditional to have a war with France first.

[Anonymous Coward]

Re: War with Spain now :)

The Scots are traditionally France's allies...

[baud]

Re: War with Spain now :)

Nah, we've just beaten England in rugby, we're not interested in getting in another fight with Blighty

[Anonymous Coward]

Down for maintenance while they install the post Brexit spying gear or they've started ripping out the Huawei gear.

[Anonymous Coward]

Not just the UK

Woke up yesterday to an email saying the major US IT supplier I was working with had lost comms to one of their biggest sites States side.

[Jou (Mxyzptlk)]

No problem here...

...I could access threregister from Germany without problems throughout last Friday. Looks like the vulture has a good connection.

[Ken Moorhouse]

Traditional Banks...

Publish their opening hours.

Looks like Tide need to start publishing moon phases together with your location so that customers can tell if it's in or out.

https://www.tide-forecast.com/

[Tom 7]

Re: Traditional Banks...

Strangely MagicSeaweed is used to check the tide round here.

[CountCadaver]

Ninewells Hospital in Dundee also

Ninewells hospital phone system crapped the bed also from lunchtime onwards also....message "our telecoms service provider is working on a resolution as a matter of urgency" for over 10 hours and counting...

[Anonymous Coward]

Re: Ninewells Hospital in Dundee also

Dundee and Angus College in lockdown due to ransomware attack...

[Anonymous Coward]

Re: Ninewells Hospital in Dundee also

given how crap their IT systems are...not much of a loss tbh, post merger they kept all the PHB from "industry" (read expensive failures with no idea how to run a college dept) and those with a clue left due to the idiotic BS emanating from their new overlords, such as shutting down the welding and fabrication vocational course (popular with people in their 20s and 30s upskilling) in favour of an dumbed down NC aimed at 16 year olds on apprenticeships and with much reduced practical welding / fabrication time (they'll do that at their work, we're here to make them literate and numerate to improve their academic skillbase etc etc etc etc ignoring entirely that the weld/fab course was popular with industry due to it spitting out coded welders at the end who went into decent paying welding jobs and that many local firms don't have the headcount or resources to train apprentices for 4 years and often would prefer someone in their 20s or 30s with a decent work ethic and practical knowedge - less risk for one and better chances of retention due to those beyond school leaving age having committments - partner, kids, mortgage etc etc), worse with talk of replacing those "dangerous" welders with some nice safe "virtual reality" welding simulators (of course costing a fortune, not replicating actual welding in anyway shape or form and likely netting the PHB a sizeable kick back)

[Anonymous Coward]

Vodafone flaky here in Nottingham

Last 24 hours the high speed broadband is down to dial-up speed.

[Version 1.0]

Highly Reliable (marketing) post exbrit profits

Internet connections, system backups, bank IT system are all highly reliable and trustworthy. This is documented by the highly paid marketing department and the corporate management teams who take a nice bonus every year because the corporate profits have been raised by cutting back the support systems, moving the system backups and IT support to the third world and cutting deals with their friends.

It used to be that when you set up a backup of any kind the prime question was always, "How many ways are there for this to fail?"

Nowadays it just, "There's a website that tells you how to do it."

[Jou (Mxyzptlk)]

Re: Highly Reliable (marketing) post exbrit profits

"exbrit" - that typo deserves a

[ecofeco]

Re: Highly Reliable (marketing) post exbrit profits

Too damn true.

[Anonymous Coward]

How long before we get?:

The Russians did it.

[Anonymous Coward]

So...

It's really a case of telco / big IT shops in the UK just being absolutely fucking incompetent....

[Celeste Reinard]

Bricks-it

Fes up, who of you lot broke the internet? Hmmmm? We know it was one of you... And we expect the real story at our desk at Mission Control, monday morning, chez BOFH, or at Who, me?

[redpawn]

Re: Bricks-it

Someone opened the email titled "Join the Crew"

[Dolvaran]

Your country has crashed

Reflect, repent, reboot

Peace will return.

(with apologies to the original)

[AshOnline]

Did BT have any issues?

Apart from downdetector who rely on user input, with users who dont have a clue.

I see no evidence that BT had an outage.

In fact, as a BT Wholesale reseller, i can say none of our Tetrabyte customer internet connections were affected, neither was our Tetrabyte hosted voip systems!

I think the key here is the fact that Gamma went down and many users checked out BT downdetector

[AbeSapian]

Is Brexit BrokesIt?

It's all in the title. Of course, being in Ameirka, I can't talk too much. We're still Trumped.