Blundering workmen cut major Docklands fibre

Workmen in London's Docklands sliced through a major fibre optic trunk yesterday, forcing businesses and ISPs nationwide to reroute internet traffic. Contractors in Yeo Street broke the link, owned by tier one provider Global Crossing, early on Wednesday afternoon. The O2 and Be Broadband network was the highest profile …

[Alex Walsh]

Ouch

I had a tour of Telehouse in Docklands a few years back. The amount of stuff in there was mindboggling and I'm sure has increased dramatically since. It seems daft to have all this stuff in one place, even if there is theoretical redundancy with other sites.

[Dick Lovewell]

Blundering workmen?

Or blundering planners? The workmen might have been digging exactly where they were told as far as I can tell. Not that I'm a shovels & bum-cracks fan, but it seemed a tad presumptious to blame them just yet.

Paris, because when her hole's being dug she's happy to show everybody the evidence.

[dervheid]

That's the weakness with fibre optics...

No electromagnetic leakage to detect and betray it's presence.

Of course, if the fuckwits had done proper checks before they started...

[Anonymous Coward]

erm

I'm a Be customer and could download at 1meg/sec most of yesterday. Good enough for me :)

[Loki]

Isnt the point...

... that the internet is a distributed network where packets will route via whatever routes are available?

Doesn't this mean that either:

a) 100% of traffic for these providers can ONLY route via the link on the docklands (ie: no other routes physically exist... which surely would be strange for large providers to have no backup links)

or

b) something is really screwed up with IP in the UK (IP as in TCP/IP ... not intellectual property)

Paris - because dockland workers are a big fan of hers.

[Anonymous Coward]

Hurr

GX and O2 are based in Global Switch, not telehouse.

[Joe]

12 Hours to Fix.....

Some of our Branches up north were affected by this. Birmingham, Leeds and Manchester all went down at 1pm. Spoke to our provider (who is provided by GC) several times and was told they had to repair 200m of fibre cable!!! From the email I got this morning from our provider it took 12 hours and came up at 3.30am!!

[Anonymous Coward]

C4L

Surprised they knew their arse from their elbows. They are the biggest group of incompetent idiots I have ever had the mis-fortune of dealing with.

[Peter Flint-murray]

Ahhhh

So thats what happened, my normally 14Mbps connection was down to about 300kbps!

[Dave Edmondston]

Typical UK

Everything routed through the centre of London. Why?

Internet - distributed network.

Britain = London = Me, me, me

Everything in this country seems to depend on the arse end of the bottom corner of England.

[Simon]

Heheh I can hear the conversation now...

"Hey Steve, I gotta bloody tough tree root here"

"Then get the pick axe on it Jo"

*thwack*thunk*Bbzzzzttt*

"Oops...."

(yeah ok, it may not have made a short circuit sound but for comic strip effect....)

Mines the one with the detailed map of cable, pipes and anything else important in the pocket.

[Pete James]

I'm with Dick!

Got to agree with Dickie Lovey here. I had a teen student holiday job on a building site in a City where a JCB went stright through a whole truckload of telecomms and power cables which looked as though they had been just thrown in . The plans we had got from the council showed them to be about 30 yards down the road.

Every cloud has a silver lining though - we cut the power to the nearby Police station. All the woodentops were forced to come outside and do some work seeing as their kettles weren't working!

[Andrew Kelly]

Centre of London?

Do you know where Docklands is?

Having had first hand of these hairy arsed navvies I would put the blame firmly with them. Dig now, worry about the consequenxces later is their motto. And then try to a pull a fast one so you don't have to pay for the damage.

[Anonymous Coward]

@ Loki

HA Ha Ha, You dont work in Telecomms do you?

"a) 100% of traffic for these providers can ONLY route via the link on the docklands (ie: no other routes physically exist... which surely would be strange for large providers to have no backup links)"

Ha Ha Ha Ha, Now back to planning that new 60 Gig flattened "ring" (where the main and protection fibres arent just in the same duct but in the same cable!

AC, because

[Anonymous Coward]

wasn't me guv

Recently working opposite a hotel being built, in the 8 months it took to build the workmen went through our electric and phone cables at least 4 times.

Very first thing that happened when a workman was asked, was to fetch the foreman who's standard answer before even knowing the problem was "It can't have been us, we are not even digging in that area" ... which sounded a little empty after the 3rd or 4th time.

[Anonymous Coward]

@Loki

That's right the magic pixie dust helps those packets get to there destinations all on there own, you click a link to the register and all you packets just magically fnd there way there, just from your pc.

Can we have a bang head against desk icon...

PS read the article, it didn't go down, it slowed down.....

[Anonymous Coward]

Ahhh...

That explains why streaming HD video was an issue last night then!

[Echowitch]

@Dave Edmondston

Hmm I wonder why.....maybe everything goes through London as...

1. Its the Capital of the UK

2. Its the Business Heart of the UK

3. Its where all the Major Businesses are based

4. The South Coast has the UK's Silicon Valley (London to Newbury to Guildford and everything in between.

5. Its the capital and heart of everything as its closest to Europe

Yes Manchester, Edinburgh, Newcastle, etc, are all big cities but frankly none of them are as vital to the economy as London. The same argument can be said of any nations capital.

Paris......as she loves mens cables, and deep dark holes ;)

[Anonymous Coward]

Backup links

its makes me laugh when i hear these isp's are affected, my company use GX which was fibrenet and had our link knocked out, but like many of our other suppliers who also were on this fibre we just moved into a at risk state, running on the other part of our ring... you have to question how tight these isp's are being as the cost of cross london darkfibre is cheep as chips..

[Anonymous Coward]

Cable

This was a 288 core fibre however, so its not as though it could have been missed...

[James O'Brien]

@Dick Lovewell

Dont know what is funnier, your comment about the cracks or the Paris one. Good one.

[Steve]

Diggers, doing it for fun

I have two digger stories, one strangely in docklands. The g/f's sister is a geo technical engineer, and was working on a site not far away from Yeo Street, and told the driller (ok, not a digger, but close) right *there* cos it's safe, look at the council supplied plan, there's only London Clay there. Next thing *kzzzzzzzzrt!* and one driller thirty foot away. They'd found a very large electricity main that no-one, not even London 'leccy/EDF knew about.

The second story concerns a council in the Midlands, who had their server room in the basement of the building. Not only did the diggers manage to sever all the comms and electricity lines, they also when through a water main and flooded out the server room. Oooh how we laughed when we took that support call!

[Steve]

digging

No amount of maps and planning will help. It's always cheaper to dig wherever they need to and pay someone to clean up the mess than it is to dig carefully with a map.

[kain preacher]

@AC

his was a 288 core fibre however, so its not as though it could have been missed...

Um a trencher thats 2 inches (5 cm )wide 5 fee(1.5m) t long can snap that before they know its there .

[Maverick]

@ Pete James

you mean the the police _actually_ started work?

now c'mon we try to be realistic on here mate - there is NO way the useless t*ssers moved away from checking speed camera films!

[Sir Runcible Spoon]

why docklands?

It's all about power, that's why.

[Anonymous Coward]

Reminds me of a joke...

(That I think I saw here.)

How does a network engineer lost in the desert get rescued?

They bury 5 metres of CAT5 and hitch a ride home with the JCB driver.

Here all week, tip waitress, veal etc.

Mines the one with the 5 metres of CAT5

[Anonymous Coward]

hurr

Any telco/network provider worth their salt *does* have resiliency though. We certainly do, just diverted traffic around the other side of our ring and customers were reconnected within minutes.

[Anonymous Coward]

The maps are shite

Anyone who has any experience in these matters will know that all these cries of "stupid hairy arsed navvies should have looked at the plans" are just bollocks. No doubt, workmen do occasionally mess up, but you should see just how fucking useless these plans are - both the councils and the utilities very often don't have the slightest fucking clue what's where under the road, their plans are totally inaccurate and hence next to useless.

Just take a moment to think about how often your own wiring plans in server rooms and elsewhere are totally unrepresentative of reality - then bear in mind that these guys are dealing with all that wiring buried under layers of tarmac, concrete and muck.

Not that any of you are going to listen to this.

[James]

Strategicaly speaking ...

.. it seems fairly stupid to have major data systems based near a large river, liable to flooding, in the centre of a city that could be a prime target for terrorist attacks.

It would be better to have our main data systems somewhere deep underground in a geologically stable part of the UK. There are plenty of old mines that could be used, as well as all those "not so" secret Nuclear bunkers....

[david bates]

@Strategicaly speaking ...

We did,

Kingsway, Anchor and Guardian (London, Birmingham and Manchester exchanges respectively). As far as Im aware Anchor is no longer mainatained (just used for ducting) and is heavily flooded. Dunno about Kingsway and Guardian.

[John Dougald McCallum]

@James strategicaly speaking

Emmmm there are not very many of these not so secret nuclear bunkers probibly only seven or eight as they were meant as regional command centers IE a seat of local & military goverment also they are for the most part rather pockey wee places not the kind of environment that I'd want to work in.Also most of the large ones have already been sold of or demolished and as for London being the center of the ISP universe as has been said it is where most if not all the major Goverment and business HQ's are,and I aggree with the posters that said that utility maps are U.S.theyare actualy worse than useless.

[Matthew]

not found

> Global Crossing's press representatives couldn't be reached

I wonder why not? where their IP phone, email, etc. not working?

[Chris Collins]

over reliance on london docklands

I agree with the concerns on the over reliance of that location for out internet transit, something like 90% of isps are based there and pretty much all the mainstream uk websites, just about all uk internet transit is routed around that area. To some it may seem fine as london is the capital but if that location was to be bombed tommorow we would be up s*** creek connectivity wise.

If you look at germany as a comparison they have major POP's in all their major cities, frankfurt, berlin, hamburg etc. with datacentres scattered around the country. Planned datacentres in sheffield and peterbrough cant come soon enough but we also need some serious major transit laid down so we have other major POP's outside london and manchester.

How many tier 1's have fibre going through a POP outside london in the uk?

[Harry]

"No electromagnetic leakage to detect and betray it's presence."

Well, there's a thought. It probably wouldn't cost much (relatively) to include a bit of conductor in the cable or trunking just for that purpose.

It wouldn't even have to be permanently energised, just turned on when work was needed in the area.

Quick, lets get an anti-patent on the idea, otherwise some country with a backward patent office will fail to spot the obviousness of the idea and the idea will be priced out of the market.

Hmmm, does publication in El-Reg constitute an anti-patent? If the page is indexed by archive.org then surely the idea becomes public domain and ensures that such a patent could be disputed as "already in the public knowledge" before the patent was applied for ???

[Paul]

Its still SLOOOOOW

Has been for the past couple of days and doesnt seem to be getting much better, I usually get 14 megabits down, service is now intermittent at best.

[Gareth Jones]

@Dick Lovewell

Damn right. I've come across several instances where major cables and fibres have been cut by workmen because their presence simply wasn't documented. You can't blame the workmen for digging through a fibre if nobody told them it was there.

[Inachu]

The old Q diagnostic center would have been nicd to use on this

But too bad they were bought out and the lead programmer for the network tool still owes me $7.

Predicting this kind of network downtime is impossible.

[Anonymous Coward]

Plans? We don't need no plans!

Plans say "dig over there, that's where you are supposed to put the cable."

Workmen say "That ground's too rocky but there's clay this side of the street. We'll stick it in over here, nobody will ever know the difference."

Proof? Bundles of cables and pipes in the right place, just a few in the wrong place... not necessarily in this instance (was it?) but just look down practically any street in our sceptic isle.

Danger! Fsckwits at work!