BT does Italian Job on London traffic lights BT is still working to restore access to thousands of people and businesses in east London left offline when a tunnel borer cut through fibre cables and copper wire. The problem is also preventing Transport for London from managing its traffic lights. Contractors working on the Olympic site in Stratford sent a large thrust …
... er it wasn't me mate.
(that is the answer we always got when a hotel was being built next door to our offices and our plusnet link was cut 7 times in a 14 month project).
... or alternatively maybe its down to the fact that in Poland they don't have fibre 32 metres below the surface, so using a boring machine (intentional mis-spelling) is just that... pay monkeys etc.
"So far we've managed to restore services to 50,000 homes and business. Work is continuing around the clock to restore service to the remaining affected customers."
That sounds like pretty bloody good progress to me. Cor blimey... Fixing a chopped fibre, doesn't sound like fun to me! I mean, which fibre connects to which? It's not like they're colour coded!
On you go BT.
You do have to wonder if all these professional building contractors and companies can actually read maps and the such to plan these things! Or do they just go, "This spot looks good...CRUNCH!"
I do hope that the bill for all the work and all the loss of business and problems etc goes to the Olympic Commitee and the contractors. Teach them to do things properly!
"On you go BT."
At least I hope you're being sarcastic!
It took the useless fuckers almost a month to repair the damaged multicore to the pole serving our house after some dozy smegger cut it with a digger.
As someone posted on the earlier article, BT need to get their act together on the diversity and redundancy factors.
With some added redundancy for those responsible for not ensuring there was enough of both in the system.
Not so much a network as a few bits of damp string knotted together incoherantly
Twonks.
Can these contractors even read a plan? or was the tunnel not marked on it?
Why did this happen? Why did this single incident cause so many people and businesses to lose their internet connection? where's the backup?
This whole incident just goes to show how BT have no viable backup infrastructure for when things go wrong. It's about time BT stopped looking at it's profits and invested in this countries quite frankly shocking infrastructure.
"That sounds like pretty bloody good progress to me. Cor blimey... Fixing a chopped fibre, doesn't sound like fun to me! I mean, which fibre connects to which? It's not like they're colour coded!"
Fixing a chopped fibre is fairly straightforward since all the fibres are colour coded thats why the fibre link is the least of the worries! in fact they have probably run some temporary fibre in around the unsafe area overground, fixed a joint with some terminating equipment at either end and there you go, service restored. I could have knocked that on out in half a day given BT's resources.
Now if they had (as some reports suggest) 50,000 subscribers on copper and they had managed to get those back on line I would be truly amazed!
The title seems to imply that it was BT's gaffe that caused the problem, but in fact it was an unnamed contractor who bored through their, hitherto working just tickety-boo, kit.
Now, I like to give BT a kicking as much as anyone, but wouldn't "Olympic contractors shaft BT up back passage" be more accurate? Or is my monday comprehension at an all time low.
All the best,
We lost an entire BT Central DSL platform as a result of this outage - impact to our customers? Oh that would be none as we actually bothered to build a resilient network, if you've got access problems you need to take a look at your ISP as much as BT for this little disaster...
> why no redundancy
... I suspect that they probably had redundancy but that in the post incident analysis they'll discover that all the redundant paths were in the same tunnel.
N.b. I recall something similar happening in California when I was there 10 years ago (construction work severing major fibre artery) and then it took PacBell 2 or 3 weeks to fix it completely
The contractors (Yhprum), working for a well known London water authority, had bored 48m sideways at the bottom of a 30' shaft. Far play they did own up when the BT vans turned up.
The boring machine went exactly 90 degrees through a deep level tunnel collapsing the tunnel and damaging 19 out of 80 cables that go through the tunnel. At present the boring machine cannot be moved for risk of damaging the other cables (which are still working).
True resilence is a problem from time to time in certain areas, that's why there are deep level tunnels, out the range of the usual JCB. Now that the cabling jointers can get at the cables they are being diverted now. trouble is, it's a big job.
"It's about time BT stopped looking at it's profits and invested in this countries quite frankly shocking infrastructure."
Err... how to make a company bankrupt... don't think about returns on your investments! Especially when the figures are in the billions...
Isn't it the job of the government to look after the country's infrastructure? And hasn't OFCOM made sure that BT cannot invest (up till now) and make a return on their fibre infrastructure investments?
If you want a decent infrastructure at cost, then the government has stump up the money. Otherwise it's market forces that do so, unless of course a regulator sticks their nose in and makes it impractical/unprofitable to do so. And anyone who mentions Virgin Media... they run a monopoly, no one has access to their network and every connection to their network makes maximal profits for them (not so for BT). Market forces work well in France cheap fibre for all, although for any given district there is only one supplier (mini-monopolies).
Not surprising that this country is in this situation it is in financially if people think in this kind of way...
"The contractors (Yhprum), working for a well known London water authority, had bored 48m sideways at the bottom of a 30' shaft. Far play they did own up when the BT vans turned up."
Not exactly an easy thing to deny then is it. It wasn't us guv. It was some other bloke with a tunnel borer, but he was last seen heading for Wapping...
And spelling the contractor's name backward's isn't going to fool a lot of people. Actually, given the mugs that fell for the Registers April 1st story, maybe they would...
It would have been much more spectacular if they'd hit one of the tunnels carrying the newly-buried HT cables:
http://www.london2012.com/news/media-releases/2008-12/last-pylon-removed-from-olympic-park-as-250m-powerlines-project-delivered-on.php
Still, there's time yet, I'll bet the new tunnels aren't on many maps :)
The problem is that most older tunnels, pipework and so on are not accurately mapped. In fact, a lot of them were last mapped by the Victorians, using survey reference points that no longer exist. And at 34 metres deep - or even34 feet deep - it is pretty much impossible to locate the tunnel or pipework by surveying. The only way of accurately locating such things is by doing what the contractor did - dig them up!
It's a nightmare in many (most? all?) cities.
Records are often not kept and when they are kept, they're inaccurate. Too many players, perhaps, plus human error both in the original record and in the aggregation that City Hall undertakes.
There's the famous story of a subway tunnel (or was it a utility trench?) being dug in New York. The contractors unearthed (but did not sever!) a pipe that appeared on no plan. No one had a clue what the pipe carried, nor whence, nor whither. Compressed air? Water? Oil? Natural gas?
As said pipe was In The Way, they finally went after it (very gingerly!) with a hacksaw.
The result was anticlimactic: there was nothing in the pipe except stale air.
Another example: a university in a small Canadian city wanted to run a wire to a remote location (RJE - anyone else remember that?) and leased underground copper from the phone company. The telco assured the network folks that said wire went directly from point A to point B.
Impossible throughput! A wire supposedly less than 2 miles long had the electrical characteristics of a wire tens of miles long and was simply unworkable with even the slow modems of the day. After much backing and forthing and finger-pointing, the telco investigated and discovered a fat inductor hanging off said wire, down a manhole. Left over from a previous use of the wire, perhaps for decades.
No record of it.
QED
"As ye dig, so shall ye unearth."
"I do hope that the bill for all the work and all the loss of business and problems etc goes to the Olympic Commitee and the contractors. Teach them to do things properly!"
So you'd be happy to pay if a contractor working for you damages a third party's property? Why should the Olympic Committee be penalized for the failings of a contractor? This is typical "government causes all problems" bullshit; just because the OC is paying someone to do a job doesn't mean they have operational control of their activities. No doubt you think that Sebastian Coe ordered them to do it?
Given the uncertain nature of records of underground services, then I wonder if the contractors generally use technologies which can detect underground services. There are several of these available, including Ground Penetrating Radar (GPR).
The range and accuracy of these can be affected by ground conditions, but a thumping great steel-clad service tunnel would surely stand out, although whether (or if) boring systems like this can be so equipped, I've no idea. I can't see a problem in principle, although I imagine it would increase costs and might slow things down.
BT do provide resilience if you pay for it.. These cables and fibres also carry broadcast, hospital and emergency service comms/data but they have backups so they all had other options.
BT is a business. Sadly what that means these days is that they are in it for the money like so many businesses unlike proper businesses which are in it for the love of the craft/product/whatever and making money is the mild irritation (instead of things being the other way around)
Doesn't have to be 'sub'
DSL failed at all the offices on our business park, no JCBs in sight, phones still worked etc. BT claimed nothing had changed at the exchange.
Turned out somebody installing a new phone line had decided to 'tidy-up' a junction box and manged to patch all the DSL lines to the exchange 1mile away, via a village 5miles in the other direction.
BT only checks the lines are intact and only do a DSL survey with a new installation - they have no system for dealing with anything that changes after a line is installed. It took a week of calls to different bits of BT to get us back online.
For gods sake. BT this BT that. It's NOT BT's job to provide a fully redundant network. If you want that then DIY. You really expect BT to put 2 lines to every sodding building in the UK? Its not hard to create a fully diverse infrastructure for your self. You start off by using 2 different companies with there own physical networks, and no that does not include 2 companies who back haul with BT.
And oh look, the internet is working for any one who is not only linked through that point. Guess it shows that it works as advertised.
Bunch of winging fucking wankers. Only Londoners can be so fucking blind to the rest of the world. your worse than Americans.
The idea of a central database to track who and when one of the 200 companies who are allowed to dig holes were going to dig so they could all synchronise their work? Would this also have (indirectly) built up a profile of what and where pipes & tunnels are located over time?
Of course setting up all those black boxes for the IMP is probably hitting the BT budget quite hard.
V.Poor
since the cut is half way through a link, you can't expect to see any 'colour coding' as you sugges. And to claim 1. you would route it overland and 2. 'you would have it done in half a day given BTs resources'? are you serious?
if a major cut occurred in our infrastructure, it would certainly be a disaster scenario only to be remedied by many hours of retubing, reblowing etc etc. of course we'd switch-reroute the less important stuff (say 50000 residential), but no monkeying around with critical infrastructure...
there are events which are financially viable to mitigate, and there are other rare-events which one should never ever pay to mitigate: ask 'what are the chances of someone cutting here... and would we have to pay the bill?' = <1e-9 = no = lose/lose for spending any money there.
Mr. Murphy's insurer will be receiving a very large bill, tant pis.
"It's about time BT stopped looking at it's profits and invested in this countries quite frankly shocking infrastructure."
Err... how to make a company bankrupt... don't think about returns on your investments! Especially when the figures are in the billions...
Isn't it the job of the government to look after the country's infrastructure? "
Ummm no, it's BTs job! They are a private company and as such they should provide the infrastructure that they advertise, NOT the tax payer. We have spent the whole bloody day arsing about because BT's idea of resiliency means shoving 2 cables through the same pipe!! BT should be paying HUGE fines for this screw up.
The reason they were not on the plans is that BT dont bother adding new stuff and upgrades to their program called Netword Records.... its updated as far as the mid 90's cos they closed down local drawing offices....
The number of time ive lost due to bad prints is soo frustrating.. so glad i left that company.... grr.
Totally possible...
Your broadband connection is only piggybacked on your phone line on the local loop, and jumps off it at your local exchange. From there the two signal paths are separate.
As you have broadband, but no analogue phone, either only the phone line goes via the broken tunnel and the IP data goes a different route, or the IP data is on one of the other links in the tunnel which survived...
So give it a day or two and I'm sure they'll manage to damage the IP datapath too :-)
"since the cut is half way through a link, you can't expect to see any 'colour coding' as you sugges. And to claim 1. you would route it overland and 2. 'you would have it done in half a day given BTs resources'? are you serious?"
You have NEVER seen a proper fibre "cable" have you, no really you havent. How the FU$K do you think they terminate main trunk fibres? Think they identify fibres one at a time or are you used to BT installations in your office where they might "blow" a fibre in from time to time? Think that when laying a transatlantic submarine cable they lay a tube and then blow fibres down it one at a time or just make a cable 3500 miles long? you have to "Joint" fibre optic cables every so often.
For your information, if you sliced a main trunk fibre (anything from 2 to XXX fibres) you would find the following colour codes:
Fiber/Tube No.
Color
1 Blue
2 Orange
3 Green
4 Brown
5 Gray
6 White
7 Red
8 Black
9 Yellow
10 Purple
11 Rose
12 Aqua
And then guess what, if there are more than 12 fibres they have these things called "tubes", they are also colour coded. so yes, identifying what fibre goes to what fibre is EASY. Thats why we have these things called "Standards" the above colour code allows for 144 fibres (12 fibres x 12 tubes), this is probably one of the larger Cables in use because with modern DWDM technology you can get many more channels per fibre so you dont need to sink as many cables into the ground (or under the sea)
As for doing it in half a day and routing fibre overland I am sure (like the huge international telco I work for) that BT would have some mobile disaster recovery vehicles which they could park say 500 M apart and run some temporary armoured fibre between so that they could put a joint at each end (because they would have access points into the tunnel at regular intervals), plug in and off you go. Yes I am being serious. The MAJOR headache would be the COPPER cables (also colour coded) but carrying less traffic per pair and needing THOUSANDS of pairs to be jointed.
"You really expect BT to put 2 lines to every sodding building in the UK?"
Er, you do know the standard BT consumer cable can support 3 separate lines as standard? And sometimes have to if one of the pairs get damaged.
Historically BT has multiple routes in its trunk network between exchanges. Partly to allow for growth (they did think telephone use would grow) and partly for resilience as at one time the telephone unions were apt to down tools regularly. Electromechanical re-routing. Who'd have thought it possible?
A tunnel 30m down sound like its a trunk line, as its a hell of a way to burrow to hook up some subscribers.
You need to understand the concept of resilience.
If you pay for a non resilient connection, this is what happens when the link is broken. The fault lies at the companies who have gone for the cheap option when implementing their comms infrastructure. I'm sick of listening to moronic comments from people who have zero understanding on telecoms networks.
Notice any banks in the list of affected companies, many of whom use BT? Thought not, because they paid for the extra connection.
"You do have to wonder if all these professional building contractors and companies can actually read maps"
If you've seen some of BT's plans you'd know why they dig the cables up.
We saw the ones for our building with a BT project leader. They were utter rubbish, saying main feeds were coming into the wrong building, feeding up on the wrong side of the road etc etc....
BT STILL managed to chop through a feed, despite us telling them they were digging in the wrong place....
It is a perfectly reasonable thing to do a cost/benefit analysis on the amount of money that you are prepared to spend on resilience. Putting in 100% fully resilient, redundant systems for everything costs money (in the case of IT systems, often more than double as all those resilience processes have to be tested, rehearsed, documented etc.). In the case of some systems, like aircraft flight control systems, the consequences of any single failure would be too large to bear. In others, then the costs of a very rare failure could well be much less than the amount of money required to protect against it.
Now this is not to say that the accountants always get this right and don't sometimes take shortcuts without due regard to the consequences, but the principle holds. Spending disproportionate amounts of money protecting against very rare, albeit expensive, incidents like this are often not cost justified. It's cheaper to take the hit, pay compensation (or, in this case, presumably the contractors insures will do that) rather than increase the total cost base.
There is also, as others have pointed out, a difference between a localised failure (albeit a fairly large area) and of a systemic failure that brings the entire network down.
The only separated network in London is Urban Wimax's. Shame it doesn't go as far as Ilford - it only covers from the city to hammersmith as yet.
BT doesn't actually know where all of its cables and ducts go. Before it was privatised record-keeping was shocking. I once asked how many Megastream circuits we had and they hadn't got a clue!
will be liable for all the costs of this outage.--
If the tunnel that got perforated was accurately shown on the maps/plans/records.
It may well be that the person doing the recording of the tunnel location made an error. It does happen.
Maybe it was a computer error. That's what gets the blame a lot of the time.
Multiple ISP's and multiple data connections don't always solve the problem of downtime. All the fixed wired providers out there usually end up in the same ducts or Exchanges anyway. Go wireless, have two truly diverse connections, one in the ground, one over the airwaves. Only way to get rid of local and common points of failure or at least not be impacted by them too much. At the end of the day most businesses have internet connections, why? To help them make money of course!
If the strategic decision makers don't realise the risks to their business they take, by putting all their eggs in one basket ,then they need to be educated on how to mitigate them. This type of article helps, a lot!
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
