Backs onto a holiday site?
In Watford??? Surely some mistake
Cable and Wireless's Watford network site went offline at 1am this morning, after thieves apparently stole vital equipment. Several Reg readers emailed us to report the outage when websites hosted at the Watford centre suddenly went kaput. They were told robbery or vandalism caused the downtime. The outage has been blamed for …
It's a building approx 10miles away from the data centre that was where the break in happened. This is where all the bandwidth for the datacentre arrives.
Insecure single point of failure and an unacceptable one at that. Irrespective of that fact that this was theft. Diverse paths are required into a facility like this. Just like power.
C&W limps from crisis to crisis, recently their operation in Leeds was taken down by connectivity failure. The Air conditioning in Watford failed shutting down the hosting rooms and taking many customers off-line a few months back. And Watford suffers from Internet outages from time to time, only after risks have been taken with single points of failure are exposed.
They are a disgrace.
Why have I posted anonymously.... I'm a customer in this facility
My coat is the one with the servers , racks and comms equipment looking for a new home.
Was reading on a Canadian news site last week about a cable theft. Problem was the idiots tried to steal live electrical cables and became a bit frazzled in the process. Love the irony and no I don't feel bad for someone stealing causing thousand inconvenience because they are too lazy to get a real job.
On top of this, Our substation earthing cables got nicked a few weeks back, and we where without power from about 5PM to 2AM!!
'EDF representative, Tracey Sparling, said:"Unfortunately EDF Energy had to isolate supplies to several thousand customers in Watford at 6.22pm yesterday because vital earthing equipment was stolen from four 132,000 volt grid transformers."'
http://www.watfordobserver.co.uk/news/2314309.thieves_cut_power_to_thousands/
This post has been deleted by its author
A few weeks ago some thieves tried to steal a great big pipe from the cellar of the hospital i work at. As they sawed away they heard some hissing and gave up. The nice big pipe was the gas main feeding the entire hospital.
The gas leak was discovered by two people sneaking into the cellar for an illicit fag (banned on hospital property anyway). Fortunately they didn't try to light up!
Yes, these idiots will steal anything rather than get a proper job. We recently hired a skip and filled it with household crap. The next morning we woke up to find two blokes clambering about on the skip with about 5 kids hanging around filling up a supermarket trolley with any metal bits they could find. We thought "not doing any harm' and left them to it. Three hours later they were still there. We rang my brothers; you should have seen the chavs' faces when a car pulled up and three huge blokes climbed out! They couldn't get that trolley down the road quick enough. Priceless! Now you see "No Scrap Metal' or "Building Waste Only" signs on all the skips round here. Sorry, a bit off topic, but there you go.
Nobody is blaming cable & Clueless for getting robbed.
However they are blaming them for having ALL their connectivity running through a single point, thereby creating a single point of failure, which is a absolute cardinal sin which cannot be forgiven.
Hell when I worked for a (largeish) local newspaper we even had comms links from different suppliers running from opposite ends of the building, to preclude a careless JCB driver (or slightly less careless carbomber) from severing our connectivity
It's in the "Little Book of Datacentre Design" right at the very start, beside things you will require:
1: A secure building (usually involves security staff and cameras)
2: Diverse power supply
3: Diverse comms links
4: Trained staff
5: Comms Equipment
6: Servers
7: Customers
Paris, cos even she could do better
For all of you that says this single point of a failure should of not happened think about this. It was the cable that was stolen. In certain areas in my state only one company owns the physical copper cables. So it does not matter which company you use, it all goes over one companies copper. That company has a problem nothing you can do. No one else has copper.
If it wasn't a data centre how can anything be hosted there unless you meant that it hosts connectivity for servers powering several major websites?
The servers are obviously elsewhere unless the DC is right next door?
Correct me if I'm wrong but I dont remember comms gear ever being able to host a website apart from its own management interface perhaps?
No matter how many diverse connections power feeds you have don't they have to converge at some point to provide service to a single point?
To those that are smug about their diverse feeds they may leave your premises with 180 degrees of separation but I bet you at some point they run side by side even though your provider may not want to admit it ;)
I am not saying if C and W are to blame or not, just it is easy to point fingers and by the looks of the previous comments it looks like the band wagon is already overloaded with the righteous :p
sted Thursday 10th July 2008 22:12 GMT
Pirate
No matter how many diverse connections power feeds you have don't they have to converge at some point to provide service to a single point?
To those that are smug about their diverse feeds they may leave your premises with 180 degrees of separation but I bet you at some point they run side by side even though your provider may not want to admit it ;)
I am not saying if C and W are to blame or not, just it is easy to point fingers and by the looks of the previous comments it looks like the band wagon is already overloaded with the righteous :p
When I worked at MCI hosting center, they had lines coming in from sprint ATT as well as their on lines. They all came into the same side of the building. You better believe that at some point, out side the building those lines ran side by side. This was more than just an ordinary data center. This is were MCI networks had a physical hand off and ties in to ATT lines as Teir 1 provider . If you were to cut the lines there, you would cut the wires here you would screw up routing between ATT and MCI in the area. If you are missing my point here it is. If you cut the lines in the right place you will cause a single point of failure. All providers have to tie into some else at some point. Break that link and see what happens. Oops ATT cant acces any one on MCI. , oops ATT cant access level 3
I knew of an major companies datacomms center (one of about 12) had a single point of failure sitting in a greybox on a small patch of grass which was in the middle of a busy traffic intersection.
Story goes if was only found out after a manager had a little chat with one of 'pans people' over morning tea break (guess the company).
The fix, in only the way it can, ran a new fibre over 50 miles the other way sharing the costs with the exchanges on the way.
"....but I bet you at some point they run side by side even though your provider may not want to admit it ..."
Oh yes!. That reminds me of a cautionary tale..............
A large facility I used to work for had a requirement to carry critical data from one of its existing sites to a new site about 150 miles away. They cleverly specified 180 degree separation at the new site and also contracted two separate suppliers to make the links. To ensure that the suppliers did not use any common carriers, they had to submit a detailed list of all their subcontractors for checking and approval by the client- etc. etc.
Only when it was all installed and working did someone find out that both suppliers had independently decided to run their cable across a particular road bridge, in conduit, along the same side of the bridge. All it would have taken was a simple traffic accident involving a large van, to rip through the conduit and take out both data links.
Lessons: Involve your critical suppliers in your technical design process. Review and check everything yourself. Trust no one.
There was a night theft of lead from our local church in the village. The next day it became obvious from the white footprints, that the Holiday Caravaners who lived in a house backing onto the church were the culprits! Some more caravaners were caught out when a wood chipper was found in a wood backing onto their holiday park which had a tracking device fitted. That was not all that was found either!
The bestest (sic) DC's I've hosted in have fail-over satellites dish uplinks. Admittedly it costs a bit but there is really no real excuse for lost of connection when there are failover options, you get what you pay for. If i was responsible for large websites like Sainsburys or the FT you can bet your bottom dollar that I would ensure that it was hosted in a DC with satellite failover.
Mine's the one with the smug c*nt sticker on the back.
Diverse feeds / power etc:
"at some point they run side by side" - not if they're diverse. You have them in different ducts coming from different exchanges / transformer stations.
"one company owns the physical copper cables" - even so, you get two feeds from that company, in different ducts, coming from different opints on their network.
Lets get with the programme here....
Yes it might be nice to have very expensive satellite based redundant links, but would you be prepared to pay for it?
Unless you work for a service provider then you will probably be quite surprised at the number of customers who won't even pay for backups never mind a full business continuity plan.
I work for a big service provider and we have some very big, household names along with a lot of government work on our books and not all of them will even pay to have redundant links into their network,so they are never going to put their hands in their pockets to pay for an expensive solution like satellite links.
It's easy to post on here what you would do, but if it was coming out of your own pocket I would bet my house that you wouldn't pay for it, unless the cost could be mitigated by the amount of money you would loose should your website be unavailable. Even then you'd probably find something better to spend your money on.
For the one person who was wondering, yes we were effected by this. I had to laugh as just last month another one of our third parties has their cables stolen causing an outage.
Would agree entirely but even if a provider did not want to go to the expense of providing an uplink to Inmarsat or one of the other providers (inmarsat was one that came immediately to mind but I know there's countless others) there are now more cost effective terrestrial options such as HSDPA or SDSL which can be used (though of course it would be much slower) though Sat is undoubtedly the best option for connection security.
As regards power issues, the only real way to secure power for a facility is to have onsite generators (where I work we supposedly have 3 diverse feeds from various parts of the town along with 3 substations onsite but we still have trouble if there's an issue with the grid locally and have had to manually switch to generator power on occasion.
The biggest problem for the customers hit by this, was the useless account management and information customers got from C and W frontline.
We were told by our account management that Watford had been repaired, yet the helpdesk people told us that we were re-routed via a failover to another C and W office!
C and W have a good network, and good network engineers/people on the ground. Unfortunately account management and the admin and front office are f useless.
Our account manager asked us what he wanted us to do when we called.
Er. How about updating us every hour?
How about giving us an estimated fix time? We kept being told "don't know", all we wanted to know was: Matter of hours? Matter of days? Matter of weeks?
So we could respond accordingly and start configuring our own alternative routes for traffic.
C and W are let down by their management, not their network, or their engineers who are doing a fine job. I hope Sainsbury's maul them.
Common practice, it is known as a flattened ring (Cables in same duct) to at least try and fool the customers that it is a protected ring!
Increasingly common is the practice of getting wavelengths from 2 different providers only to find they share the same physical fibre. Happens ALL the time in EVERY telco so dont be suprised to hear of more "unforgivable" single points of failure
Cowboy and Witless supply the ISDN to the UKs Post Offices (that aren't yet or can't be on ADSL) meaning hundreds of sites were down... meaning I had to escape the PO's Helpdesk and go and eat doughnuts in the hub cupbaord for a couple of hours whilst phones exploded.
Bringing it back to a relevant angle, you have here second hand experience that C+W said what the problem was quite quickly: "naughty vandals" and "engineers ready to replace equipment as soon as the police piss off" was the theme, but they steadfastedly refused to give us any eta on a fix time. At all.
Not very good when they're supplying a vital service for a British Institution like the PO.
did the greater London area degenerate into a third world nation?
All these stories of stolen cables, scrap salvaging, dumpster diving, its like its still Dicken's London over there. You think the USA is so bad but maybe if you shot a few of these criminals you'd have less problems.
I live in Watford, I didn't realise I could holiday there! Wow what a carbon footprint saving I can make!!
I think what they actually mean is "traveller's site" but in a PC way! Now that the "travellers" have damaged their local internet access, how are they going to do their Ebay listings?
... don't you understand what they are talking about??
"Holiday site" is a very unusual term... and why would a "holiday site" be "raided by police only last week, looking for scrap metal theft evidence"?
Think why the word "holiday" might be relevant... think "caravans"...
Now think of "A site in Watford on which there are caravans, where it would not be a massive surprise to find evidence of theft of scrap metal, which attracted a large posse of Her Majesty's finest - with dogs and wearing personal protective equipment (PPE) in order so to do..."
Geddit?
God, PC-speak is destroying the meaning of plain English, simply in order to ensure that minority copper thieves... ermmm... I mean, people of a travelling persuasion... don't have their sensitive egos offended against... :-(
with copper and other metals at a high price there is bound to be instances like this (which of course doesn't make it right)
I blame the speculators in London and New York for pushing up the price of metal such as copper to a ridiculous price.
But pity the poor FT reader who cant get his fix, maybe he needs to find something more worthwhile to read that doesn't chargw a subscription (you are reading an excellent example of such a publication right now)
This post has been deleted by its author