Fasthosts goes titsup after storm-induced power outage

Gloucester-based Fasthosts appears to have been a victim of the storms battering the UK over the weekend, after it was struck dumb by a power failure that knocked its service offline for much of Saturday (4 January). The company, which was still busily restoring its servers into Sunday morning, told customers that it had …

[Anonymous Coward]

A bit amateur-ish and not entirely honest, it seems

"although power was quickly fully restored to both datacentres we experienced a network issue that occurred as a result of this interruption to service"

...and...

"UPS' are designed to handle a temporary loss of power. As the supply wasn't switched back to mains, eventually they drained"

...sound like very different explanations.

Also: "the firm's communication systems - including its status page - were unavailable"

Status page in-house? That's going to be very helpful if things go belly-up. Not.

[druck]

Re: A bit amateur-ish and not entirely honest, it seems

This is cock-up just the latest instalment in a long long list of failures. Look for the many previous stories about this blight of Gloucestershire's IT reputation.

[Anonymous Coward]

Where were the generators?

"UPS' are designed to handle a temporary loss of power. As the supply wasn't switched back to mains, eventually they drained"

... which also ignores their diesel generators which supposedly keep things going without mains power for 30 days. Why didn't they help?

Naturally I have little faith in either getting complete answers from them, nor in their ability to prevent this sort of thing reoccurring. But luckily we live in a market economy, and can take our business elsewhere.

[druck]

Re: Where were the generators?

You need those generators to last 30 days if you are located in Gloucester.

North Gloucestershire came within 1 inch of loosing its main substation following the 2007 floods. The substation was unmanned and wasn't checked until 48 hours after the rainfall, at which point they found the control room starting to flood. The only reason it was saved is the flood barriers meant for Upton-upon(in)-Severn where stored elsewhere and couldn't be transported to the village in time with all the roads blocked, so were available to be redeployed to the substation. The army had to be called in to top it with sandbags before the spring tide arrived (the river Severn is tidal to south of Tewkesbury), and just kept it from going under.

As it was Gloucester was only without power for 3 or 4 days when it was shut down as a precaution, but it could have been weeks if not months if it had flooded when operational. But just as that crisis was averted, the water treatment plant at Tewkesbury flooded, and we were without mains water for 6 weeks - which wasn't pleasant.

[DaveyG]

Re: Where were the generators?

Even of the substation exploded they could have brought in a portable one and had it back working within a few hours. I doubt it would take weeks.

[druck]

Re: Where were the generators?

We are talking a substation covering several acres, I don't think they come in portable versions.

[Jamie Jones]

Re: A bit amateur-ish and not entirely honest, it seems

" Status page in-house? That's going to be very helpful if things go belly-up. Not."

Indeed. And their DNS servers, according to Coward, above. Even my 2-bit setup has DNS in geographically (and 'networkally' different areas)

Dear Fasthosts,

I understand you have infrastructure issues. To help, I offer to host your network status page.

HTH

HAND

[Anonymous Coward]

Could have been worse

Could have lost all the data.

[JimmyPage]

A timely reminder to all

to check EVERY SINGLE element of their business infrastructure.

A few years ago, the clowns that hosted our web site and database had a power failure. When I asked why this had taken our website offline for 2 hours, they replied it was because the primary and backup servers were on the same circuit. Which they thought (in writing !!!!) was a perfectly acceptable setup, and they had never had any complaints before.

[Mike Pellatt]

Re: A timely reminder to all

The most recent timely reminder (before this one) was that O2 wholesale outage in Northamptonshire (and surrounding areas) when it turned out that Level 3 had a PoP in a light industrial unit, when an adjacent unit caught fire. So much for VESDA and automated fire extinguishing systems...

[Anonymous Coward]

Re: A timely reminder to all

VESDA systems detect fire. Gas suppression systems aren't designed to put out raging fires that come in from another building - because they'll presumably be bringing in a source of oxygen from the building they started in! They are designed to stop fires from getting going and to put them out in contained environments - once it is no longer a contained environment all bets are off.

[Alan Sharkey]

I'm a fasthosts customer and was affected by this. The irritating thing for me was that I was able to get through to their status page which assured me that all systems were working normally. It was only when I rang them up that I found out they had issues.

Alan

[Anonymous Coward]

Probably their status page is static html, updated manually when they feel like :)

[Anonymous Coward]

Sounds like an ATS problem

Probably confused by the return of mains before the generators had come fully online and didn't switch from UPS to either supply. I've been caught by ATS issues before, no doubt nobody qualified (or knowledgeable) on site to manually switch the ATS before the batteries gave out.

[Tony S]

Better get used to this

The current UK power generation system and distribution network are starting to creak at the edges due to many years of under investment. (However, it would appear that the UK is not alone in this.)

Many of the hosting companies are trying to cut their costs, so that they can offer the cheapest service. When that happens, something has to give and first thing is usually quality of service. I've seen a couple of these hosted places and I'm not convinced that they are serious about their business continuity and disaster recovery planning. They may talk a good game; but what you see and experience tells a different story.

I've recently had to deal with a number of organisations that have outsourced various parts of their IT provision; their senior managers seem to think that this removes the chance of loss of service and worse, they then don't check to make sure that the provider is doing their job.

As a result of this, we are probably going to see a lot more outages, reduced service levels, loss of business and almost certainly, companies going out of business.

Need an Eeyore icon!

[Anonymous IV]

Has anyone checked whether there was a power outage in Gloucester?

Or wherever their data centre is housed? All we saw in the local paper was a story about an outage in Podsmead, a minor suburb - and they're hardly likely to site them there!

(On the other hand, we are talking about FastHosts...)

[Steve Foster]

Re: Has anyone checked whether there was a power outage in Gloucester?

"(On the other hand, we are talking about FastHosts...)"

Ah, yes, for "datacentre", read "cupboard under the stairs in minion's house".

[Flywheel]

Re: Has anyone checked whether there was a power outage in Gloucester?

Podsmead .. They're probably running it all from Eric at Number 32's shed..

[A Bee]

Gloucester

The data centre is probably in Gloucester for the convenience of our friends in Cheltenham.

[Anonymous Coward]

dns was down

All 3 of their dns servers were down for a number of hours.

A lot of customer's kit/sites where up but with no dns they were no use. Similarly those who at some time registered a domain with Fasthosts and are still using their dns were affected.

Their dns strap line seems a little flawed just like their tecnology is:-

"Fasthosts provides a primary nameserver plus two extra backup nameservers, ensuring that

your domains won't be susceptible to failure."

Funny how a similar event (reported on the reg) happened in 2007 (this sort of flooding in that area is on a 7 year cycle from what was reported by one of the locals affected, strange his pub was still open for business!)

[1Rafayal]

if its going to happen...

.... It will happen to FastHosts...

[stu 4]

yeh...bloody fasthosts

I host my business on there too - I know I'm mental.

I spent the weekend wondering WTF was happening - I'd edit my website -adding new images, CSS pages, etc and it would look great for about an hour.. then I'd go back and check it and all the images had disappeared... checking the source - they no longer existed on the server.

I thought I'd done something wrong - spend another hour updating everything - same thing.

After the 3rd time this happened I gave up and decided to wait till monday to call support.

At no point did any part of the control panel tell me 'hey - all our systems are f&cked - best not waste your time making updates just now'.

Oh ... and don't get me started on the other quality decision I made - to use their built in website designer... I was taken in by the simplicity. I managed to put the whole site together in a few hours.

www.rocksolid-tech.com

great.

Until I realised cutting and pasting txt doesn't pick up the right fonts, there is no spell checking, etc.. so the only way of checking it all is now to copy and paste everything into an editor, and then when you find something to change - manually working through the changes on the website editor.

Also - no undo - so screw something up and yer f$cked.

No export html... and they html it creates is of course, utter crap.

sigh. u get what you pay for I suppose.

If anyone can recommend a friendly online editor plugin supported by fasthosts (im on linux just now) with themes, etc so I can use it quickly, then please do - I'd much rather be using open source stuff on my server than their in house mince.

[Jay 2]

When I didn't get any mail on Saturday morning I figured something was up. The support/config website took a while to load and was a bit useless (and it said all was well).

I figured that Twitter would probably know what was going on, and it did.

I have to agree with other commentards, in this day an age a hosting company having such an outage is a pretty poor show.

[redwood]

Action Group due to Fasthosts Non-Contingency Plan??

I would imagine that a company of this size should have to have a responsible and assured contingency plan should their ever be a power out. I am personally not happy with the replies I have been given from Fasthosts, and feel that responsibility is being shrugged off. They have told me that my hard drive is corrupt and I can now not mount this drive to the server, in order to recover the data. This they are blaming on me, rather than their recent issues. It would be interesting to know what other peoples experiences are in light of this recent issue......

If I can ask everyone here, if it has an effect on their business and revenue, and if so, would anyone be interested in setting up an action group with the potential of legal action to be brought against them. If you would have interest in this, and have been affected in any way, then please do contact me at your earliest convenience.

[MarcusAH]

Re: Action Group due to Fasthosts Non-Contingency Plan??

I have also had the HDD issue, Fasthosts have been trying to assist but I do feel that this issue is not all what it seems. I have been with Fasthosts for 14yrs and never had issues like this previously so this is not the ranting of some newb punter with no knowledge of the products supplied.

Our story is as follows.....

After the "Power Outage" our server did not come back online and when we contacted Fasthosts again (as we had been calling there premium rate number all weekend), they told us that our hard drive had gone. Confused and bemused I asked "where has it gone?". I was informed that it hadn't left and that it was actually still in the server but as sometimes happens it had "Packed Up", "Give up the Ghost", "was kaput", "Busted".

Once I had retaken control of my emotions, I explained to Fasthosts that the data and websites on the drive were extremely important and that I would need access to the drive, something they refused. To be honest Fasthosts have actually so far been quite helpful, although our facilities are still down and getting them back online is a mammoth task.

My thoughts are this... There was no power outage as if there had been the UPS and Gensets would have kicked in and we would not have even been aware of the problem. Whats more if this was the case and the power outage has damaged hardware then we all have a claim against the insurance cover supplied by the provider of the UPS and back up facilities.

Could it actually be the case that it was in fact a security breach and Fasthosts pulled the plug in which case we have a claim against Fasthosts.

Either way its sounds like someone needs to be footing the bill... Oh and to add insult to injury on the Sunday after the event I got an email reminder to pay my bill... :D