Fasthosts admits email destruction fiasco Fasthosts, "the UK's number 1 web host", has admitted to a botched update to its mail server that permanently deleted customer emails. The cockup last week was compounded when the firm's backup system failed, meaning about half of the affected emails are lost forever. Fasthost sent us its side of the story: Following a …
I guess they just make a mirror, and don't archive snapshots or do incrementals.
That's the problem with backups. They are expensive as hell, no one likes thinking about them, they should never be needed, and you always find out they're broken when you need them most.
"I guess they just make a mirror, and don't archive snapshots or do incrementals"
Do you *really* want your ISP keeping incremental backups of all of the mail they've ever received for you?
Your ISP might never need those backups, but it's a slippery slope from there to the government requiring your ISP to not only keep those backups but to hand them over on demand too...
It was not all that long ago that I received an email from those lovely people at Fasthosts informing me that whilt carrying out routine maintenance work overnight, they had inadvertantly wiped the hard disks of a number of servers. Regretably, apparently, they were unable to provide any backups of lost data but they were looking to ensure that this could never happen again.
Seems they forgot.
I remeber seeing an article just the other day, might have been Gartner, not sure, saying that in house email is expensive and many companies should just use hosting services... Looks as if we can see why the hosting services are cheap, doesn't it. Put the backups to /dev/null and they complete in no time every day...
Any "backup" process that can corrupt all backup copies when an error occurs doesn't deserve the name. You need at least two generations of backup to guarantee data safety. There must be at least one acceptably recent backup generation offline, and preferably offsite, at all times.
In this case the backup process succeeded, but with damaged data being saved which made the backup worthless. However a power outage, equipment failure or fire during the backup could have destroyed both the main disks and the backup medium. Its the same result in both scenarios: without an earlier offline copy you don't have anything to fall back on.
Fasthosts evidently got caught by the first case but could just as easily been hit by the second.
Email should be on a 7 day rotating backup. That should be enough time to notice an error like this and recover, but not so long as to be usefull to government snoops. Anyway, can't they send the disk out for STM data recovery? Seems like this might be one of those times when it is justified.
so this must be that period last week when none of my emails were responded too, and the ones that i sent TO MYSELF never arrived. marvellous fasthosts. actually some of them did arrive 8 hours later. excellent.
thank you reg for giving me an explanation that was not forthcoming from fasthosts themselves.
it did actually force me to create a googlemail account, which until now i had been reluctant to do on the grounds that they store everything for government processing for ever.
hmmm. ok. well, just leaves me with one gripe, which is to find out how someone accidentally created a mysql database in my account. i trawled through the mystery database emailed the administrator contact and it turned out that they had just been trying to create a database in their own account. ridiculous.
are the humans at fasthosts more prone to errors than in other hosting companies?
oh, and the frustration of them merging several of my accounts into one earlier in the year and changing my username. this would be fine except that 3 other people
shared access to one of those accounts which was very deliberately intended to be kept seperate from my personal one.
fasthosts is a complete joke a comedy of errors.
a mess.
a nonsense.
a laughing stock.
From our long standing time co-lo'ing and previous to that hosting with FH I am quite impressed they actually had email to loose. The 50% they lost was probably about half of the 50% they actually deliver and don't "loose" in the MTAs along the way. This applies to both Exchange and all their lesser mail boxes as they insist on using the same rubbish MTA's for all of them (a shelf of beige box servers with linux on them).
At least their customers can now say they only lost 25% of their email due to this fault as the rest was dumped at the MTA due to disk and processor contention.
I am pleased to say the customers we took with us happily receive two disk snapshots per 6 hours in two different locations, held until the NAS's are full, then oldest in is deleted first. We are at about 8 months retention at the moment, so bring it on FH!
You should of course run a backup BEFORE doing any major upgrades/housekeeping not AFTER, then ensure it has worked before starting your work (as much as is possible). Belt and braces says you run one after as well and these are kept out of the standard backup cycle for a period until you are sure there are no negative affects/data loss.
Any upgrade/housekeeping 'plan' should have a regression plan and a testing phase. Clearly there was neither. Questions should be asked of who ever is in charge and signed the work off. Simple change control procedures and standard operating procedures prevent cockups.
Pretty simple stuff.
Hiding behind their equally weighted AS Peers, which explains their transit flips 24x7...
Or is it not load balanced and just a single host!
OMG, you folks really need to look elsewhere ;0
Non-authoritative answer:
fasthosts.co.uk MX preference = 5, mail exchanger = mailserver.fasthosts.co.uk
mailserver.fasthosts.co.uk internet address = 213.171.216.195
>
"The firm apologised and says its fix means the same disaster cannot happen again."
I'm sure that FH customers will be ecstatic to know that FH will be bringing them *new* disasters, rather than simply recycling the old ones.
This is a perfect example of why I run my own servers.
Trust no one. Keep your laser handy. The Computer is your friend.
@Peter Knapp
Since when was Linux and a decent open source MTA be classed as rubbish? You can have a fully redundant and reliable MTA platform setup relatively easy if you know what you're doing.
I'm just guessing the guys at FH were simply lacking in the "know what you're doing" section...
... that those people in the "wot no backup" and "inadequate backup" camps should try actually doing it. It's trivial to do a backup of someones desktop - kick them off, do a backup, job done.
Now try doing a backup where files are appearing and disappearing at a startling rate - at a place the size of FH, probably thousands a minute. Yes, ANY snapshot backup you take is out of date and incomplete (not to mention inconsistent) BEFORE it's finished. Sure you can do real-time stuff with a mirrored filesystem - but then when you type the wrong command it takes out the clone at the same time !
The only way I can see to avoid this sort of problem would be to duplicate all emails as they come in and keep a separate backup store - then if you lose the mail you can go to the backup store and re-inject all the mails for the last <however long it's been>. That would be a huge store with it's own reliability problems, and you'd still have the problem of having to go back 3 (or more) weeks for the person that never checks their mail whilst "somewhat annoying" everyone else who suddenly gets all their mail (and spam) from the last three weeks for a second time.
If anyone SERIOUSLY thinks they have an answer then I'd like to hear it - whining pillocks who think it's as simple as managing two desktops in a network need not apply !
Well it doesn't surprise me things are going "belly up" for Fasthosts - things are getting very bad for them - any one can see that.
Backup systems failing
Internal Databases hosted on NT4 (old) hacked etc
Whats next ?
I for am glad I dont host any sites on there systems. I found a far superior small UK host and they provide me with everythign I need - the personal touch, the time and the effort the chat about anythying
Personally - we wont be using FH for anything else - I just can't trust them with my domains - ill be paying £200 to transfer all my domains out
As per usual FH tell us no real details - hide the fact that their internal systems are insecure
new host are: www.sar-hosting.co.uk
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2025
