Backup software vendor Veeam deleted forum data after restoration SNAFU

Data management software vendor Veeam has admitted to an embarrassing oopsie: messing up a restoration job and erasing data. The good news is this wasn’t a mission-critical mistake, or the result of problems with the company’s products. The company’s error was disclosed on its forums in a February 11 thread in which a product …

[Alex 72]

Taking Responsibility

An American entity which owns up to its mistakes and keeps on serving sensible solutions thereafter with no ill effects, it can be done. People in Europe can be nice about it. I wonder who could learn from this.

[john.jones.name]

otherwise known as hobsons choice

you have none

get over it

press relations matter

get a clue

[rcxb]

Didn't backup frequently enough?

So they restored a day-old backup, and there's nothing they can do about it?

So they weren't taking backups frequently enough to have a recent copy just before the overwrite? And/or they can't be bothered to compare two versions of the database, and merge the changes?

[DanAU]

Re: Didn't backup frequently enough?

They said that comparing the two versions and merging the changes would have been too hard:

> As a result of significant changes (new topics and comments) made since the time of the restore, we are unable to integrate the topics and comments from yesterday and todays [sic] morning into the current version.

I'd believe it... They'd need to change every overlapping auto-increment ID, and it's easy to mess that up.

[vulture65537]

Re: Didn't backup frequently enough?

Maybe they could have gaps in the ID range reserved for corrections.

Maybe they could have a journal of changes capable of replying parts.

[Danny 14]

Re: Didn't backup frequently enough?

not really. Data will now be lost either way. You lose the delta between old backup and new data, or lose the delta between the new backup and new data. Not all databases can be merged cleanly in these situations, depends how complicated your IDs are. Restore plus increment of IDs would be great in hindsight of course, but since this was a mistake rather than a need to restore I can see why this happened. Still, it shows their restores work fine, their development practices not so much.

[Jason Bloomberg]

Move fast and break things

"a backup of the forum database from one day ago was mistakenly restored by internal infrastructure DevOps team over the production database"

Maybe they'll put some mechanism in place to prevent that, or at least force a backup to be taken before it's allowed.