The wrong guy: Backup outfit Spanning deleted my personal data, claims Cohesity field CTO Cohesity field CTO Chris Colotti has said that Kaseya Spanning eradicated years of backup data belonging to him after he unwittingly breached a usage limit for an account that was supposed to have no pre-set limits, and then didn't upgrade his plan within a five-day period. Writing yesterday in his personal blog, Colotti …
This post has been deleted by its author
This time is was Kaseya being arse holes, next time it could be some malfunction, either way Kayeya are unlikely to worry much. How much work would you do for $48/year ?
The only way of being safe is to take your own backup, preferably two that you keep in different locations.
Yes: it is harder, more work, needs thought - but how much is your data worth to you ?
"I still wait for someone creating a “neighbour” based system. Both you and your neighbour buy in, then each has one backup in their home and one in the other home."
I think crashplan used to offer that model. The backups were even encrypted. I liked their service until they dumped us home users out in the cold.
Yes Crashplan for home users *used* to be good.
Peer to peer backups were available in addition to storing in their cloud (unlimited for a price).
The peer to peer stuff could even handle making the initial backup to a storage device locally (eg at full speed), detaching it and giving it to your neighbour, family member etc and from that point on it only had to transmit the deltas via the Internet which made things a lot more viable.
The NAS versions took a bit of tinkering to get working but I ran a setup for several years until they gave home users the old Spanish archer.
Some NAS systems support this. The problem, right now, is finding a neighbor with adequate internet bandwidth and a working WAN address to reach. 'Merica still has mostly antiquated Internet. IPv4 with NAT, 0.1 to 10 Mbps up, and monthly usage limits.
Business users too. Back as a networking tech we were on the phone with customer's ISP trying to negotiate symmetrical speeds. They simply didn't offer it. Customer had 100 down/10 up, would've even taken 50/50 Mbps. But just to get better upload speeds, they said the only plans available were 300 down/25 up, or 1 gigabit down/50 up, for 2X or 3X current bill. Arizona.
if you read El Reg then you should be able to work this out for yourself. Just to kick off an idea:
Raspberry Pi 2B or later with Ubuntu for you plus friend with Bacula configured to either
a) Backup locally (with encryption) and then sync the backup files to friend via rsync, syncthing etc
b) Backup (with encryption) direct to friends Bacula Pi.
Friend does likewise to back to you. Encryption means friend doesn't need to be trusted not to look at your stuff.
Yes, you'll need enough disk at each end for each others data (or both if you use the first approach), but disks are cheap and for a domestic setup how many people genuinely need a working backup set of more than a few hundred GB of stuff? When you get more then 5 GB of anything you want to keep forever such as photos etc, burn to a DVD. 2 copies of course.
I have one half of the above setup to back up data to a pair of 1TB USB disks - full data once a week, incremental data every night, OS backups every so often. 1 disk is in the house being used, other in the garage just in case something bad happens in the house. Swap over once a week. Only had to do a restore once so far, and for that my most important user was extremely grateful....
There's gotta be many other ways you can do this. And not a cloud in sight.
Symform used to do that. Then Quantum bought them and shut it down. Seems there's the generic term for this tech: cooperative storage cloud
For less than twice the cost of 1 year's subscription, you can buy a 1TB portable HDD. This will back up 10GB of data per month for more than 8 years. If it is only powered up when you backup or restore data, it is likely to be more reliable than an online backup service - as well as being a heck of a lot faster.
Plan on buying at least 2 or 3. One can fail and you are out of luck if you find out after your PC dies to.
And you need to take a copy off site from time to time.
Houses can burn down, get hit my lightning or be damaged by storms.
Or even hit by a car. Saw the video of that a few years back. Car hit the second floor. The house was on flat ground. But the divide road by it had a curb and the car was going way too fast.
Yes: it is harder, more work, needs thought - but how much is your data worth to you ?
Agreed. I wonder if his data was worth more than the £1500 to £2000 a 48TB NAS would have cost him (36TB plus parity data.) I guess he knows now...
I've always followed the policy that backups are not done unless I do them myself. I once returned from a hoilday to find that the sysadmin had reformatted my hard drive and deleted the backups because he thought I had left the company. Fortunately, I had my own backups on computers in other cities that he didn't know about.
Exactly, my important home stuff is backed up on my own hardwear with paranoic care. Since we last covered this on an El Reg thread I've done what I'd said I'd do then and bought a large capacity memory stick which takes all the family photos and a few important documents. This is now in my daughter's home, two or three mile away.
Then, after backing up the important stuff onto separate internal HDDs and again onto a pair of external USB HDDs that I alternate, then and only then do I consider cloud backups as a final fail safe for as much as I can save on free accounts ( several different free accounts, because, well, who knows).
Coincidentally a colleague used your backup to memory stick scheme for all the pics of his children, the data recovery bill for the the failed hard drive wasn't too terrible, only £500 quid.
I'd trust a memory stick less than almost any cloud backup, they have a nasty habit of being fine ten minutes ago to where's all my data gone?
They're only any good for transferring data between machines, the number of "high quality, lifetime guarantee" sticks that have failed for me isn't small.
And yes, I use name brand sticks, no they're not cheap knockoffs.
But, in my case the memory stick is the fall back in case my PC is destroyed, taking out the internal HDDs, the external USB HDD which is sitting on top and the other USB HDD on the shelf. And I can still reclaim most of my important stuff from the various clouds.
But now you have me thinking I should put the second USB HDD in another room.
Back in a minute.............
.....Oh. Did that and discovered I already have another backup in there, on an old laptop hdd that I'd stuck in an enclosure, but it only goes up to last September, I'd forgotten about it ( and to be honest there's not been much to backup since then anyway).
Terabytes stored ONLY in cloud backups? That's bad planning.
Sure, you need an off-site backup for the worst possible case. For many large companies, it can be at another site that they own or have long-term secure access to. But for small companies and individuals, the cloud is fine.
However, you should always have a local store as well. Because a large amount of data is going to be faster off local spindles than down through your internet connection. Sure, your worst case is that fire or flood means you have to go with offsite. But onsite is what you should be reaching for in most situations.
To not have that for 36Tb of data is baffling. Is it actually a backup? Or is it an archive? If it's just an archive, then it needs its own backup.
Either way, it's bad planning. Hopefully he's learned a lesson here...
What makes you think it was only stored there? It's certainly not mentioned in the article.
> "found his backup data had been deleted when he tried to login on February 18 this year"
Doesn't mean that he _needed_ to get something from the backup, nor that there wasn't any other backup. It's quite possible that this was when he logged in to test his restore process.
.....about backups...........the general tone of the article implies that backups are a "push" process.......along the lines of "upload the backup AND ONLY download if you happen to need something".
*
This implication is quite misleading. It's important to do regular testing of the restore process. If these tests expose ANY FAILURE AT ALL, then the backup is likely to be either corrupt or completely useless when a restore is really needed.
*
This regular testing is necessary for local backups, and I would have thought it is EVEN MORE necessary for cloud backups. I wonder if Chris Colotti was completely certain that his 36TB would be effective when he really needed to restore all or part of it! If he was not certain, then he was wasting his $48/year fees.
Why bother to try to read the EULA? We all know that it says "You agree to send us money and we don't promise to do anything -- at all ... ever -- in return" in about 100,000 carefully chosen, incomprehensible, words.
As far as I can see the only hope we consumers have is that a vengeful and angry deity will someday turn up and start casting nasty vendors and their legal staffs into a blazing pit. I do not anticipate that happening any time soon.
And this is not a new thing. 50 years ago it was known as the bus ticket clause. Because the Ts and Cs referred to on the back of the bus ticket basically said that in return for buying a ticket they'd try to take you somewhere other than the place where you got on, if they could.
> this guy is a STORAGE EXEC
> unlike some of you guys, I have no sympathy for him. He definitely should know better.
I trust he knows better if he preferred to avoid own backup company. As every exec worthy of own ambitions, he could happen to have a large mortgage on an expensive modern glass house. I hope he is careful when throwing stones into other businesses, they tend to return home.
Source: https://www.cohesity.com/what-we-do/
Begin with Backup
Your data is a uniquely valuable asset to the business, but first, it needs to be protected and available at all times. With the Cohesity platform, you can ensure continuity and repurpose your data for additional value. Start by eliminating infrastructure silos to remove cost and complexity, then do more with your data through value-added apps and services.
(no popcorn icon, so yet another beer for me)
Cohesity does offer data protection of some SaaS products. They do not appear to offer one for G-Suite (now Google Workspace). They also, like my employer, Druva, do not market to the "prosumer" space.
I therefore see nothing wrong with him using another company's service to back up his personal data. This isn't a case of "he didn't eat his own dogfood."
Thanks for clarification regarding Cohesity not supporting Formerly G Suite, upvoting. I did not spend much time checking what they can or cannot do. I checked again now and it seems to be one of those companies whose product is very difficult to understand. There seems to be not much technical data at all: I did not find any technical manuals, documentation, technical product description, reference architectures, installation and integration guides --- but there is a lot of marketing glitter indeed. They seem to sell something that can be installed in minutes, does not require maintenance, provides 95% reduction in recovery times, 20x faster restore time (thanks, I can do math), fully protects from ransomware and reduces backup costs by 40% --- you just need to believe.
When thinking about advantages of believing in marketing, another thing came to my mind. I can question this fine gentleman's canine food preferences however I certainly cannot question his faith: he seemed to seriously believe in "no limit on storage capacity or number of backups" for just 4 dollars a month. It is truly a disaster to lose faith, I wish him the best of luck in recovering it.
I would be interested to see the Mean Time to Restore Faith metrics though.
Honestly, yeah, if you sign up with a company that offers unlimited hosting and then they change around the EULA so that when you renew, you're kicked off, that's absolutely shitty and I sympathize. I doubt this dude lost everything, or anything like that, but he's royally pissed off that a promise that was offered wasn't kept, and I would be too.
Let this be a lesson to any company that offers "unlimited" anything.
If a service is really offering to store 36TB for only $48 a year then either (i) it is a scam - the data won't be there when you need it, or (ii) the company is going to go bust and the data won't be there when you need it.
If you really want to store important backups online, check out major market leaders (Amazon, Google, Backblaze, etc) and expect to pay prices similar to those. Otherwise: the data won't be there when you need it.
For my personal data, I have good local backups and I use cloud storage for offline backup - expecting to never need it but just in case. I am storing a few TB and I am paying much more than $48/year.
I'm not suggesting that AWS S3 Glacier is the cheapest online storage available, but it's certainly not expensive. Their quick calculator suggests that they charge ~$4/TB/month, so he'd be looking at ~$144/month plus retrieval charges. The guy supposedly understands storage so how did he think he was going to get away with 36TB for $4/month?
Such things like this is why I prefer to keep three copies of backup.
One on Amazon glacier. At this stage it is cheap enough, so we'll just keep on filling it up until we reach a certain point, then look at deleting older data.
One on a NAS - weekly
One on a removable HDD (a new HDD is procured every month and a full backup is chucked onto it) - monthly for 6 months.
Not to trust ANYONE else with your data. My 16 TB is replicated across two servers and a tape system; none of the three are collocated and both servers are dissimilar systems. Compared to the loss of the data, the cost of the system, built in 2013, is immaterial.
I have a similar amount of data; one live set, one local backup and another in a different city at a location I work from on occasion. I haven't used tapes in a long, long time as hard drives are pretty cheapish.
The physical servers were built in around 2010 and I've had to replace the odd power supply as the one location gets a lot of spikes killing both UPSes and PS.
I use OmniOS with Napp-it for a fairly minimal Solaris derived OS with ZFS to reduce the risk of data corruption. I've had the odd drive fail after the initial rash of a dozen of my 3GB Segates clicking to death.
I did reinstall the OS when OmniOS went under and became the community version just to start clean, but that is a quick task.
The HGST's and WD Reds I use now are rock solid, and I usually just swap in place with larger and re-purpose the smaller old ones to another machine when I want more space on the servers.
Most of the data is pretty static, usually deltas are less than 50GB per week. I either replicate over the wire or if there is a large change I replicate the delta to a hard drive and physically transport that and replicate to my off site.
Anyway, the TL;DR moral of my story is;
One copy local,
One at a safe site,
ZFS for integrity.,
Sleep well at night
For the rapidly changing data (about 500GB) I usually keep at least 4 copies\versions on the on site backup server with timelines based on my risk aversion level for each workload. It varies within that data set.
I do have an old desktop (another OmniOs) with a pile of drives in it on site that I replicate manually to every so often in case the source replicates a corruption problem on the more static data as you mentioned. Its a lot of smaller drives and weighs a ton so I don't travel with it.
I eat the risk on the deltas for the more static data, but its nothing that I cannot re-source for the most part (i.e. ripped copy of some of my DVD's or something along that line)
Really with ZFS you can do periodic snapshots and rollback (aka manually go dig out to recover) older versions of files.
This however does consume more space for every disk write (all deltas have to be preserved) but you can do different pools based on what criteria you are looking for. I guess you could even do a file comparison hash on source and destination files to alert on differences. That would take some time on my current setup as I am sure there are millions of files.
NOTHING is unlimited. I don't expect average people to understand this. But ANYONE working a technical job in the IT industry must know this. Even CTOs.
Honestly, I would expect England's advertising standards body to go after anyone making an "unlimited" offer.
But I'm having a really, really hard time sympathizing with this guy.
Back in the AOL days, I used to have a dedicated phone line just for the AOL freephone dial up connection. Was connected 24/7.
When we ended up getting adsl after a few years of that, when cancelling, we complained a little when we didn't get the correct refund. The guy on the phone said, you had connected a couple of times in that period, and looking at your connection history, your account is probably the most used we have.
I said, ok then, didn't matter. 3 years 24/7 connected for 9.99/m. 56k, nice.
The AOL client would disconnect after 15 mins of inactivity, just set a ping up on the computer sharing the connection, always on.
Didn't read the EULA.
Didn't do his sums.
If it's too good to be true, it probably is.
Assuming even at bulk hard disk prices with no resilience, you're talking at least $10 per TB, or minimum $360 for the amount he was storing. Mirror it and it's $720. And then there's data transfer costs too. Far less than the total amount he paid. He thought it was a "win-lose" situation, it was actually a "lose-lose" situation.
It's AWS-hosted - and even at Glacier prices, 36TB would be $1770 per year. No wonder they fucked him off, the only surprise is they didn't do it much MUCH sooner...!
Also, if they'd let him export his 36TB backup, then the AWS Data Egress charges would have been over $3,000. I'm not surprised they just threw it in the bin if they were legally covered to do so...
Of course they are.
Until you decide that they aren't any more.
Anyway, this whole thing just demonstrates once again that you should actually read the EULA for products that deal with backing up your data. Nothing is more important than that.
That said, Spanning is definitely guilty of false advertising. I checked out their website and it is clearly stated : "With no limit on storage capacity or number of backups ". To put that on your website and then bury limits in the EULA is really bad form. Spanning is clearly spinning a lie to bring in victims customers, no doubt betting on the fact that most people don't have all that much to backup.
Sneaky and dishonest.
"Unless he's suffered a catastrophic local disk failure between nov 28 and now"
The reason you back stuff up is to protect against the possibility of such a failure. Irrespective of whether he suffered such a failure or not the possibility that he'd paid (a disproportionately small sum) to mitigate remained.
That's how you get to 36TB.
Anyway, he may not have lost any current data, but if he was using Spanning for archival purposes he's and he wanted to access some historical stuff he's got a problem now.
Say, the IRS wants to have a closer look at some large tax deductions he'd claimed for a previous declaration ...
Have you seen the mind-boggling sizes of recent iPhone images and video? Photos are 12000x9000 and video is 4K, plus you have motion photos which are a photo combined with 5 seconds of video, and that's on by default! If you take a lot of pics and vids, you can accumulate terabytes in no time. Same if you're a photographer using a recent high-end SLR.
It's not your daddy's data storage world anymore.
This CEO is an idiot. Don't care what the EULA says don't be paying such tiny amounts of money for such a massive amount of storage, the math doesn't work out, not even close. It's like people wanting to leverage google drive for a few dollars a month and storing hundreds of gigs to tens or hundreds of TBs and somehow think that is a reasonable thing to do. It's so beyond absurd I don't even have words.
And what kind of small business has that much data? Really sounds fishy. Maybe this CEO is trying to disguise his hoarding habits by saying it's for his "Small business".
To add insult to injury the guy is a CEO of a storage company. He has no right to be angry he should be embarrassed for being so stupid about this, and then doubling down and going public about it. Hell he could of fired off an email to people on his IT team and said "hey I'm thinking about using this for X, what do you think?" Or maybe he did and he didn't agree with their response.
@Nate Amsden
I totally agree with you. But there is maybe another scenario…
1. He convinced his wife to backup her business data to a different company to the one he works for, because he knows best…
2. Now it has gone wrong, because he did not know best, so now he is trying to blame anyone but himself…
3. I wish him good luck in getting a shag from his wife in future.
4. The man is now, literally a cheapskate I know everything, wanker.
They really shouldn't use the "unlimited term" - if you can't handle the extreme users, you're not really offering it.
I use Backblaze, which unfortunately uses the same word, and I'm at about 2TB, which is unlikely to grow. Decent company, it seems, but I'd quite like them to stop saying that. I wonder if saying "up to 5TB" makes people want the ability to pay less for less storage?
Maybe I'm missing something but doesn't "Backup" imply that you are making a copy of data and that this is not the only instance of the data?
To give an example for the benefit of Mr. Colotti: I might take a photo with my camera and the resulting picture may only be on my camera. If I copy the picture to a backup and if this backup is later deleted, my only pain will be that I need to backup the picture again; I won't actually loose any data.
How could somebody in the industry not understand the difference? How could somebody in the industry who doesn't understand the difference be so stupid as to come out with a public rant which just serves to prove his stupidity and lack of suitability for the job he is in?
Either:
He genuinely has been using Keyasa for primary storage, in which case he's a tool and should know better given his job, or...
He's just lost the copy and not the original, in which case he's a tool for making a fuss about nothing.
Either way he's a tool. Sure he didn't get what he thought he was paying for, but he should know better than anyone to read the small print.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
