The Lord Giveth and the Lord Taketh Away
See the title ^^
Microsoft accidentally wiped some nonprofit customers' M365 data before their retention period expired – and now says it cannot tell them what disappeared, or even whether they lost anything. The cockup followed Microsoft's retirement of its Microsoft 365 Business Premium grant, which gave eligible nonprofits ten free licenses …
"The company said it regretted the impact"
The only thing MS regrets here is not trying to force these nonprofits onto a more lucrative plan sooner. Totally unacceptible and completely untrustworthy.
At some point MS will screw up again like this but with a major business, such as a Fortune 500 company or a big law firm. It'll be interesting to see the fallout from that.
It would seem that the hyphenated form is standard but there are plenty of comparable words that typically aren't eg makeup, holdup, lockup...
I would use cockup to for an equally hyphenated fuck·up, reserving cock·up for an erection but of course I am perhaps with slight difficulty, taking the piss.
Curiously I just noticed make, hold and lock are verbs whereas cock is mostly a noun (to cock (v.) to set at an angle isn't so common.)
Gone off at half cock?
Good old Brown Bess [1]. Mostly reliable unless the squaddies of the day didn't look after them properly.
For many, many years the favoured musket of the British Army. Simple and robust (much like the average trooper then - as Wellington observed "they are the scum of the earth but they're *my* scum of the earth". Capable, in the right hands, of four shots a minute, thus neatly out-shooting the French.
Good morning ladies and gentlemen.
I'd like to start off with an idea for you to keep in mind. Your data in our cloud can be deleted by accident. It has happened before and it will happen again. Even though we will try not to. So now you've lost a load of data, or maybe not as we cannot tell, you would be a complete bonehead if you trusted us again.
Let's move on.
Silliness. Not in any way an accident.
It's an intentional, predictable flaw in their ops management.
It's an astounding lack of competence, even for the known to be diminished version of M$ that operates today.
We can drop this once M$ makes adequate restitution.
>” it takes a bit of a fool not to have a backup elsewhere, too.”
Trouble is the size of not-for-profits for which 10 free 365 business premium licences were welcome, would be the smaller organisations without a full time IT expert. I suspect they never really understood backups, because everything was in the cloud on 365…
Would not be surprised if some ran their accounts via a 365 account…
That is true, but it doesn't really change the situation. There are a lot of tiny businesses and charities and they often don't have proper systems because they don't know what to do or have the resources to get proper advice. I've been volunteer IT for multiple charities, including one affected by the license change*, and I've seen that firsthand. That does mean that these organizations are more likely to experience significant tech failures because they haven't done what they need to, and while it's sympathetic, it's also predictable. Other than hoping for knowledgeable volunteers** or these places putting more of their limited resources into competent IT staff, there's not a great solution to that.
* The charity I dealt with used the licenses that Microsoft decided not to offer anymore and had to change to other ones, but they did not lose data in this most recent event. I saw the license change notice and shifted accounts well before expiration.
** I don't necessarily include myself as the best volunteer IT, being a programmer in my main job and a better Linux admin than a Windows or Mac OS one. In the charities I work with, though, volunteers are difficult to obtain and to keep for most activities. Fortunately, I haven't had to be the person who finds and manages volunteers. It sounds like an unpleasant job.
"There are a lot of tiny businesses and charities and they often don't have proper systems because they don't know what to do or have the resources to get proper advice. "
There's a line between wanting to do good and the ability. Those with limited resources shouldn't try to do too much. There is also a point where paying for certain services is a necessity over relying on volunteers.
It takes a significant fool to use microsoft's software and SaaSS.
Such fools of course don't have a backup (isn't the disservice specifically designed to prevent backups from happening - after all, the sucker can't get the data out, aside from manually one at a time?).
Now, let's get rid of these negativistic sentiments and focus on the future! It is true that some data *might* have been lost but really maybe it wasn't even there in the first place, who's to know!
Instead allow me to show you the shiny new future that our new Copilot and AI enabled Microsoft 365 services will enable for your organization! You will be amazed at the ease with which Copilot will help you move on from old, transactional, document and files based activities, to new, hyper-converged, impactful and meaningful achievements in your strategic posture on [whatever the Copilot said that the NGO is about].
And on and on and on. Sorry, can't keep this up, I think I'm going to be sick.
(regardless on whether they use a human or AI "concierge" to do this; icon because we all know the tone of voice of those sales people)
Most of the non-profits affected are tiny. My own Mrs used to chair a small childcare charity and they were affected by MS decision to claw back the 365 license.
In terms of technical skill the workers at the charity could switch on a laptop and use a basic spreadsheet. That was about it.
Anything else had to be done by volunteers. We were engaged with another service which provided tech volunteers to charities that couldn't afford it to use their 365 bsuiness to set up workflows and forms to let the workers get on with the work without having to fight big, old spreasheets. But the license removal fked that up.
I suggested getting them off 365 at that point an onto a suite of FOSS tools but they were terrified of change and wouldn't do it, choosing to struggle on using the managers 365 student license.
This latested MS cock up will have lost their data on a significnt number of vulnerable children and families. Yes they should have back up but in reality the concept of a resilient IT systems with backups and BC is beyond them. And even holding their hands they didn't have the time to learn or get anything set up. It's corporate cruelty from MS and really really dystopian to just allow this group of people to be fucked over.
I do wonder if anyone has rasied it with their respective GDPR regulators. I think there is a decent argument that MS might well hold (joint) culpability for a major breach
re. I do wonder if anyone has rasied it with their respective GDPR regulators. I think there is a decent argument that MS might well hold (joint) culpability for a major breach"
What breach would MS be responsible for?
The data has been deleted, the data controller is responsible and that would be the people who gathered the data and put it in the cloud.
You're told that backup of data is your responsibility.
I don't think it's quite as simple as saying the charity is the controller, therefore Microsoft has no responsibility. The charity certainly remains accountable for its own arrangements, including whether appropriate backups were in place, but Microsoft as processor has direct obligations under Articles 28 and 32 UK GDPR. If Microsoft committed to retaining the data until a particular date and then deleted it early through its own error, the loss of personal data could constitute a personal data breach under Article 4(12), which expressly includes accidental destruction or loss. Whether that ultimately amounts to an infringement would depend on the circumstances, but saying “you should have backed it up” doesn't really answer the issue. A backup might have reduced the impact; it doesn't remove Microsoft's potential responsibility for deleting the data prematurely.
How about you should have switched to a different subscription or service or company when Microsoft told you your subscription was ending and wouldn’t be renewed?
I doubt if Microsoft is required to keep your data once you stop paying for a subscription, they do that to cover end user mistakes.
The fact that Microsoft actually lost client data is a shocking reminder to anyone who relies exclusively on cloud services. It doesn't matter who you are or how important you think you are, you and/or your company are insignificant when it comes to hyperscalars - and when they screw up (not if), you are going to be toast - with no comeback...
But there's a bigger story here... This morning, Microsoft's market cap - according to Google Search - is $3.74 trillion dollars. You would think that a company worth nearly four trillion dollars could offer a better level of compensation than "Ooops..."
Here, they have screwed up and wiped out data belonging to non-profit customers... That tells you an awful lot about just how out of touch Microsoft are. Maybe they rely on the attention-span-of-a-goldfish of the modern world... but that's a disgusting attitude from one of the biggest companies in the world.
The deeper perspective, perhaps, will be from commercial customers of the service. If Microsoft can do this with non-profits, then they can and will do so with anyone.
Is your business data important to you? Are you keeping that data in the Microsoft cloud? Good luck with that.
Perhaps more helpfully - read the small print. Know what would happen in the event of a future loss like this... Know what your contract small print says. Have a recovery plan. Test it.
"The Cloud" is just another way of saying that your data is on servers operated by a soulless third party who has no loyalty to you, doesn't care whether or not losing it will destroy your business, and can't be arsed to take proper care of it.
Act accordingly.
Maintain independent backups on hardware YOU control, and VERIFY that you can get the data back WHEN, not IF, the soulless third party loses it.
No, that's not what cloud was sold as doing. Cloud was sold as rentals of stuff that could be annoying to set up yourself, not end-to-end systems that removed your requirement to design your systems. You rent disks, servers, and network connections, either individually or with a software layer connecting them. You can easily rent a second set of those in a different location so you have geographic redundancy, but that doesn't happen unless you specify and pay for it. Depending on the specific product, they usually have some basic redundancy so you don't spontaneously lose data from a random disk failure, but that's not the same as a backup and competent admins wouldn't use it as such. But I think you knew that already.
Here in Wales the WGA has set up a Wales Wide Azure/Entra/Google cloudy behemoth called Hwb for use by Education related enties (schools, libraries, colleges etc.).
It works (for special values of works).
It isn't backed up.
It is mandatory.
Hwb don't provide any real support in backing it up (because all of Microsoft 365 onedrive/sharepoint and Google Workspace data stores are available so it was too hard for the poor loves who "designed" this mess to design in a proper backup system) and their published approach to headmasters is "its the cloud, you don't need backups".
To back up our school data from Hwb would cost about £22k per year - more than the total current IT budget - because, typically, backup is licensed by user (we have >700 users) not always by data volume (surprisingly we have less than 5Tb on prem, possibly another Tb in Hwb at the minute in Google Classroom). When we move to Hwb we aren't going to be backing up the data.
I am going to be arranging a bunch of CYA emails in the near future so I don't have to say "I told you so" too often.
That might be the case here... I think it will depend on the small print of the service terms that Microsoft were offering.
But, "Backup as a Service", BaaS, is very much a real thing, commercially offered to customers. Popular examples of BaaS include iDrive, Backblaze and AWS Backup. There might be a similar functional service in Microsoft's service - but as I'm not a user I don't know for sure.
On the other hand, if there clearly isn't - and if Microsoft's service descriptions clearly say that - then the impacted parties don't really have anyone else to thank but themselves, hard as it will be for them to accept that.
It definitely is a product you can buy, and Microsoft would be happy to sell it to you the same way many others would. The problem is that, if you want it, you have to specify it. If people believe that backups are included by default because it's cloud, they're mistaken and will learn to their cost that they never bought that part. In the same way, cloud can give you geographic distribution and automatic failover, a global CDN, as many disk snapshots as you want, and a bunch of other things, but only if you select it, and that mostly because they'll be charging plenty for each of those you choose to buy.
I've seen people who assume these things are supposed to be included with any basic cloud service. Sometimes, these are stupid people who decided that some big company running a building with servers in it meant they didn't have to care about configuring their ones properly. Other times, it is stupid people who dislike cloud who decided that lying about what the product was would make it easier to campaign against it.
"Popular examples of BaaS include iDrive, Backblaze and AWS Backup."
For some situations, that can be fine. If I'm a photographer working at a remote location, having a "cloud" backup in addition to making multiple copies on-site that are retained by separate people, it's good insurance. Personally, I'd just upload files to my own FTP space that comes with my hosting account. Most other photographers I know have gone with hosting companies that don't provide drive space so they have to subscribe to BaaS and services to deliver media to their clients. Mentioning my setup goes straight over their heads and is thought to be "too hard". They've been learned to pay money for things they could have for free.
" rent disks"
That's false: No-one rents you disks. They rent you disk space, some amount of gigabytes. Duplicated and/or otherwise guarded against data loss, so you don't need a backup.
That is the product rented.
Usually it also is that and you don't need your own backup, as long as you pay the rent.
Then the cloud provider fucks up and data is lost, oops.
Having your own backup is intentionally hideously expensive as every cloud provider charges arm and leg for data exports, i.e. your own backups.
That is not an accident: They got the data, they really, *really* want to keep *you* from having it. You may only have an access to it.
That is the whole business model: Vendor lock-in and the data is the extorsion tool.
"And if you have to go through the trouble of keeping your own backups just in case the cloud lost them, why bother with the cloud?
The bait on the hook is the data file is able to be accessed by multiple people without needing the IT experience to set that up. It's not that hard, but somebody will point at the easy button and the discussion ends.
"Maintain independent backups on hardware YOU control, and VERIFY that you can get the data back WHEN, not IF, the soulless third party loses it."
It also needs to be data you can do something with. If that data is your accounting file and you've gone with a web-based accounting product, the backup may not be worth much. For many things, the software being under your control and able to run without connections to the mothership is important.
I am surprised that people are surprised about this.
I have used MS as my primary OS for 4 decades since DOS 6*
I loved and apreciated what this company did, and enabled, all the way until doubts set in with ME, but then doubts totally confirmed since the advent of Windows 8, and completely unequivocal since the 360 product range and even worse since CoPi$$pot.
I have used MS as my primary OS for 4 decades since DOS 6*
My first OS was VMS. (Just a user). First IBM PC with DOS encounter was ~'85 -- v2.10, iirc. Found SunOS in '88 and Microsoft has always been considered inadequate ever since.
I'm really happy UNIX(-like) distros (BSD & Linux) became FOSS, because I don't think I could have afforded SCO on all these silly little home boxen.
" Test the DRP rigorously."
This is funny.
I've actually written disaster recovery plans, but not a single company has ever had time or money to actually test the plans, so I'm absolutely sure there is glitch *somewhere*.
If I knew where, it wouldn't be a problem.
One company did an estimate of work hours needed and it was in hundreds, for the actually competent people. So they skipped it.
An overview is an easy peasy: Find hardware, find OS, find application sources (and binaries), find the data(base), find compiler and libraries needed, compile. Voilá.
... and then there's the reality where even finding the hardware will be difficult ... and that's the easy part.
Well yes, back data up etc etc. I know. Have you actually tried? To get a regular unattended backup, you need to buy someone else’s cloud backup service. When you’re running on a charity shoestring, you have to somehow afford this. But then you’d think that a company worth shitloads would be a bit more generous with licensing for non profits. Google gives eligible non profits licenses for 3000 users and a free 100 TB of storage. Just like that. And no I don’t work for Google, I’m a charity IT admin trying to keep the ship running on donations and scraps.
I used to work for a firm of solicitors in London, back when using 4mm tape backups which were always verified before being taken off site by the it manager. He actually gave them to his secretary who took them home on the tube (London Underground) and had them safely on the floor next to her seat by the door which was nicely over the carriage motive unit. Each week we would get back a clean wiped tape until we worked out what was happening.
You cannot make this stuff up.
We had a regular rotation of floppy disks between IT and the main building across the street for certain days but whenever one particular operator took them, they ended up erased. Until we figured out they were using the entrance right next to the portable PET trailer and the MRI room.
If we step away from the customary Microsoft bashing, and ask ourselves what this means, I come to the following conclusions and I think they apply to all/most of the regular cloud service scammers.
1. We don't have proper backups that we can recover select data from.
2. We have no proper handling of accounts and data so we can't rollback our f...ups.
3. Since it's cloud it's hard enough / discouraged to manage proper backups of your data yourself.
"If we step away from the customary Microsoft bashing"
It's Microsoft in this case but it's the whole cloud concept that's the issue. It's somebody else's computer that you don't control. Don't make it your primary source. Use it if you want to share something from your own computer with somebody else. Use it for a remote backup. Do you trust a large corporation better than yourself? No? Then why trust them with your data more than yourself?
"Since it's cloud it's hard enough / discouraged to manage proper backups of your data yourself."
Not only that, it's hideously expensive if you want your own data to your own tape in your own safe, just as "transfer costs".
Which of course is BS, they just do not like you having your data outside of the cloud. Very simple.
Many UK training and education providers are non profit organisations. Which meant a good deal on M365 licensing.
If all their data has gone (which could not have been stored outside the M365 garden because of Data Protection concerns) then a fucktonne of students will have suddenly failed.
Although I guess it's a good lesson to learn early in your education.
The been counters insist that you pay the likes of M$ to host the company data, and to provide the 'office productivity applications'. The same bean counters parrot back the reason they do this is because they say that if something goes wrong they have someone to sue.
They refuse to let the company run FLOSS applications because there is nobody to take to court if something goes wrong - because they wouldn't have a 3rd party to blame.
Yet at the same time, when M$ cock-up (I am going with the hyphenated version) the company doesn’t take M$ to court because they know that (a) they wouldn't win (b) M$ has a war chest big enough to bankrupt your company dragging out the legal process.
This has got to change...
If you actually read the "license terms" for most of microsoft software and SaaSS, it requires the sucker agree not to sue and instead only do arbitration or whatever.
They don't seem to realize that at most microsoft will pay out somewhere around $2k USD, so after the company loses another 100,000 and decides to go through the legal process, it ends up with $2k (companies never actually cut their losses - the losses just keep piling up).
Not really. Microsoft were paid, or not in this case (donated licenses for up to ten people is what was discontinued) for hot storage of that data. They messed that up, but that's not the same as assuming they were also hired to have your backups, which they weren't. M365 data storage doesn't do what proper backup schemes are supposed to do, for example having historical versions, optimized for restoration, in separate locations. That's not the product. If you think that's what you're buying, you're not. Microsoft do offer that as Azure file storage, so if you want to hire them to do it and you trust them not to mess up with those*, that's what to buy. Or you can do it yourself, or you can decide to live on the edge and not have backups.
* I would tend not to use 365 for documents and Azure for backups of the same documents because I don't like the single provider risk. I don't particularly mind if you want to use either of those, but I would aim to have something not dependent on that account because that could break or be attacked.
With QuickBooks it is not in the agreement for the lower tiers used by smaller organisations. It is however provided as part of their enterprise offering. However, even then they managed to unrecoverably loose customer data when they migrated their payroll package a core of years back (failed to include notes/comments fields containing customer data in the backup/export file used to migrate customers to the new system...).
So I expect many are using Quickbooks and not regularly doing a data export/backup...
Think of dodgy foundations (e.g. Clinton, Blair, and Ayn Rand). Is the Atlantic Council technically a non-profit, alongside other tanks wherein supposed-thinking is claimed to take place? Then there are charities with chief executives on City of London 'compensations'.
Presumably, Microsoft's charity towards the virtuous world of non-profit paid-off through people in influential positions speaking well of Redmond.
Does your neighbourhood animal rescue centre have free access to MS subscription services? Would the centre be pilloried if it tried to exist on a home licence for its only computer?
We, Microsoft, know that non-profits are usually charities and you provide essential services to the most vulnerable members of society. We, Microsoft, know that you are providing your services on a financial knifedge. We, Microsoft, understand that the removal of the free 365 offering we lobbied you to sign up to and leaving you with either the shit online only version or a bill was a blow to you.
And we, Microsoft, understand that you likely struggled to get any technical support to help to move the data we, Microsoft, had incessantly pushed you to store in our system through forced synchronisation and nagware, into another form of storage. We Microsoft, are aware that our deletion of your data before the actual deadline with no notice or warning will be a real kick in the teeth to you and may actually put vulnerable people at risk.
But we have Claude tokens to pay for so fuck you.
As an early adopter of Microsft Skydrive, I was given 100GB of free storage when I signed up. It was a really neat way to use office apps and store files to share and collaborate on. One day, without any warning, I was locked out. Microsoft had rebranded it to One Drive, and everything was now gone. Thankfully I had it all backed up offline, but I was completely locked out and there were no humans to help. There was only a cycle of useless automated help that went nowhere.
From that moment on I've never trusted the cloud with important data. It's 20 years later and Microsoft are still doing this to people.
The office STILL backs everything* up to tape. Yes, the tape drives are expensive. Yes, the tape is expensive. Yes, certain people (Accounting!) have 'raised questions' about 'unreasonable expenses' when we could be using 'cheap/free cloud storage' instead. And, yes, such questions were answered by requesting that the questioner put their recommendation down in writing so that we could know whose fault it was when, not if, something happened to the cloud storage. Strangely enough no-one wanted to state in writing that they wanted to 'save money' by getting rid of the tape and relying on cloud storage.
*by 'everything' I mean EVERYTHING. The OS, the apps, the settings files, and, of course, all data. Everything**. And every ever so often we start up our test system and do a restore just to make sure that the backups still work; Accounting really hates us having some machines sitting 'idle'; if tape is so good, we shouldn't have to test things and we should be able to put those idle systems into use. Our ability to restore any machine on the network within hours, using tape and one of the 'idle' systems, has shut them up, so far.
**naturally we do NOT use MS software to do backups and restores. Accounting really fumes at the 'unnecessary expense' of using backup/restore software which actually works instead of the free MS software, which is almost worth every penny it cost.
I've been backing stuff up to tape for 40 years, since I was working on IBM and Harris and DEC mainframes and minis. I used the big tape units then, then DAT, now LTO. You'll pry my LTO tape out of my cold dead hands.
You could suggest to accounting that they arrange, from their own budget, that they move all their accounts to this cloud they're on about on the strict understanding that their only support will be what the cloud vendor provides. If the cloud provider loses their stuff it will be gone.
If they actually do that, wait a couple of weeks, then take them to your DC and tape store. Point out where everything that's not in the cloud it - live systems, tape backups, everything. Then ask them to point out where their stuff is. Probably best to be close to the toilets when you do the last bit.
Remember, ‘we are not at home to Mr Cockup’.
Oh too late, he’s through the door, and seems to be currently occupying various senior management positions at Microsoft.
All of the big cloud providers do make it clear that they won't be held responsible for your data - wasn’t there a story a little while ago from AWS who, having lost an entire availability zone due the Iran War (special military operation), told clients who had irretrevably lost data to ‘activate their disaster recovery plans’ and restore data from their geographically isolated backups?*
However although a lawyer will claim that this is ‘clearly’ in the Ts & Cs, it’s never front and centre when you sign up, all of the claims are about how much ‘the cloud’ frees you from worrying your pretty little heads about operations, because.... well cloud?
* I’m sorry what do you mean by ‘we are a tiny non-profit and don’t have any disaster recovery procedures or separate backups’? Well, sorry about that but if you can’t even take the simple steps of having multiple cloud providers in diverse locations with 15 minute snapshot backups of your primary data, you deserve everything you get.
anyone whose EVER said "don't need to backup on the cloud" or "cloud is cheaper & ALL your workload should be on the cloud" needs to be slapped.
The last 10 years my life has been full of these lunatics who moronically think that MS, Amazon or Google will adhere to their advertising & that on premises is always wrong...
even if on premises is more expensive (it isn't-it's LOADS cheaper than cloud) things like this show that you can't trust cloud ever!!! these are NGOs & there WILL be lives lost because of microsoft greed!
fine if every prick from gartner et al and even fucking architects who should know better isn't telling you that "cloud is best", "everything should be on the cloud", "you don't need to backup the triple redundant cloud", "it's 99.99999% uptime"
Also see aws in the middle east losing both DCs in the region
bet some CONsultancy is getting a sweet sweet pay off for that.
the self righteousness here annoys me.....
these are charities & MANY would have a sysadmin whose been thrown the job & doesnt have the knowledge base or the budget to do what the self righteous here are crowing over.
In my IDEAL situation EVERYTHING would be routed back onsite to a dish backup on SAN & then tape from that. HOWEVER egrees fees fuck your on that..... i'm hoping the EU REALLY kicks the hyperscalers & makes egress fees illegal.
The 10 we had, despite us having thought we'd moved everyone, started to randomly go pop so the user would come in with access to nothing. Then saw in the logs MS had automatically removed their license and gave them nothing back. So we were able to give them a new license instead which reinstated their mailbox.
"I think the moral is more "make sure none of those places is Microsoft's cloud"."
Wasn't it Amazon (or Google?) where they deleted an account by an error and *everything* related to account was instantly deleted: Whole infrastructure said *pofff*.
No way to restore anything at all, they said: What is gone, is gone. Sorry for that.
Not only the customer lost data and application, they lost whole application development also, i.e. source and documentation.
How do you recover from that?
Read the article again. The data did, in fact, come from expired subscriptions, after the access expired, but before the extra retention period. Accounts like this are supposed to keep their data for 90 days after the app access expires. Microsoft didn't do that properly and deleted the data earlier than that, but still after access was cut off. They shouldn't have done that, but that doesn't make your complaint correct.
This post has been deleted by its author