Being one of the 1% sucks if you're a Rackspace user As the Rackspace email fiasco approaches week three with the company's hosted Exchange customers' data in limbo, Rackspace execs still won't put an exact number on how many customers were affected by the ransomware-induced email outage, or when — if — they'll be able to recover their old messages and contacts. When asked for …
This post has been deleted by its author
I'm sure that many more than 1% are learning what "fanatical support" from Rackspace looks like, and those few who are still under the illusion they are receiving something of value or that reduces their risk will have been disabused of that belief. Come to think of it, reputation, loyalty and stickiness are a significant factor in determining goodwill on the balance sheet, and thus enterprise value; I wonder if that won't have to be adjusted by substantially more than 1% after this?
With a friend like that handling corporate communications, who needs enemies ?
Oh, you're aware that Rackspace has lost the trust of its customers ? No kidding. I'll wager that Rackspace lost a lot more trust when you openly stated that it's "just 1%" of your customers that have been impacted.
And I'm thrilled that you're committing to a full disclosure - when the time is right. When you're ready to. In the fullness of time, as Sir Humphry would say. Except that that is not how it's done when you want to show that you're actually committed to transparency and want to demonstrate that you're doing everything you can to get everyone back on their feet.
Vague promises of "the vast majority" is just a load of hot air, and you're blowing very hard right now with nothing to show for it.
This post has been deleted by its author
Too many corporate comms teams have the "important thing here is we don't want to say anything and ever have to walk it back" mindset.
The important thing is to be truthful, with the facts that you have to hand. And be honest about any ambiguity. And when you say something that later turns out to be wrong, own it, admit to it, and do better next time.
The fact that it seems to be small businesses bearing the brunt of this makes it even harder to accept because they are the sort of organisations who can ill afford to have a day of downtime, let alone three weeks. Rackspace has been dropping in quality like a stone for years and their indifferent response to this problem makes me want them to finally disappear into a big black hole and never surface again.
This post has been deleted by its author
.. how hard a restore would have been with a simple, Open Standards compliant server setup.
Oh no, wait, Outlook only talks IMAP/SMTP, not caldav or carddav, and anyone wanting to interface with Exchange will have to pay a license fee for the protocol.
Anyway, that aside, isn't the expectation of outsourcing that someone else takes care of tiny details like running decent snapshots and backups which are tested to ensure restores actually work? Or am I missing something here? Is it more "hand us money and we'll do it as badly as you would have done it inhouse because us sharing resources is only to make us more profit?".
That restore should have taken a day, and that's being very, very generous - I would have expected hours at most.
I have a feeling that 1% number may be growing soon. If I were a bigger tier company I'd look at that and start asking some major questions. I'd also start working on a plan B (aka another provider) with a fair degree of urgency because it has laid bare an attitude to resilience I would be very uncomfortable with from a risk management perspective.
Or, if you're EU based, you might want to bring it inhouse. Easier from a GDPR compliance perspective.
This post has been deleted by its author
"Step 1 - Spin up parallel server instance, different dns name"
How do you know the parallel server instance also isn't vulnerable to the same intrusion that got the first instance?
All you'd be doing is creating an even bigger clusterfuck.
Microsoft have revealed time and time again, the Exchange they host themselves hasn't been vulnerable to the zero days the Exchange they give to others.
They should be sharing best practice for securing Exchange environments (whether that's on-premises Exchange or SP's using it for Hosted Exchange). If they can secure 365 Exchange then everyone running Exchange ought to be offered the same capabilities.
That's assuming of course RackSpace had implemented all the security recommendations they should have.
>>That restore should have taken a day, and that's being very, very generous - I would have expected hours at most.
ha ha ha ha ha ha ha ha ha ha ha.... and breathe..... have you ever restored an Exchange server? If you have, with a large server and many users, and it took less than a day please accept a christmas beer my heart felt apologies and my congratulations for managing something many people before you have failed to do.
If you haven't, then I suggest not pronouncing upon things that you wot not of! Exchange is a prime example of software that claims to JustWorktm and really doesn't when it comes to restoring the mail database.
While I agree that Exchange is a total nightmare, if you have the right backup software, it can be done without too much effort.
Back when we had an on-premise Exchange server, we had Exchange specific add-ons for our backup system. I had the ability to restore anything from the complete information store, to an individual email. And, yes I tested it regularly.
I often had to restore email folders that people accidentally deleted (most users were set to clear deleted items on Outlook close - storage issues). I restored full mailboxes a couple of times where people had left the company years ago, and suddenly some exec needed to see the mail in the box.
I did a test restore of the full information store to a new Exchange instance a few times, just to make sure it was possible.
Exchange Backup for small/home business
Bacula triggers script to call Exchange backup to dump the database to a file, backups up that file, and them sweeps it to an archive area and delete yesterdays archive. Nothing fancy beyond a bit of config and a bash script. Using free Bacula edition too. Annual VM export to speed up a full rebuild should the worst ever happen.
Exchange Restore for small/home business
Upload VM export in the event of total loss. Restore backup file if that's been lost. Use Exchange tools to restore what you need from that. Win.
Only time I had to do it, it took most of a day to get some lost mails back - and that was mostly googling for the clue on how to do it for real. Not difficult, but not exactly scalable either. And not something for the non technically minded to attempt so hardly suitable for most small businesses to attempt to do in-house.
Bootnote
Now the whole lot has been migrated to postfix & dovecot for mail and Nextcloud for shared xDAV calendar & contacts - and the other goodies it can do which I'm still playing with. Mail is easy to restore (at least under test conditions) the calendar & contacts will be a problematic if somebody accidentally deletes something - still not something for the non technically minded to attempt. And yes, getting postfix to work was an interesting learning experience...
This post has been deleted by its author
You have obviously never had to keep an Exchange environment running, secure from malware, phishing and up-to-date 24/7/365. To suggest bringing it in house for a small non IT focused company is sheer idiocy. Move it to O365 or Googles business product. MS has a team of 10k people to keep it running. Also make your own cloud backups, you control, separate from MS. Oh and Postfix is a pain in the #ss.
This post has been deleted by its author
At my previous job, one of my last projects to finish before I left was to complete the migration away from Rackspace. I closed out the last business we had with them about a month before I left. They were really great at one point, but started rolling downhill fast.
We noticed it when all of the really good support people were leaving one by one. We had senior support people that we worked with regularly, and every few weeks, we would call in, just to find out that the person we were asking for was gone. Then, they completely off-shored all of their support. That is when we really started moving services fast.
I'm not defending Rackspace, but what basis do you have for claiming the restore "should have taken a day"?! We had a fileshare server whose differential backups completed handily overnight, but a bare-metal restore from tapes -- we had some kick-ass, auto-loader tape library boxes connected (each) via two 1GB/sec Ethernet links (1990s) -- took three days!
In that case you've picked the wrong plan or the wrong MSP. Or both.
Outsourcing done well can save you a whole lot of problems and money, and most techies will go a above and beyond, especially if treated with courtesy and respect, whenever something interesting occurs.
But you have to make real sure that you have the right contract and you're buying what you need instead of the cheapest you think you can get away with. If your sole aim of outsourcing is to cut costs and to screw every last penny out of what should be a partnership then you are very likely going to end up with regrets.
"you are very likely going to end up with regrets"
Ah yes but the people who architect these deals typically move on before the shit hits the fan, so they definitely don't have any regrets.
Not unless their past catches up with them and people start to realise the cost "saving" they're so proud of ended up ultimately taking the ship down.
I guess you didn't didn't read the article. It's mainly small and medium businesses that took the hit. The ones, that can't affird a server team, a network team, a comms team, a security team.
Also the one that can't say, fix this, or your going to lose $50million of business from us, oh and we'll sue your ass into bankruptcy.
Er, I thought this was hosted Exchange? Why are they blathering on about PST files when Exchange stores its mailbox database as EDB files, and not certainly separately for each mailbox?
I'm glad I bit the bullet years ago and bought the MS Action Pack for £350 as I am happily running my own Exchange 2016 server at home instead of relying on someone else's "cloud" solution (yes I'm still in the license terms - no a linux email server doesn't suit my needs). If I cock it up and don't take backups or don't take security seriously enough, I've only got myself to blame.
My understanding is that they are doing some sort of automated extraction. It can be done… but it ain’t fast, and if the database is corrupted it won’t be 100% accurate either.
The contempt Rackspace has shown for customers is unbelievable. Yesterday, for the first time since this dumpster fire was lit, I received an email from Rackspace telling my that pst files may soon be available, but it may only be a partial recovery.
What. A. Joke.
Rackspace is dead to me now. I managed to recover > 95% of emails and moved on.
This post has been deleted by its author
"When asked for an exact customer count, "It's 1 percent of our overall company revenue," Rackspace Chief Product Officer Josh Prewitt told The Register..."
This kind of crap drives me nuts. He was asked for the number of impacted customers, even if he can't give an exact number he could give an estimate, but instead he chooses to focus on the only thing that matters to them, revenue, and in doing so makes it all about the company rather than the affected customers.
I'm so sick of this kind of attitude from businesses and governments who think everybody is obsessed with revenue / growth / share price etc, and seem to forget that without the customers they wouldn't have any revenue in the first place
Typical selfish, greedy minded response from a C-Suite and tells you all you need to know about the companies focus and priorities.
Or in modern parlance..."Tell me you're obsessed with revenue without telling me you're obsessed with revenue"
/Rant
Let's say their exchange is competitive with office357 or whatever their uptime is.
Exchange mailbox plan 2 100GB sounds good for a small or midsize business and costs £6 a month or £72 a year, roughly $85.
That could mean their 1% or $30M in yearly revenue would be as many as 352,941 exchange mailboxes and the CEO doesn't give a shit about any of these clients.
Mmm, great company, think I'll look elsewhere.
We do a lot of our big server downtime over the hols. This year was a big one. Migrate the UK servers from the steaming mess that has become of Memset/IOMart to Rackspace.
Looks like a quiet holiday period for me now :)
It's rare that you see such utter unashamed contempt for your customers, and here we are saying the ones this affects the most don't matter to us. So that's ok then. Well you've lost our contracts now. There's a difference between mild ineptitude doing a major job, rolling it back and saying sorry, and actual arrogance to cover up the fact YOU screwed up and don't seem to be able to fix it.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2025
