back to article Being one of the 1% sucks if you're a Rackspace user

As the Rackspace email fiasco approaches week three with the company's hosted Exchange customers' data in limbo, Rackspace execs still won't put an exact number on how many customers were affected by the ransomware-induced email outage, or when — if — they'll be able to recover their old messages and contacts. When asked for …

  1. Blake Davis

    Customer count

    'When asked for an exact customer count, "It's 1 percent of our overall company revenue"'

    Let's see...1% of 3 Billion is 30 Million customers. Did they mean 1% of the revenue of a specific time?

    1. katrinab Silver badge

      Re: Customer count

      Also, 1% of revenue is not the same as 1% of customers, because presumably those customers would also buy other products from Rackspace.

      1. BOFH in Training

        Re: Customer count

        It may be just 1% of the yearly revenue, but the bad PR is probably costing them alot more then the 30mil.

        Future customers may just look at how long it takes RS to recover from an issue and decide not to go with them.

        3 weeks and counting is pretty long.

        1. Crypto Monad Silver badge

          Re: Customer count

          I read it as code for:

          * Exchange Hosting counts for 1% of Rackspace's overall revenue, and

          * 100% of Exchange Hosting customers have lost their data

          If it were any better than this, they would have given the figures in a different form.

  2. Lil Endian Silver badge
    FAIL

    The Conceited Fuckers!

    Prewitt: "From an overall perspective for Rackspace, at 1 percent of the company's revenue it's not as impactful to the company overall..." and we don't give a fuck about you, you scum! Ha! Proles!

    Corporate sociopathy!

    "...when we say something, it's true."

    Well, that's okay then. Honesty vindicates you entirely! Except, no, it doesn't. You might get to spend a few years in Spandau rather than popping a pill, but guilty is guilty.

    *puke*

    1. ffeog
      FAIL

      Fanatical Support

      I'm sure that many more than 1% are learning what "fanatical support" from Rackspace looks like, and those few who are still under the illusion they are receiving something of value or that reduces their risk will have been disabused of that belief. Come to think of it, reputation, loyalty and stickiness are a significant factor in determining goodwill on the balance sheet, and thus enterprise value; I wonder if that won't have to be adjusted by substantially more than 1% after this?

  3. Pascal Monett Silver badge

    Rackspace Chief Product Officer Josh Prewitt

    With a friend like that handling corporate communications, who needs enemies ?

    Oh, you're aware that Rackspace has lost the trust of its customers ? No kidding. I'll wager that Rackspace lost a lot more trust when you openly stated that it's "just 1%" of your customers that have been impacted.

    And I'm thrilled that you're committing to a full disclosure - when the time is right. When you're ready to. In the fullness of time, as Sir Humphry would say. Except that that is not how it's done when you want to show that you're actually committed to transparency and want to demonstrate that you're doing everything you can to get everyone back on their feet.

    Vague promises of "the vast majority" is just a load of hot air, and you're blowing very hard right now with nothing to show for it.

    1. Lil Endian Silver badge

      Re: Rackspace Chief Product Officer Josh Prewitt

      Yes, I do hope the statements have not been mutated by the media[1] and are reliable. Is there a complete transcript available of his statement?

      It just seems to me that Rackspace are doing everything in their power to devalue the company. (RXT stock "surges" up $0.08 to $2.84.)

      ("In the fullness of time" made me lol, Sir Humphry was on my screen when I read that! Good timing!)

      [1] Jessica, I am not implying you here. Season's greetings!

    2. MyffyW Silver badge

      Re: Rackspace Chief Product Officer Josh Prewitt

      Too many corporate comms teams have the "important thing here is we don't want to say anything and ever have to walk it back" mindset.

      The important thing is to be truthful, with the facts that you have to hand. And be honest about any ambiguity. And when you say something that later turns out to be wrong, own it, admit to it, and do better next time.

  4. The Oncoming Scorn Silver badge
    Coat

    Its Panto Season

    In a similar vein, the company's not ready to disclose how the intruders gained access to the hosted Exchange servers,

    Intruder Windows! Obviously.

  5. Doogie Howser MD

    Poor

    The fact that it seems to be small businesses bearing the brunt of this makes it even harder to accept because they are the sort of organisations who can ill afford to have a day of downtime, let alone three weeks. Rackspace has been dropping in quality like a stone for years and their indifferent response to this problem makes me want them to finally disappear into a big black hole and never surface again.

    1. Lil Endian Silver badge
      Thumb Up

      Re: Poor

      ...their indifferent mercenary response to this problem...

      Other than being far too generous, I'm with you.

  6. Anonymous Coward
    Anonymous Coward

    I just wonder ..

    .. how hard a restore would have been with a simple, Open Standards compliant server setup.

    Oh no, wait, Outlook only talks IMAP/SMTP, not caldav or carddav, and anyone wanting to interface with Exchange will have to pay a license fee for the protocol.

    Anyway, that aside, isn't the expectation of outsourcing that someone else takes care of tiny details like running decent snapshots and backups which are tested to ensure restores actually work? Or am I missing something here? Is it more "hand us money and we'll do it as badly as you would have done it inhouse because us sharing resources is only to make us more profit?".

    That restore should have taken a day, and that's being very, very generous - I would have expected hours at most.

    I have a feeling that 1% number may be growing soon. If I were a bigger tier company I'd look at that and start asking some major questions. I'd also start working on a plan B (aka another provider) with a fair degree of urgency because it has laid bare an attitude to resilience I would be very uncomfortable with from a risk management perspective.

    Or, if you're EU based, you might want to bring it inhouse. Easier from a GDPR compliance perspective.

    1. Lil Endian Silver badge
      Thumb Up

      Re: I just wonder ..

      Nicely posed the question in paragraph three, and answered.

      Is it more... "...only to make us more profit?"

      One does so enjoy such rhetoric!

      1. KLane
        FAIL

        Re: I just wonder ..

        Step 1 - Spin up parallel server instance, different dns name

        Step 2 - Restore last good backup of mail database(s)

        Step 3 - Give affected users read-only access to their messages

        How is this not doable?

        1. usbac Silver badge

          Re: I just wonder ..

          You're assuming a backup was actually done. Without a backup, how do you do Step 2?

        2. cb7

          Re: I just wonder ..

          "Step 1 - Spin up parallel server instance, different dns name"

          How do you know the parallel server instance also isn't vulnerable to the same intrusion that got the first instance?

          All you'd be doing is creating an even bigger clusterfuck.

          Microsoft have revealed time and time again, the Exchange they host themselves hasn't been vulnerable to the zero days the Exchange they give to others.

          They should be sharing best practice for securing Exchange environments (whether that's on-premises Exchange or SP's using it for Hosted Exchange). If they can secure 365 Exchange then everyone running Exchange ought to be offered the same capabilities.

          That's assuming of course RackSpace had implemented all the security recommendations they should have.

          1. Anonymous Coward
            Anonymous Coward

            Re: I just wonder ..

            They should be sharing best practice

            The vast defect list over the last few decades suggests there is no such thing for Microsoft products as such, only for selling them.

    2. 42656e4d203239 Silver badge
      WTF?

      Re: I just wonder ..

      >>That restore should have taken a day, and that's being very, very generous - I would have expected hours at most.

      ha ha ha ha ha ha ha ha ha ha ha.... and breathe..... have you ever restored an Exchange server? If you have, with a large server and many users, and it took less than a day please accept a christmas beer my heart felt apologies and my congratulations for managing something many people before you have failed to do.

      If you haven't, then I suggest not pronouncing upon things that you wot not of! Exchange is a prime example of software that claims to JustWorktm and really doesn't when it comes to restoring the mail database.

      1. usbac Silver badge

        Re: I just wonder ..

        While I agree that Exchange is a total nightmare, if you have the right backup software, it can be done without too much effort.

        Back when we had an on-premise Exchange server, we had Exchange specific add-ons for our backup system. I had the ability to restore anything from the complete information store, to an individual email. And, yes I tested it regularly.

        I often had to restore email folders that people accidentally deleted (most users were set to clear deleted items on Outlook close - storage issues). I restored full mailboxes a couple of times where people had left the company years ago, and suddenly some exec needed to see the mail in the box.

        I did a test restore of the full information store to a new Exchange instance a few times, just to make sure it was possible.

        1. cb7

          Re: I just wonder ..

          "if you have the right backup software, it can be done without too much effort"

          Which software did you use?

      2. Anonymous Coward
        Anonymous Coward

        Re: I just wonder ..

        If you have to come back from a cold/offline backup, sure, but this is what snapshotting was invented for.

        That said, otherwise you're right, it's a sh*t product to start with. Truly 100% Microsoft compatible.

      3. The Basis of everything is...
        Pint

        Re: I just wonder ..

        Exchange Backup for small/home business

        Bacula triggers script to call Exchange backup to dump the database to a file, backups up that file, and them sweeps it to an archive area and delete yesterdays archive. Nothing fancy beyond a bit of config and a bash script. Using free Bacula edition too. Annual VM export to speed up a full rebuild should the worst ever happen.

        Exchange Restore for small/home business

        Upload VM export in the event of total loss. Restore backup file if that's been lost. Use Exchange tools to restore what you need from that. Win.

        Only time I had to do it, it took most of a day to get some lost mails back - and that was mostly googling for the clue on how to do it for real. Not difficult, but not exactly scalable either. And not something for the non technically minded to attempt so hardly suitable for most small businesses to attempt to do in-house.

        Bootnote

        Now the whole lot has been migrated to postfix & dovecot for mail and Nextcloud for shared xDAV calendar & contacts - and the other goodies it can do which I'm still playing with. Mail is easy to restore (at least under test conditions) the calendar & contacts will be a problematic if somebody accidentally deletes something - still not something for the non technically minded to attempt. And yes, getting postfix to work was an interesting learning experience...

      4. ecofeco Silver badge

        Re: I just wonder ..

        Exchange truly is a giant piece of shit. There is no polite way to put it.

      5. Lil Endian Silver badge
        WTF?

        Re: I just wonder ..

        So don't use (support, or promote) stuff that's not fit for purpose, inclusive restorative procedures.

        I deploy Postfix/Dovecot. I restore with no problems, as a matter of course, to ensure back ups have worked. WAI all the way.

        So, allow me:

        ha ha ha ha ha ha ha ha ha ha ha.... and laugh

    3. schmitzr2018

      Re: I just wonder ..

      You have obviously never had to keep an Exchange environment running, secure from malware, phishing and up-to-date 24/7/365. To suggest bringing it in house for a small non IT focused company is sheer idiocy. Move it to O365 or Googles business product. MS has a team of 10k people to keep it running. Also make your own cloud backups, you control, separate from MS. Oh and Postfix is a pain in the #ss.

      1. Lil Endian Silver badge
        Pint

        Re: I just wonder ..

        Postfix is a pain in the #ss.

        I've never encountered a problem with Postfix. I'm interested to hear your position, maybe I've missed something and I'd appreciate your views.

        [This is nothing to do with fanboi, appropriate solution for objective is all. -->]

    4. usbac Silver badge

      Re: I just wonder ..

      At my previous job, one of my last projects to finish before I left was to complete the migration away from Rackspace. I closed out the last business we had with them about a month before I left. They were really great at one point, but started rolling downhill fast.

      We noticed it when all of the really good support people were leaving one by one. We had senior support people that we worked with regularly, and every few weeks, we would call in, just to find out that the person we were asking for was gone. Then, they completely off-shored all of their support. That is when we really started moving services fast.

    5. An_Old_Dog Silver badge

      "That restore should have taken a day"

      I'm not defending Rackspace, but what basis do you have for claiming the restore "should have taken a day"?! We had a fileshare server whose differential backups completed handily overnight, but a bare-metal restore from tapes -- we had some kick-ass, auto-loader tape library boxes connected (each) via two 1GB/sec Ethernet links (1990s) -- took three days!

      1. Anonymous Coward
        Anonymous Coward

        Re: "That restore should have taken a day"

        Ah, but is that not exactly where the questions start? What massive disaster has taken place there that they had to restore from offline, how can customers be sure it won't repeat itself and have they actually ever tested a restore from scratch?

  7. Anonymous Coward
    Anonymous Coward

    "isn't the expectation of outsourcing that someone else takes care of tiny details"

    Err - no?

    It's the C-Suite showing "savings" so they can justify their massive pay-rises. The MSP will do the barest minimum they can for that price. Looks good on paper until the shite hits the fan....

    1. The Basis of everything is...

      In that case you've picked the wrong plan or the wrong MSP. Or both.

      Outsourcing done well can save you a whole lot of problems and money, and most techies will go a above and beyond, especially if treated with courtesy and respect, whenever something interesting occurs.

      But you have to make real sure that you have the right contract and you're buying what you need instead of the cheapest you think you can get away with. If your sole aim of outsourcing is to cut costs and to screw every last penny out of what should be a partnership then you are very likely going to end up with regrets.

      1. cb7

        "you are very likely going to end up with regrets"

        Ah yes but the people who architect these deals typically move on before the shit hits the fan, so they definitely don't have any regrets.

        Not unless their past catches up with them and people start to realise the cost "saving" they're so proud of ended up ultimately taking the ship down.

    2. IGotOut Silver badge

      I guess you didn't didn't read the article. It's mainly small and medium businesses that took the hit. The ones, that can't affird a server team, a network team, a comms team, a security team.

      Also the one that can't say, fix this, or your going to lose $50million of business from us, oh and we'll sue your ass into bankruptcy.

  8. Sp1z

    "Rackspace has recovered all of the PST files on the compromised servers"

    Er, I thought this was hosted Exchange? Why are they blathering on about PST files when Exchange stores its mailbox database as EDB files, and not certainly separately for each mailbox?

    I'm glad I bit the bullet years ago and bought the MS Action Pack for £350 as I am happily running my own Exchange 2016 server at home instead of relying on someone else's "cloud" solution (yes I'm still in the license terms - no a linux email server doesn't suit my needs). If I cock it up and don't take backups or don't take security seriously enough, I've only got myself to blame.

    1. neilo

      Re: "Rackspace has recovered all of the PST files on the compromised servers"

      My understanding is that they are doing some sort of automated extraction. It can be done… but it ain’t fast, and if the database is corrupted it won’t be 100% accurate either.

      The contempt Rackspace has shown for customers is unbelievable. Yesterday, for the first time since this dumpster fire was lit, I received an email from Rackspace telling my that pst files may soon be available, but it may only be a partial recovery.

      What. A. Joke.

      Rackspace is dead to me now. I managed to recover > 95% of emails and moved on.

      1. Ken Moorhouse Silver badge

        Re: It can be done… but it ain’t fast,

        That brings back bad memories of...

        Inbox Repair Tool

        No wonder it's taking so long.

        1. ecofeco Silver badge
          Pint

          Re: It can be done… but it ain’t fast,

          You had to remind. You just HAD to remind me.

          That's to help me forget again. --------------------------------->>>>

        2. Ken Moorhouse Silver badge

          Re: Inbox Repair Tool

          If you are a Rackspace Engineer: I hope it is by now on the 20th Pass, Phase 24.

          To everyone, including the above: Have a Happy Christmas!

          Not at all happy for those in a war zone though, but we're thinking of you. Let's hope 2023 somehow brings world peace.

          1. Lil Endian Silver badge
            Happy

            Re: Inbox Repair Tool

            Considerate sentiments Ken.

            Sir, to you and all, peace, happiness and friendship for today and the future.

      2. mike_coreit

        Re: "Rackspace has recovered all of the PST files on the compromised servers"

        I'm still waiting for Rackspace to fix the PST files as the ones they provide don't import properly into Outlook

  9. TiredNConfused80

    But not quite yet.

    About 4:59pm on the 24th December is looking good at the moment...

  10. HankScorpio

    "When asked for an exact customer count, "It's 1 percent of our overall company revenue," Rackspace Chief Product Officer Josh Prewitt told The Register..."

    This kind of crap drives me nuts. He was asked for the number of impacted customers, even if he can't give an exact number he could give an estimate, but instead he chooses to focus on the only thing that matters to them, revenue, and in doing so makes it all about the company rather than the affected customers.

    I'm so sick of this kind of attitude from businesses and governments who think everybody is obsessed with revenue / growth / share price etc, and seem to forget that without the customers they wouldn't have any revenue in the first place

    Typical selfish, greedy minded response from a C-Suite and tells you all you need to know about the companies focus and priorities.

    Or in modern parlance..."Tell me you're obsessed with revenue without telling me you're obsessed with revenue"

    /Rant

    1. wi94e&*L2Xm?

      Companies don’t exist to provide you services. Companies exist to make their owners and shareholders money. It is you who needs to rethink your position here.

  11. Anonymous Coward
    Anonymous Coward

    What an rsole

    Let's say their exchange is competitive with office357 or whatever their uptime is.

    Exchange mailbox plan 2 100GB sounds good for a small or midsize business and costs £6 a month or £72 a year, roughly $85.

    That could mean their 1% or $30M in yearly revenue would be as many as 352,941 exchange mailboxes and the CEO doesn't give a shit about any of these clients.

    Mmm, great company, think I'll look elsewhere.

    1. katrinab Silver badge
      Meh

      Re: What an rsole

      They charge £8.79 per user per month.

      https://www.rackspace.com/en-gb/applications/microsoft-exchange

  12. ecofeco Silver badge

    I never get tired of saying it

    So, how's that cloud thing working for ya?

  13. morsey

    Pffft serves them right. About 10 years ago I filled out a survey with them on the promise of a free t'shirt with the 'fanatical' slogan on it.

    Still hasn't arrived... although I have moved twice since then. Still, no excuse!

  14. Piro Silver badge

    3 weeks of downtime is unreal

    How do they not just restore that shit from backup, patch and change admin passwords and continue?

  15. Evilgoat76

    Well that's changed the holiday plans..

    We do a lot of our big server downtime over the hols. This year was a big one. Migrate the UK servers from the steaming mess that has become of Memset/IOMart to Rackspace.

    Looks like a quiet holiday period for me now :)

    It's rare that you see such utter unashamed contempt for your customers, and here we are saying the ones this affects the most don't matter to us. So that's ok then. Well you've lost our contracts now. There's a difference between mild ineptitude doing a major job, rolling it back and saying sorry, and actual arrogance to cover up the fact YOU screwed up and don't seem to be able to fix it.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like