Rackspace confirms ransomware attack behind days-long email meltdown

Rackspace has admitted a ransomware infection was to blame for the days-long email outage that disrupted services for customers.  The security snafu took down some of Rackspace's hosted Microsoft Exchange services on Friday afternoon. In its most recent update, posted at 0826 Eastern Time on Tuesday, Rackspace said it has now …

[Anonymous Coward]

Aim for the moon, fall into the Black Hole

July 21, 2021 - "Multicloud solutions provider Rackspace Technology has announced plans to lay off one in 10 of its workers, with 85% of these roles being replaced through offshore service centres. ... At this point, Rackspace layoffs have become an almost annual event, with the company shedding between 100 and 200 workers each year since 2017. It laid off 275 employees – representing nearly 6% of its workforce – in 2017 after being acquired by Apollo. This new layoff is much larger and may have more significant implications for the company. Initial estimations suggest Rackspace will cut about 700 jobs at its headquarters in San Antonio, according to Texas Public Radio. Rackspace’s UK workers will seemingly fair slightly better, according to The Register’s sources, with fewer than 100 employees being made redundant."

Rackspace is owned by Apollo Global Management for £3.3 billion in 2016, to whom it was sold in 2016.

"Apollo is a high-growth alternative asset manager. Our asset management business provides companies with innovative capital solutions and support to fund their growth and build stronger businesses. Across all parts of our business, we invest alongside our clients and take a responsible, knowledgeable approach to drive positive outcomes. ... we are dedicated to a more inclusive and sustainable economy."

[Jellied Eel]

Re: Aim for the moon, fall into the Black Hole

Ah, owned by the spawn of Drexel. So the 'innovative capital solutions' are usually loading the company with debt and sucking any cash out. Then 'building stronger businesses' usually means ruthlessly cutting costs, or funding stops. So skilled/experienced staff get managed out, and replaced by off-shored 'service' centres staffed by low-cost drones who've just passed an MS boot camp and paper mill.

And then this kind of event happens, and we IT folks get a sense of deja vu. Customers get screwed, but they went for the low cost option, right?

[Anonymous Coward]

Re: Aim for the moon, fall into the Black Hole

It's the same model adopted by energy billing parasite companies in the UK. During the good times, you run a shoe-string operation and price your products just above cost. Then as soon as the bad times come along, you declare bankruptcy and then do the same thing all over again with a shiny new brand.

[Paul Hovnanian]

Re: Aim for the moon, fall into the Black Hole

"Rackspace is owned by Apollo"

Are we certain it isn't the Kerbal Space Program?

[M.V. Lipvig]

So...

How's that hosting critical data on other people's equipment working out for you? Just think, you're paying all that money for new points of attack that are not only not under control, but you have no idea what goes on in Cloud City.

[Dr Who]

Re: So...

As many have said before, that's all very well if you have an in house IT team and your own geo-redundant hardware infrastructure. Reading various articles about this disaster, most of the hosted Exchange customers are small businesses with 20 or 30 users. They haven't got a cat's chance of running their own mail systems (especially Exchange based). They have no choice but to trust someone else.

We've used Rackspace, amongst others, for dedicated servers and VMs (not email) for a couple of decades and they really were fanatical and technically excellent with their support and services back in the day. Recently we've been steadily reducing what we have with them. The aforementioned job cuts and service centre offshoring have reduced Rackspace to a budget operation of the 1&1 (now Ionos) ilk.

For the average small business, it's very hard to know who to trust with their mission critical stuff. They don't even know what questions to ask of a supplier, let alone what the right answers would be.

[Pete B]

Re: So...

"They don't even know what questions to ask of a supplier, let alone what the right answers would be."

Until senior management/ business owners realise that IT is just as important to their business (not just a cost center) as the beancounters and needs to be resourced appropriately they will continue to have this problem. Few people would try and run their business without any accountants so why do they try and do without IT?

[Dr Who]

Re: So...

Most small and many medium sized businesses employ service providers for things like accountancy, legal and payroll/HR. They couldn't possibly do it in house, so the problem is identical. You need to find someone you can trust, and until fairly recently Rackspace had a good record. There's nothing to say that the accountancy practice you use won't go bust, or mess up - in fact they often do.

[Yet Another Anonymous coward]

Re: So...

That's why the main suspects for this attack have to be amazon or Microsoft.

After all you can't get fired for hosting your Microsoft email at Microsoft - just like you can't be blamed for getting Arthur Anderson to audit your accounts

[rg287]

Re: So...

Until senior management/ business owners realise that IT is just as important to their business (not just a cost center) as the beancounters and needs to be resourced appropriately they will continue to have this problem. Few people would try and run their business without any accountants so why do they try and do without IT?

Some of them probably did. And their Microsoft-Partner/Certified MSP sold them some managed email using Rackspace Hosted Exchange...

If you're a small company of 20-30 who can't afford internal IT, then you'll probably take your MSP's advice (just as you would your accountant's). These days, your MSP will probably stuff you in Google Workspace, M365 or resell a hosted solution like this.

[Yet Another Anonymous coward]

Re: So...

>Until senior management/ business owners realise that IT is just as important to their business

that's why we build our own computers in-house. It's vital to the operation of our Tea Shops

[Anonymous Coward]

Re: So...

Nonsense. What do you think small businesses did before the cloud and hosted exchange was available?

They bought a server, plugged it into their network and hired an IT guy to manage it.

[katrinab]

Re: So...

Mostly they had a hosted pop3 mail service from whoever supplied their website/domain hosting. Or the really small ones used their ISP's mail service.

[rg287]

Re: So...

For the average small business, it's very hard to know who to trust with their mission critical stuff. They don't even know what questions to ask of a supplier, let alone what the right answers would be.

Which is why Google Workspace is so popular in SMEs. Sign up, give them your card details and away you go - through one (count them Microsoft! One!) management portal.

As much as I try to avoid the Googleplex in everything I do on moral and ethical grounds, I have to hand it to them that Workspace does make business email and document sharing extremely straightforward (with protections for auditing and over-sharing!). Just a shame it scrapes and spies on everything it touches.

I know a couple of small (2-5person) companies that used to self-host their website and email (cpanel on a rented server, usually with no backups or redundancy). Inevitably they all ran into having mail delivery issues because they hadn't fettled SPF/DKIM or someone on an adjacent subnet at the datacentre was sending spam and their IP had got blacklisted with a thousand other servers. Or Google/MS had just decided to be awkward.

Migrating half a dozen users to GSuite was a morning's work - set and forget. Some went to O365, but Google's pricing was marginally better (not sure now) and Google's onboarding really laid out the red carpet and held the hand of less technical users.

[A2Wx8]

Re: So...

That's why we eventually went Google Workspace for our email. With everyone at home our old legacy on-prem email kind of fell over during the quarantine as our email volume went up about six hundred percent. We were leaning that way and decided to make the jump and, well, it just works, spam protection is great, it has a malware sandbox for attachments, they make SPF and DKIM dead simple, and nearly everyone's familar with it from their personal Google accounts. I have enough other stuff on my plate that I don't miss the rote maintenance of the aging on-prem system that even the vendor kind of gave up on.

I am coming at it from a somewhat fortunate position, though, as I'm in the EDU space and my state has a data protection contract with Google to stop a lot of the mining (or at least the obvious kind).

[vtcodger]

Re: So...

"So...

How's that hosting critical data on other people's equipment working out for you?"

I'm no fan of the Cloud (Or the Fog -- which I think might be a more apt description). But neither do I think that the universal solution to the internet security problem is "host everything yourself". Especially not for the large number of very small companies that need a digital connection to the world in order to function. The job of securing a data connection to the entire universe is likely to be too complex a small IT operation to handle. I do agree that larger operations with functioning IT should think long and hard before outsourcing their IT operation to some some bunch of smooth talking grifters who are probably fronting for a questionably competent gang of coders 13 time zones deep in the developing world. What comes to mind is a Stan Rogers lyric Smiling bastards lying to you everywhere you turn

I think that the underlying problem is basically that digital attack surfaces in general have become way too large to reliably secure. That and the fact that there's little or no money to be made from reducing attack surfaces. At least not today. And probably not tomorrow. Most likely not until the security situation becomes so dire that any solution -- no matter how unpleasant -- looks better than business as usual. I reckon that might take another decade.

[Pete B]

Re: So...

"I do agree that larger operations with functioning IT should think long and hard before outsourcing their IT operation to some some bunch of smooth talking grifters who are probably fronting for a questionably competent gang of coders 13 time zones deep in the developing world"

Ah - so you're back talking about Microsoft ;-)

[Kevin McMurtrie]

Bad response

Taking so long to reply is going to drive away even customers that weren't impacted.

9n the flip side, DigitalOcean would have left them all running with the infections.

[Anonymous Coward]

Fanatical Support

[Ali Dodd]

Farcical support more like

[Phil O'Sophical]

These costs will not be passed on to Rackspace customers, according to the spokesperson.

If it affects Rackspace's bottom line then of course it will be passed on to customers, one way or another.

[Mike 137]

The important question

I wonder whether they'll ever let us know how the ransomware got in in the first place. That's the real crux from the security perspective.

[Paul Hovnanian]

Is this more of an attack ...

... on Exchange than on Rackspace?

Had all of these Exchange servers been sitting on customers' premises, would the attacks still be possible? It would be a matter of attacking systems hosted on different subnets, behind a wider variety of firewalls. But as these have to interoperate between different enterprises, they are already exposed to the Internet. Or how would Company A send a messsge to Company B?

[neilo]

Failure

On every level, Rackspace has failed its customers here:

1. This attack was allowed to occur, either by negligence over patching or some other mechanism

2. They took far too long to let customers know what was happening.

3. Every "suggestion" they made was less than useful

4. This forced-push to Microsoft 365 hosting says that Hosted Exchange, as we know it, is gone - but Rackspace are not acknowledging this

5. The "let us know an external email address and we'll forward email" smacks of desperation

6. This is the 6th day of the outage, and still no estimation for restoration of Hosted Exchange from backups so archival email can be retrieved.

They have posted a Q&A update that basically restates everything know and doesn't give us any more information.

How Rackspace emerge from this with any sort of positive reputation is beyond me.

[Claptrap314]

Re: Failure

"How Rackspace emerge from this with any sort of positive reputation is beyond me."

You've not been paying attention then. They take their customer's security, seriously.

[Phil Kingston]

lawyers suing already?

Geez, that's not even ambulance-chasing, that's trying to get a guy to sign-up while he's still being stabbed.

[Paul Hovnanian]

Improper use of commas.

The panda eats, shoots, and leaves.