Microsoft cops to 775% Azure surge, quotas on resources and 'significant new capacity' coming ASAP

Microsoft has revealed "a 775 per cent increase of our cloud services in regions that have enforced social distancing or shelter in place orders" and is "expediting the addition of significant new capacity that will be available in the weeks ahead", but has already imposed some quotas to cope with huge demand for its cloud. …

[Pascal Monett]

“We are expediting the addition of significant new capacity"

While your workers are supposed to be at home in social isolation ? I doubt that mounting a new server in a rack can be done by a click of the mouse, so that means you have people that are not isolated doing the job.

I hope that that means you are adding this significant capacity in countries that are not under the pandemic panic at this time.

[robidy]

Re: “We are expediting the addition of significant new capacity"

Here in the UK essential workers and those who can't work at home are entitled to attend their place of work, not sure on your country at present but this is true of many.

Prehaps a little less panic and a little more thinking things through, will make life just that little bit better and help us through this challenge.

[Captain Scarlet]

Re: “We are expediting the addition of significant new capacity"

Telecomunnications looks to be classed as Utility Workers, so essential.

[Captain Scarlet]

Re: “We are expediting the addition of significant new capacity"

"Telecommunications" not "Telecomunnications"

D'OH!

[Anonymous Coward]

Re: “We are expediting the addition of significant new capacity"

The response to the pandemic is, on the whole, reasonable, and in some areas insufficient.

Most companies, including Microsoft, are doing what they can given the rules in the areas where they operate.

Trolling them is not called for.

[Craig 2]

Re: “We are expediting the addition of significant new capacity"

"Trolling them is not called for."

Now there's a genuine first for Microsoft!

[IGotOut]

Re: “We are expediting the addition of significant new capacity"

I've mounted plenty of servers in racks on my own, in huge empty data centres.

But being a keyboard warrior, I doubt you have even seen the inside of a proper DC.

[Anonymous Coward]

Re: “We are expediting the addition of significant new capacity"

Data Infrastructure workers are classified as key workers in the UK and I would presume other territories.

If you think things are chaotic now, try and imagine what it would be like if internet connectivity, streaming, government and health websites, online games, etc. were to rapidly go offline.

[ElectricPics]

Re: “We are expediting the addition of significant new capacity"

If every worker was at home in isolation we'd be in a true apocalypse. Who do you think is maintaining the infrastructure that allowed you to post such drivel?

[Anonymous Coward]

It's interesting to compare Microsoft with Amazon. While Teams wobbles under the increased load, amazon.co.uk is as snappy as ever, despite almost certainly seeing a huge increase in demand.

Amazon eats its own cloud dogfood. Indeed, the AWS cloud was originally built internally to support the shopping platform. It also has a healthy market in spot capacity and reselling of reserved instances.

[Anonymous Coward]

I wonder if ms are signing up for some aws buckets under a psudonym?

[werdsmith]

I don't think Amazon Workspaces or whatever they use is anywhere near as popular as Teams for remote working/collaboration.

[Anonymous Coward]

Possibly not but zoom runs on AWS as does Netflix, as does a whole bunch of banks who are all seeing a massive spike in usage right now.

[robidy]

Yes, I also find it interesting comparing apples and tomatoes, both fruit, one more seen as a salad item the other more dessert based.

[doublelayer]

Compare Microsoft via Teams with Amazon? I'd never thought about that before, but sure, let's do it.

Teams: Used primarily during one point of the day, namely local business hours.

Amazon: Used throughout the day.

Teams: Now relied on by whoever chose it for business meetings, meaning many new customers and many old customers using it a lot more.

Amazon: People are buying more things, but they aren't adding users hand over foot.

Teams: Once a user starts to use it, they continue until this quarantine period ends.

Amazon: Once a user makes a purchase, they stop using it unless they're planning a new purchase. This implies a gap between each interaction.

Teams: Remains in the background sending information and maintaining connections for hours.

Amazon: Sends web pages. People click when they want new ones.

Teams: Must send high-bandwidth audio and video streams around, live, with low latency, worldwide.

Amazon: Text, static graphics, videos that might be hosted anywhere and are also cacheable.

Teams is slower. Imagine that.

[werdsmith]

Also, my car has hardly been used since the distancing measures, but my kettle has been hammered.

Apparently Amazon has something called "Chime" but I had never heard of it, nor have my colleagues.

So my conclusion is that it is not used very much compared to the ubiquitous Teams. Which probably explains why Teams is busy.

[NightFox]

It's also interesting to compare the NHS with Just Eat. While the NHS wobbles under the increased load, Just Eat is as snappy as ever, despite almost certainly seeing a huge increase in demand.

ergo...?

[Captain Scarlet]

Amazon are likely to have all the hardware they need in stock for anything they aren't keeping aside just in case.

[Anonymous Coward]

Teams and Amazon Shopping are both customer-facing applications which ought to be on elastic cloud infrastructure. Amazon Shopping has managed to scale well to increased demand; Teams less so.

Recent events suggest that Teams might not be built as a properly cloud-native application - if it were, they could just scale it up at the drop of a hat and nobody would have noticed any problem. It could also be that they've tried to do this, and eaten up all their Azure spare capacity in the process.

[IGotOut]

Last time I looked, Amazon shopping doesn't run live video, whiteboaringd, conferencing and a host of other thingsAnd I have noticed a slow down on Amazon.

Add to that the use of CDN's which serve cached content (steaming video, images and text), then non-realtime content produces much less of a load on the core systems.

[Anonymous Coward]

“if it were, they could just scale it up at the drop of a hat and nobody would have noticed any problem”

That is what marketing says. Reality is no cloud app can “just scale” and no one notice. There is a hell of a lot more going on which prevents this. Often they can do a good job, but at some point reality kicks in.

[A Non e-mouse]

Teams sizing issues

I'm not usually one to stick up for MS (I've sworn at them many times over the years) But the commentards here who are saying "Why don't Microsoft just expand capacity - it's just cloud". Why not consider these minor details called "facts":

Firstly, the number of users of Teams has dobuled in less than three months, from 20 million to 44 million users.

Secondly, the amount of use people are making of Teams is going to be much, much higher. I don't have a figure for this, but let's guess and say it's doubled.

So the number of users have doubled and the usage has, in effect, quadrupled. Running a service at this scale isn't going to be just a couple of virtual machines. So quadrupling the capacity in such a short space of time isn't going to be a trivial exercise. You're also going to be needing more hardware and more bandwidth - the cloud has to run "somewhere".

[NetBlackOps]

Re: Teams sizing issues

Having done the work myself, holding up to a 8.75 times usage profile *is* impressive. Much as they tick me off personally, in the business space they have my attention.

[Anonymous Coward]

Re: Teams sizing issues

"Firstly, the number of users of Teams has dobuled in less than three months, from 20 million to 44 million users."

Let's assume Team server requirements are similar to Skype for Business 2019 - a typical dual CPU with a decent amount of memory will support 100,000 users. If this isn't too far from accurate, Teams capacity has increased requirements by 2500-3000 servers. For a cloud service that is likely to consist of 3-5 million servers, this isn't a significant increase and even if the SfB assumption is incorrect by a factor of 100, its still likely to be less than 1% of total capacity. Microsoft's big issue with its cloud service is that many of its popular regions are hosted in third parties where there is limited capacity to grow (<10,000 servers in third parties versus 20k-50k servers in the facilities they own or that are dedicated to Azure) and significant demand.

Does the software or storage scale to meet the requirements? Less likely given the limits that we have seen placed on the services this week.

[The Mole]

Re: Teams sizing issues

That seems at best a rather dubious assumption at best.

Building a system to handle 1 million users isn't going to be the same difficulty as building ten systems that handle 100k users.

With 100k users you can rather trivially hold the state of all those users in memory at all times within that single machine. You could do query operations by linearly searching through that list (e.g. find every body with e in their name) without much noticable lag.

For a world wide resilient solution host may orders of magnitude more users clearly proper optimized distributed algorithms are needed. with realt databases etc. These have very different scaling characteristics and different bottlenecks will be the most important. The bigger the system the bigger coordination/synchronization issues become a bigger cost.

[Anonymous Coward]

Re: Teams sizing issues

And how about if you broke that 1 million users down into 5,000 user servers as VM's?

Like in Teams.

[A Non e-mouse]

Re: Teams sizing issues

a typical dual CPU with a decent amount of memory will support 100,000 users

Doing what? Chat? Yeah, that's probably do-able.

What about some of the other toys MS-Teams has: File sharing/syncing, voice calls, video calls, conferencing, etc.

You're not going to get 100,000 video (Or voice) users on that dual CPU box.

[Anonymous Coward]

Re: Teams sizing issues

At a guess?

20 x 5,000 user VM's per server. Not particularly arduous as long as you have enough memory and IOPs.

Can all users run video calls simultaneously? No. But then it's unlikely to be common anyway as your users will sleep 1/3 of the time and a significant number will be occupied with other tasks for otherparts of the day.

[Anonymous Coward]

Re: Teams sizing issues

School & University terms are ending now. I suspect that in these last couple of weeks institutions were more on a pause. But when they come back after the Easter break, they're going to want to get on with teaching. I've heard a figure of a million extra users for online collaboration tools coming on stream in the next few weeks.

[AMBxx]

Re: Teams sizing issues

They've gone from 22 million users, most of whom just don't know how to disable teams to 44 million users actually doing something.

It's far more than a doubling of active users.

[msroadkill]

afaict, the take home is that MS cloud appeals to home workers more than Amazon?

[Miss Lincolnshire]

Thumb twiddling

Can't migrate anything since Microsoft turned off additional cloud resources.

Busying myself going through all the slide packs that are used to persuade people to move to Cloud and removing all references to "limitless resources" "ramping up and down", "turning machines off when not busy and back on again when you want them"....