Yes, it's down again: Microsoft's Office 365 takes yet another mid-week tumble, Azure also unwell

Microsoft says it is figuring out why its cloud-based unproductivity suite Office 365 is down yet again for unlucky subscribers. The knackered platform knocked over Teams, Exchange Online, SharePoint Online, OneDrive for Business, and more. The Windows giant has little in the way of details at the moment. The outage started …

[ericsmith881]

For all the companies rushing headlong into putting absolutely anything and everything into the cloud, this *should* serve as cause for alarm. Of course, it won't, but it should. Cloud does not equal infallible. Too many companies are putting all their eggs into one basket. This has always been a bad strategy and it remains so even in the cloud era.

[Imhotep]

I beg to differ. Consider the efficiency in no longer being limited to bringing one company to its knees at a time.

[yoganmahew]

Why kill one SRE error budget, when you can kill budgets all over the world?! Of course, companies don't factor that in to their customer experience when they move to the cloud, nor do they care that they are moving from their own DC, limited complexity so good recoverability times (if they staff it) to a complex cloud (the underlying infrastructure is complex) with poor recoverability times (even if they bother staffing it).

The thrill of it all, is a large corporation moving to Salesforce Cloud, Oracle Cloud (or RE), Office364&Azure for desktop, AWS for workloads, F5& Centurylink for networks, you're beholden to all those actors and more doing their jobs; an endless patch cycle (because everything is at least semi-publilc), an endless cycle of EOL mattering.

You cease to be a company in a business sector for your customers and become an operations company for your own inftastructure.

[Tilda Rice]

yoganmahew, bit too much drama in that.

> companies don't factor that in to their customer experience when they move to the cloud

Really? Effective planning matters in the cloud as much as it would have done for on-prem.

Also, never heard of multi-cloud?

>You cease to be a company in a business sector for your customers and become an operations company for your own inftastructure.

Most companies (at least in my vertical ) spend somewhere around 1%-1.2% on IT operations.

Just part of the cost of being in business, and certainly doesn't mean your primary focus isn't your actual core business.

[Anonymous Coward]

> Really? Effective planning matters in the cloud as much as it would have done for on-prem.

Of course it does, it arguably matters *more* since so much of the infrastructure is outside your control, so you need to allow for more "what if?" contingencies at minimum.

But you're missing (dodging?) an important point: (most) companies don't consider that in their rush to the cloud. They've been told (sold) that "it's easy" and "it never goes down" and "it's someone else's problem" and so on. May as well have said "it's magic!" and have done with it.

Anyone who has done IT duty for a while knows that's pretty much bollocks, but that hardly matters when the big wheel execs and their pet beancounters are swallowing the marketing.

[Gene Jones]

You can bet the sales shtick was "100% uptime."

[N2]

Office 365

Shirley breaking the trade description act ?

must be about Office 355 by now

[Anonymous Coward]

Office (365 - N)

where 0 <= N <= 365

[Alumoi]

Re: Office (365 - N)

Office |365| would be better.

[Anonymous Coward]

Re: Office 365

Like our "Always On" VPN :-)

[sgp]

I can't help the feeling that re-using their old platforms in Office not-really-365 was a mistake.

[JakeMS]

Do you remember?

Do you remember when the term "Cloud Computing" first came about in mainstream media?

Remember how everyone was saying "It's time to move fo the cloud! Never experience downtime again and save tons of money by not needing your own DC and servers!"

How's that working out?

[katrinab]

Re: Do you remember?

25 years ago, it was “thin client”, which seems to mean exactly the same thing.

[jake]

Re: Do you remember?

60 years ago, but it was "service bureau" ... the difference was that back then the provider actually knew what they were doing.

[Locky]

Re: Do you remember?

It almost like all IT projects are either centralisation or decentralisation, and all we do is move the "You are here" arrow between the two...

[Doctor Syntax]

Re: Do you remember?

"which seems to mean exactly the same thing."

That depended on what the thin client was connected to. If it was some external service then, yes, it was the same thing. If it was to your own service, then no.

[Anonymous Coward]

Re: Do you remember?

Spot-on.

It's not so much that cloud is a bad thing in and of itself. It's that it has been so grossly over-sold and over-promoted as a panacea nirvana that cures all ills.

Cloud is fine. Just as virtualization, VLANs, and any other shiny new high-tech IT-related thing all the way back to the mainframe are fine. But none of it, repeat none of it, is the solution for everything.

And all of it, repeat again, all of it, needs looking after from time to time.

[Teiwaz]

so karma does exist.

....ohh, the arrogant presumption of naming it '365'.

Office 'once a week' does have a rather laid back relaxed feel, and might well suit the post-covid world.

[David 132]

Re: so karma does exist.

<Carry-On-Mode>

"Please, Matron, I was once a weak man..."

"Well once a week is enough for any man!"

</Carry-On-Mode>

[fidodogbreath]

Re: so karma does exist.

2020 is a leap year, so they get one day of free downtime.

[sanmigueelbeer]

Has MS tried turning them off-and-on again?

[Teiwaz]

Has MS tried turning them off-and-on again?

Well, appears they got as far as turning it off, but they are really struggling to turn anyone on these days.

[Anonymous Coward]

BEWARE: PC SPEAK - no problems, its only a "subset" of users!

I remember working at a large multinational financial organisation, and they would use the media releases saying "only affecting a small number of users" even when the entire datacentre went black, and it wasn't possible for ANYONE to access the service.

"Affecting" a small subset of users, because everyone else wasn't actually trying to use the service!

[JimboSmith]

Re: BEWARE: PC SPEAK - no problems, its only a "subset" of users!

A few years ago an ex-colleague of mine found his phone down when he tried to use it. He complained to the IT Support phone folks about this at his carrier in the USA. He was told it was dead for everyone and he vowed to switch from the Sidekick immediately to another phone.

[JimboSmith]

Re: BEWARE: PC SPEAK - no problems, its only a "subset" of users!

It put him off the Cloud for life. For those of you too young to remember (I envy you) see here: https://www.theregister.com/2009/10/12/microsoft_t_mobile_danger_server_outage/

[fidodogbreath]

Re: BEWARE: PC SPEAK - no problems, its only a "subset" of users!

99.999% is a subset of 100%...and they can still claim five-nines availability (albeit with a footnote on page 417 of the TOS).

[Kevin McMurtrie]

Cloud uptime

Don't forget how cloud uptime is usually calculated: The cloud is up if services are generally available to most customers. This is how 5 nines of uptime is claimed when individual hosts are more like 9 fives. It's up to you to keep purchasing more redundancy until you've compensated for instance and region outages.

[MatthewSt]

Re: Cloud uptime

That may be how you run your cloud, but Microsoft's SLA is calculated on a per tenant basis: If you can't access the service then you get a refund.

[jake]

Re: Cloud uptime

Let's say I have 100 employees who live in Office354. I'm a skinflint and only pay them an average of $40/hr. With 4 hours of downtime, I'm out $16,000 in employee downtime today ... Is Microsoft going to pay me back for that? Are they fuck ... Three times in three weeks, that's $48,000 and starting to smell like real money.

Open your eyes, people, you are being had!

[N2]

Re: Cloud uptime

Got any jobs then?

I can manage 4 hours of uptime once a week!

[Anonymous Coward]

Re: Cloud uptime

4 hours uptime? Time to call the doctor...

[MatthewSt]

Re: Cloud uptime

If you absolutely rely 100% on something and it costs your business when that something is not available, then buy a service that costs more than $5/month. You get what you pay for.

We're not being had, because if email is down for a few hours we do something else. Everyone has desktop Outlook installed so can still reply to the backlog that they already have, we have documents sync'd to our desktop so can work offline etc.

[sanmigueelbeer]

The Windows titan said the root cause of the downtime was "a change to network infrastructure" that broke something "within Microsoft 365 services."

Let me guess: A "latent defect"?

[Doctor Syntax]

A defect which is active is more than "latent". Another weasel word on which they should be called out.

[jake]

So even if Redmond isn't telling porkies ...

... that's 4 hours times how many users? How many dollars did that cost, world-wide?

And one wonders why the nice things we can't have cost so much ...

[Anonymous Coward]

Make no mistake

Cloud computing as well as outsourcing or managed services, have one major benefit. They allow managers to cover their ass. Imagine if your on-premise Exchange or other mission critical system is down. You will have a long line of managers up to the CEO asking every 5 minutes "how long it will take?", each one terrified of the consequences his actions/inactions might have on company's bottom line or on his career advancement projects. Now look again at the scenario where everything has been pushed out or up in the cloud and it breaks. No worry at all, we're waiting for our provider to come back to us with an update. Add to this the opportunity of firing all sysadmins and an army of techies and you'll soon come to the conclusion that is better to work in these conditions. And that's all that matters.

[Anonymous Coward]

Re: Make no mistake

You call that a benefit?

If I make a decision, I do it in the belief I am trying to make things better for my customers and the people I work with, not to cover my ass. If a PHB is constantly in terror of the consequences of their actions, then I question their ability to fulfil that role in the first place.

[Doctor Syntax]

Re: Make no mistake

OTOH after the first, or at least the first few, outages the ass should be what the manager who made the cloud decision is out on.

[JSIM]

"unlucky" subscribers

Funniest part of the article.

I actually did laugh out loud.

[HalHanson]

Exchange is bad news...

I recall many the overnight lying on the floor next to a client's Exchange server throwing tech-fu commands to "fix" the Exchange store and in the end the M$ support line said the dreaded 'restore from backup' which is bad because important emails would be gone. I really wish I would have not bought into the Small Business Server lie and would have installed Samba and Postfix instead. I does not seem that MicroSlop has changed or beefed up their Exchange code.

[N2]

Re: Exchange is bad news...

Oh, the joys of small bastard server...

[Morat]

Re: Exchange is bad news...

I feel it my duty to shill Zimbra at this point. I was moving from Small Bastard Server and needed a replacement for Exchange so I look at other options.

No, it's not mainstream but it's a hellofalot cheaper than on-prem Exchange and does the same job. Your users even have the choice of native webmail or Outlook.

If you need support, it's actually very good.

Try it free at home if you don't believe me!

Sadly my PHB has decided that we need to embrace O365 so we'll be making the final migration "soon" - or as soon as I don't have anything else more pressing to do.

[grumpy-old-person]

Re: Exchange is bad news...

Commissioning a new head office there were problems with M$ software and people were flown out from the UK.

Quite soon the network (installed and well tested) was blamed for the inability of Exchange to work.

When asked how the M$ people were connecting to the servers the reply was 'network' - simply told the M$ cretin that the network did NOT discriminate traffic and that M$ should extract their digits.

M$ eventually found the problem/s.

[David Roberts]

Reminds me

Of why RAID came about.

No matter how good the individual discs beyond a certain number in use you were bound to get regular failures.

Sounds as though major cloud services are hitting that point now, and don't have the resilience and diversity to survive a glitch in the infrastructure.

Redundant Array of Inexpensive Clouds, anyone?

[Doctor Syntax]

Re: Reminds me

Good idea but you need a supply of freshly made and inexpensive ones to replace those that fail.

[seven of five]

Re: Reminds me

Easy, just virtualise them.

[General Purpose]

Re: Reminds me

Redundant Array of Incessant Nebulisation, now proceeding over south-east England.

[WolfFan]

Re: Reminds me

Isn’t that called ‘normal weather for SE England’?

[Doctor Syntax]

Re: Reminds me

It's certainly normal weather up here in the North.

[Cuddles]

Obligatory xkcd

"DownDetector.com's map showing where people are complaining about Office 365 et al being down"

Population maps - https://xkcd.com/1138/

[Anonymous Coward]

The cloud is down...

Too sunny a day, perhaps?

[Anonymous Coward]

Thanks God no-one is using this for important stuff like Policing... Oh hang on...

Police in England and Wales decided to move to Office 365 almost wholesale a couple of years ago, and whilst informed at the time about why this was a really bad idea in terms of individual and collective Force resilience, chose to ignore that in pursuit of getting everyone on the cloud.

In the past systems may not have been from a single over-arching supplier and that probably did make data sharing harder (though not as hard as is claimed - the systems to securely and safely share data absolutely existed, as did a closed secure and UK sovereign networking capability - now also pretty much close to abandonment in order to pursue 'Cloud').

What you did however have was inter-force mutual aid - which meant if a single Force's systems went down they cold easily rely on their neighbours to take up the slack.

Now that nearly every Force is moving to public cloud M365 (a move without any global precedent, and almost certainly breaking multiple UK laws in the process), when one is down all are down and hence mutual aid fails.

How is this progress?

[Michael Wojcik]

An offline version of Office 365

What, like a version of Office that you run on your own machine, even if it's not connected to the cloud? Inconceivable!

Though personally I'd be happy if they just stopped at "an off version of Office 365". I've been using Word (reluctantly, when forced to do so) since a floppy with a preview of MS Word 1.0 was bound into issues of PC Magazine, and Excel (with loathing) since the mid-1990s. If I never had to use them again it would be a minor but significant improvement in my life.

[Rich 2]

Bizarre

I’m obviously getting old but the whole concept of running a word processor over the internet is bloody bonkers! It’s annoying enough to get half way through typing in a delivery address to sexystuff.com only to have the internet go down (see what I did there?). I’d hate to get half way through writing a book and the same to happen. Not to mention any security nonsense. It’s a bloody stupid idea!!

[vtcodger]

Re: Bizarre

It makes some sense to put collaborative work on the internet -- provided the aggravation of preventing/controlling simultaneous update is less than the aggravation of shipping files around manually. Why someone would pay money to put personal files on someone elses' computer subject to all manner of calamities? Beats me.

(OK, OK, backup to the cloud might make some sense for some use cases. But I bought a 16GB flash drive at the grocery store last week for $8 USD. That works too. Ironically, the drive was purchased to sneakernet some large files via snailmail that were proving to be too difficult to move through dropbox.)

[swm]

Re: Bizarre

Remember time sharing? All users logging in to one mainframe. One mainframe crash and all of the users idleized. This seems like a '20s version of time sharing.

[Doctor Syntax]

Re: Bizarre

That's what we've been saying for a long time. We're due for the invention of a small, relatively self-contained computer that can sit on your desk of just on your knee and only connects to the wider world occasionally. I can't think what we'll call it.

[Morat]

Re: Bizarre

A Chromebook!?

[T. F. M. Reader]

From the article...

Microsoft is planning to deliver an offline-capable version of Office toward the end of next year.

Oh...

[vtcodger]

Oops, sorry

It's clear from the article and comments that many of you think the number 365 has something to do with service availability. Actually, the number is just there because the folks in Accounts Receivable were adamant that their sanity depended on different Office products having clearly distinguishable names.

If you MUST know, 365 is the street address of our regional office in Minot, North Dakota. We're sorry for any confusion this may have caused.

[Rich 2]

Re: Oops, sorry

Well I for one can’t fathom why anyone would make that mistake.i mean, it’s SO OBVIOUS!

[Blackjack]

Wanna WINE about it?

Wine works decently running old versions of Microsoft Office.

That being said I moved to LibreOffice over a decade ago and I am happy with it.

[Anonymous Coward]

if you speak privately to their internal supports staff....

you will find out there is all sorts of shit going down and that they are battling to keep the whole system up...

these "outages" are basically when the system manages to get an outage that effects the public badly...

Most of it they are palming off , saying it is individual installs...... we had a 4 day outage, that eventually led to top level staff contacting us

Insisting we tell them, how "WE" broke THEIR system......... basically trying to coverup their F**up by saying it was our users.

Which i pointed out that if they wanted to stick with that narrative ,

I would be contacting the press to say how a single user could take down the whole of a 365 installation in MS own data center...

then all of a sudden it becomes "don't be negative.. we need to work together", but they completely backed off with their bullshit.

[Tempest]

10-9 = Where's my InterNet? Azure Screw Up Affects Police in USA, Singapore, etc

Microsoft's IoT product for cops - most easily spotted when the cops are viewing the InterNet on the Panasonic laptops in their cars - runs on AZURE. It supplies various services including facial and fingerprint recognition inter-car communication, etc.

Usually MS blames 'leaves' as in when the satellite signal is obstructed. Always thought British Rail had a patent on leaf interruption.

[Tom Paine]

Round and round we go...

This may be a good time to note that Microsoft is planning to deliver an offline-capable version of Office toward the end of next year.

It's with astonishing innovation like this that Microsoft keeps driving technology forward to hithertoo undreamed-of heights. Next they'll be planning an AD that is physically located **in your actual offices!** Imagine how cool that will be!

[jake]

Re: Round and round we go...

"Next they'll be planning an AD that is physically located **in your actual offices!** Imagine how cool that will be!"

It'll never happen ... Everybody knows there is a world market for maybe five data centers large enough for that kind of thing.

[Jboy74]

To many issues with Azure, AWS is the way forward .