Office 365 enjoys good old-fashioned Thursday wobble as email teeters over in Europe

Office 365 is enjoying a Thursday wobble this morning, at least as far as users in the UK and much of Europe are concerned. Users began noticing that something was amiss when they encountered issues connecting to the company’s cloudy email servers, with the first reports coming in at 9:30 UTC. Office 365 email appears to be …

[Lee D]

Technically, my in-house Exchange has had better uptime than Office 365's Exchange servers.

Same for my AD.

Given that I've had power cuts, people digging through cables destroying leased-line fibre and joining 100kW 3-phase to all the earths in the site with a digger-bucket, etc. not to mention things like server RAID failures, etc. that's quite a poor show on Microsoft's part.

I could understand if it was "the connectivity to the cloud for certain people, but the cloud was up" kind of things, but this is literally "you can't get into your mailbox in any way, shape or form" for large portions of their customers.

[israel_hands]

Yarp.

Our organisation is boned, no e-mail availability across the board, even via OWA. What's really daft is that my Outlook client runs in cached exchange mode and stores everything offline just in case of something like this but it's now crapping out whenever I try to load it so I can't even look at e-mails stored locally because the bloody thing can't connect to the server.

[Szymon Kosecki]

Shut your network adapter down, and try again... worst case - unplug at the back of the pc.

[israel_hands]

Good shout, chap. At least I've managed to pull some files from older e-mails. Thanks for the tip. Have one on me.

[Domquark]

That's the problem, isn't it? Most BOFH's will tell you that on-premises Exchange is far batter than O365 - sorry O364.

I've got a client with an SBS2008 box chugging away. Now 10 years old, it's only had 2 hours downtime (not counting broadband failures), so that beats O364 for the whole of 2019 so far, let alone 10 years.

This situation is the making of the bosses who listened to the salesmen instead of their own IT departments. Karma's a bitch!

[A.P. Veening]

Karma

Karma is a lovely bitch, never have any problems with her ;) But I am glad she is my bitch, she will bite those who deserve it.

[dnicholas]

There are Chinese fireworks with better uptime than O365

[Trixr]

The last on-prem Exchange environment I managed had better uptime over its 7 year lifetime from 2010 ownward vs what O365 has been able to achieve in Australia in one year.

Absolutely no exaggeration. If you have DAGs and multiple sites, maintaining the servers is a doddle, since there's no outage to the mailboxes if you shift them to another DAG member prior to reboots.

In fact, even the 2003 Exchange environment was better in terms of "unexpected outages". I had one Exchange server decide not to deliver mail for a few hours one day - one site affecting about 200 users at the time (5% of the userbase). Patching <1 hr per server per month.

[Robert E A Harvey]

and yet..

Yes, $MEGACORP has also been immune from this, running its own servers on its own hardware. They give barely any trouble.

Yet Microsoft want large corporations to sign up to Microsoft Managed Desktop, which I suspect will include a requirement to use thier azure structures, their AD and their Exchange Server. Something users will push for, as it will mean they can collaborate on Onenote and and end to 3rd party sharing systems like Syncplicity and Unify's Circuit for communications.

Nuts to that. The arbitary and high handed update and product dropping, and the sort of cloud unreliability they have displayed makes MMD a non-starter.

[Anonymous Coward]

Damn

Mine's still working, dammit :-)

[Casca]

Strange, no problems with our Office 365 in Sweden. Can't recall when we had a problem with Office 365 services.

[Rich 11]

Have you checked your email for outage notices?

[Anonymous Coward]

Have you checked your email for outage notices?

Just ping an email to your sysadmin

[Anonymous Coward]

Day one of Office 365 outages in 2019?

Office 364, and counting!

[Pascal Monett]

Re: Day one of Office 365 outages in 2019?

Absolutely that !

It's important to keep track of the official availability number :)

[MyffyW]

Re: Day one of Office 365 outages in 2019?

My money is on 333 days for 2019, but I'm willing to be proven to be a grumpy pessimist.

[a_yank_lurker]

Re: Day one of Office 365 outages in 2019?

More like wide-eyed optimist.

[Twanky]

Re: Day one of Office 365 outages in 2019?

'My money is on 333 days for 2019'

What? You reckon they'll have 333 days downtime? Shirley that's a bit pessimistic?

[}{amis}{]

huh?

I haven't noticed outlook being any worse than usual but in my experience, an ant could jump over the bar set by o365 apps standards of stability.

[beardman]

well...

... your own site has wobbled this morning, dear vulture ;)

[Potemkine!]

Re: well...

your own site has wobbled this morning, dear vulture

Got that one too. Nonetheless, it seems to be more reliable than Office 36X ^^

[A.P. Veening]

Re: well...

"Nonetheless, it seems to be more reliable than Office 35X ^^"

FTFY ;)

[colinb]

Re: well...

yes but no one relies on this site for getting work done..

quite the opposite

edit: ah forgot about the BOFH training material

[FrogsAndChips]

users lacking the usual “checking my email” work avoidance excuse

Well, they do have the "can't work as I can't access email" excuse instead.

[Paul Eagles]

I'd be nice the admin centre actually did contain any mention of this incident. There's loads of grumbling replies to their tweet saying that people aren't seeing that incident and I'm not seeing it either. At present it shows just 1 incident which is related to Teams. There is an advisory against Exchange Online though but it's nothing to do with this issue.

Thankfully we're not impacted as of yet and hopefully it'll stay that way.

[ShadowCopy]

I manage quite a few different subs for various clients. It seems to only show the incident if you're affected, or could potentially be affected. It is only affecting a small number of clients for us.

Today i've seen some with no incident showing, and then some showing the incident yet their email is working fine.

They're saying it's down to a problem with a domain controller, but no update since 12:57

[Paul Eagles]

I did wonder if that was the case, but my own 365 tenant is affected yet the admin centre shows nothing about the incident.

[Paul Eagles]

So my personal tenant doesn't have anything about the incident in the admin centre but email to my peronal domains is getting delayed:

-- 4.3.2 The maximum number of concurrent connections per resource forest has exceeded a limit, closing transmission channel. pod51050ehf.outlook.com. PRX8 [AM5EUR02FT050.eop-EUR02.prod.protection.outlook.com]

[Terje]

Luckily it seems as if my company managed to dodge the bullet this time. Maybe it's just some bored Microsoft techie playing server roulette to get something to do.

[Anonymous Coward]

Ours works

On Office 365 web version -Not so much as a blip. Now if only they could have simple search functions in Outlook rather than having to type From:"[e-mail]" -"josephine Blogs" etc every time I want to find anything.

[typeo]

Re: Ours works

Try selecting a folder and using Advanced Find (CTRL+Shift+F). It's far better and more reliable than the quick search crap.

[Anonymous Coward]

Sadly

Another cloudy day where something goes ‘titsup’...

One would think ‘titsup’ would be a good thing, but alas... English.

[Maelstorm]

And this is why

And this is why the cloud sucks. Yes, it reduces IT costs across the board, but there are a number of flaws with it.

1. Increases the size of the line at the unemployment office with unemployed IT workers.

2. Your data is at the mercy of the cloud vendor.

3. If the cloud vendor has a security breach, unless encrypted, your data can be exposed.

3a. This can have major legal consequences depending on the nature of the data.

3b. Everyone's data will be at risk depending on the nature of the breach, extent, etc....

4. If the cloud vendor suffers an outage, then your data is inaccessible.

5. The cloud vendor can hold your data hostage if you fail to pay for services.

6. (Related to 4 and 5) If your data is inaccessible, then your business may suffer depending on what the data is and what it is used for.

7. Data in the cloud is not private unless encrypted, and can be accessed by third parties without your knowledge (spies, government agents, etc....)

I am amazed at the speed to which people are moving their data to the cloud. The cloud was never a good idea because it introduces a single point of failure for everyone using it, and it exposes your data to inspection by third parties without your knowledge. And in case you don't know, here in the U.S.A, we have the CLOUD Act. https://en.wikipedia.org/wiki/Microsoft_Corp._v._United_States I don't trust the cloud, and for good reason.

[Jay Lenovo]

Re: And this is why

The cloud is largely a surrogate for businesses who want to avoid being parents to their data.

When you allow an unrelated third-party to hold your data in undefined locations under their sole control, I guess you might get flustered at times.

[Trixr]

Re: And this is why

Personally, I think the purported reduction in IT costs across the board needs closer examination as well. Sure, if you have an environment with a lot of Dev/Test and/or a lot of scalable production load, it's great for spinning servers up and down.

For constant enterprise loads, I think the costs look different, especially these days when it's easy to rent tin in a bitbarn near your premises.

Of course, these days, the big vendors are trying to drive customers to their SaaS offerings, with all the drawbacks you describe (but marketing hype the business users fall for).

[Twanky]

Re: And this is why

Our new CIO wanted everything moved to 'the cloud'. I asked why. I was disappointed to be told that he 'didn't want the company to be left behind'. I'd rather hoped for something along the lines of 'to save money' or 'it's more secure' or 'less downtime' - you know: something measurable. Bah!

Icon because that's what I had to do --->

I was made redundant a year later... :)

[A.P. Veening]

Re: And this is why

"I was made redundant a year later... :)"

You should have walked asap, it was a very clear warning your new CIO was just a PHB.

[Twanky]

Re: And this is why

Yes. But... They paid me off and then I retired.

I did keep in touch with former colleagues. Apparently the costs went up and the reliability went down - but to try to be fair any change is likely to introduce more costs before any payback.

[Anonymous Coward]

Re: And this is why

What the cloud does is enable managers to lower staff count and affect budget numbers that way. It changes accounting numbers. It appears not to matter whether it is more or less effective. It's all about how a company can make quarterly profits work. This is a big part how China is kicking the Western Worlds collective ass. Managers have no worry about the long term viability of a company, only quarterly profits.

[hoola]

Re: And this is why

It turns capita into recurrent expenditure. All the fluffy presentation from smart sales executives go in at board level with this promise. This generally makes people happy as smaller payments going out all the time look good on the books. There is next to no interest in whether it is better, cheaper, more expensive or anything else.

Add this to the band-waggon effect where it is perceived that using the cloud is fast, modern and agile.

The bottom line is that the people that do care (old techies who genuinely understand what they are doing, and a dying breed) are perceived to be inflexible and just putting up barriers.

All the rest in the food chain up to the top of the tree are just interested in where the next bonus is coming from.

[TimMaher]

Re: And this is why

You are right about Capita. They are a “recurrent expenditure” and with no apparent value in return.

Should have ‘capitalised’ it though.

<smirk/>

[mr-slappy]

Re: And this is why

8. You can lose control of where the data physically resides. If it's personally-identifiable data relating to citizens of various countries (Singapore, Switzerland etc), and has moved out of those countries. you may well find yourselves in regulatory hot water.

[Hans 1]

I literally could not work, Jira was down (inhouse) all afternoon and email was dead ...

Icon ? well deserved, Pelforth Blonde, a Heineken lager from nothern France ...

[gjmboise]

I'm sure Speaker Pelosi would blame it on President Trump and the US Government shutdown!

[A.P. Veening]

Speaker Pelosi knows better (she is smart enough), but I can believe Trump blaming it on her and her colleague representatives in the DP.

[dnicholas]

It's not even Monday anywhere, come on guys

[dnicholas]

I didn't even notice

Had a few complaints in the AM (UK time) but just told everyone to be patient...

Result

[ahdguy]

Finally Back...

But lost an entire day's worth of customer emails - I'd assumed the emails sent to us would just have been backlogged rather than simply dropped...

[Anonymous Coward]

Generic anti-MSFT rant

I blame Brexit

[streaky]

This..

Keeps on happening with Microsoft over and over and over again. Our corp overlords when we were sold pushed us over from corp gmail to corp off365 for reasons I don't really understand and we've been having issues like this ever since. Firstly it's not in any way difficult to keep an email service up, secondly google have never had this issue. There's something in MSFT's processes somewhere at some level that is simply _wrong_ - perchance Windows is involved somewhere? If it is, get it out.

[Major N]

Its still wobbly again today, emails went down again across our business about 40 minutes ago

[ShadowCopy]

Incident still going, affected clients from yesterday still can't access email. EAC just times out

Current status: Our telemetry data is indicating connection time outs within the Exchange authentication infrastructure, resulting in impact to the service. We're looking into the Domain Controller logs to understand the cause and remediate impact.

[ICPurvis47]

Affected my day

This outage affected my day badly too, and I don't even use O365 or the cloud. I was trying to arrange (and pay for) the delivery and connection of a washing machine, but the insurance company could not work due to not having O365 up, so they referred me to the manufacturer. They were experiencing difficulties also, so they referred me to the warehousing company, who also (you guessed it) could not function, and referred me on down the chain. In all I spent over an hour on the phone to six different companies down the supply chain before eventually being referred back to the original insurers. They took the details and promised to ring me back before end of business day to tell me whether they had managed to arrange the delivery. This they eventually did, so I could pay up and relax. O365's unreliability doesn't only affect its users, it has a wider impact on the end customer, like myself, who wouldn't touch it with a bargepole in my own business' dealings, but was forced in this case to waste over an hour chasing something which should have taken less than five minutes.

[Anonymous Coward]

O365 upgrade

By coincidence, yesterday I received an 'invite' to have my machine upgraded to Office 365 because "loads of you have been telling us how much you want [it]"

[Anonymous Coward]

NHS Scotland is considering O364. Difficult to know whether it will be better than the current shower of shit provision by Accenture.

[Anonymous Coward]

Very reliable...

...I think I’ve had one short outage in the past five years, and I’m using O365 all day long. Love it. There again, I am a user, my job not threatened by the company outsouring to MS rather than in-house.

[RL57a]

Remove all trace

It’s as if Microsoft wanted to remove the bad press from the internet.

As if by magic, EX172491 (which is STILL an issue for us), has magically become EX172564.

[Alfie Noakes]

It is not even the end of January yet!

Down yet again, but seems to be coming back :(