So should we now call it Office 364?
Oh, hang on, it's a leap year, isn't it?
Black Monday: Office 365 down and out in Europe
The exact cause of today’s periodic Office 365 blackout remains a mystery some eight hours after the service in EMEA first fell over. Poor unsuspecting souls took to Twitter this morning to vent their frustration at their lack of productivity caused by the cloudy productivity suite’s outage. “We are experiencing a national …
-
-
Monday 22nd February 2016 17:35 GMT Anonymous Coward
With my maths hat on there was a 7 day outage earlier in the year plus this 1 and then taking and into account the leap year that's Office 358.
Anyone care to guess the final number for the end of year? My money is on Office 323. Maybe El Reg should have an official countdown just for fun?
-
-
-
Tuesday 23rd February 2016 12:50 GMT Stoneshop
Re: Don't worry
"five nines is just a recommended uptime level."
MS: "Huh? Oops! We thought it was nine fives!"
Office nine-to-five (in a timezone that does not have this interval overlapping with your working hours).
And it was not only email that was out, you lying Putrid Relation twonks at MS. Unless you define the rest of Office to be email too.
-
Monday 22nd February 2016 22:25 GMT Mark 85
With my maths hat on there was a 7 day outage earlier in the year plus this 1 and then taking and into account the leap year that's Office 358.
Anyone care to guess the final number for the end of year? My money is on Office 323. Maybe El Reg should have an official countdown just for fun?
I'm not a Nigerian prince, but if you'll email me your banking particulars I will make a bet on this in your name with a very honest bookie.
-
Monday 22nd February 2016 19:21 GMT Anonymous Coward
News just in...
MS are increasing synergies and enhancing operational efficiency by aligning their online services across Xbox and enterprise software divisions. Going forward, the new cloud productivity suite will be known as Office 360.
New features include team chat with your co-workers while editing a document, ability to block co-workers who you don't like, a feedback rating if they chuff up your document, and co-worker score allowing you to compare how many documents you've edited and words you have typed etc.
-
-
Monday 22nd February 2016 17:27 GMT The_Idiot
“A limited number of customers"
Sigh. I've blethered on about this before - so my apologies for being even more boring than usual. Of course, everything (with the apparent exception of ISP bandwidth and capacity offers - right until you try to actually use them) is 'limited'. but if you're one of the 'customers' in question, the 'limit' on you is closer to 100% than the marketing-speak attempt to minimilise the issue.
I could go on to, um, go on about the current trend towards online services for productivity tools, but I think I'd either be preaching to the choir or flogging a dead equine, depending on readers' personal preferences.
-
Tuesday 23rd February 2016 06:53 GMT Pascal Monett
Re: “A limited number of customers"
A limited number. Yeah. Limited to all UK users, or half of Europe's users, or whatever.
PR guys : limited does not mean what you think it means. You think it means that you can make us believe that this outage is not that important. We know it's important because it's all over the news.
If you really want us to know how small the issue is, just come out and say "only 12,000 out of <insert millions here> are impacted by this outage". In that case, we will be forced to admit that, indeed, it is not all that many compared to the total connections.
But you won't ever do that, will you ? That would be telling people the actual situation - and you don't ever do that, oh no. That would be an unacceptable break of tradition.
-
-
-
Tuesday 23rd February 2016 14:04 GMT EdFX
Re: A message to all who thought Office 365 was a good idea
I don't think most people actually use it but just like to shout it down. We have used as a business with several hundred users all the way back to when it was in previous incarnation of BPOS, and its been excellent. No issues, always on, no matter whether on road, home or office...and I have used Mac, PC, Laptop, mobile, tablet etc..in fact anywhere in world I have travelled.
I'm not anti or pro MS but...it just works, whats to complain about ? And no we weren't affected by the EU outage at all.
PINT because...overs the years its saved us a huge amount of hassle running Exchange....
-
Tuesday 23rd February 2016 00:31 GMT Anonymous Coward
Re: A message to all who thought Office 365 was a good idea
Microsoft needs to fix this issue, goes without saying, but people always freak out when there is some intermittent outage for a brief period of time in a limited geo on these cloud apps (O365, AWS, whatever). Your local sys admin would have just as much down time running Exchange on prem as any cloud service, likely more time. People have different levels of expectations for cloud vs on prem. If on prem goes down or has a planned outage for maitenance, that is just the way IT works. If a cloud service is ever down, it is front page news and goes to prove that cloud is no good. So definitely needs to be fixed, but put it in perspective.
-
Tuesday 23rd February 2016 03:27 GMT P. Lee
Re: A message to all who thought Office 365 was a good idea
>Your local sys admin would have just as much down time running Exchange on prem as any cloud service, likely more time.
While you are correct, your local sys admin is very much beholden to your local management.
When you scale the customer base up, each individual customer becomes less important and you have no possibility to hire a difference sys admin. Worse, there is a loss of skill in the alternatives to O365... until the price goes up. By then... too late. That's the price of monoculture and monopoly.
What happens when MS has been using email as loss leader, driving alternatives out of the market and then a Lehman Brothers-style accounting fraud bankrupts the company? You can just watch your email cloud waft away, taking your "aas" with it.
At that point, try calculating your savings from cloud.
-
Tuesday 23rd February 2016 17:34 GMT Anonymous Coward
Re: A message to all who thought Office 365 was a good idea
"When you scale the customer base up, each individual customer becomes less important and you have no possibility to hire a difference sys admin. Worse, there is a loss of skill in the alternatives to O365... until the price goes up. By then... too late. That's the price of monoculture and monopoly."
Although when you scale up, each individual customer becomes less important, the overall service uptime becomes far more important. If a cloud service is down, there will be articles like this written everywhere (even for a fairly low impact intermittent outage like this one). The cloud providers cannot screw up, ever, or this article happens. On prem IT has some intermitted fluttering of email for a few hours and there are calls into the helpdesk, IT says we are working on it, no one remembers a week later.
On the "hire another admin point", I think that is actually one of the greatest benefits of cloud. If your on prem IT has some sort of massive screw up, your only recourse is to fire that IT admin... which doesn't really fix the problem or put money back in your pocket. Cloud providers have SLAs, admittedly very low bar SLAs at this point... but they will be forced up by competition as the cloud services mature, which they need to hit or something is owed. There is zero chance that an internal IT department ever cuts the line of business a substantial check or gives them a future discount based upon a screw up.
I don't understand this cloud lock in argument that is throw around.... If you have some issue with Exchange or SQL or Oracle DB or SAP, etc on premise (they start hosing you on support for instance), are you going to just switch to a competitor? Maybe after several years... but still probably not. At worst, I think cloud services put you in the same position you're already in and potentially could make migration a viable option. People have been upset with IBM mainframe costs for decades and no one does anything.
On the what if Microsoft goes bankrupt situation, it seems supremely unlikely with MS (Amazon or some small SaaS provider might be a better hypothetical), but maybe some Lehman situation happens. In this situation, they would transfer the assets to a receiver and continue to run the apps until some unwinding can occur, like selling the cloud business to another company. I think it does make sense to look at the providers more carefully when considering cloud. You have more of an interest in their financial viability. Still though, what if your teleco went bankrupt and everyone walked away tomorrow... that would shut you down. Notice how little sleep you lose over that possibility.
-
-
Tuesday 23rd February 2016 07:28 GMT Ken Moorhouse
Re: A message to all who thought Office 365 was a good idea
Quote:-
Microsoft needs to fix this issue, goes without saying, but people always freak out when there is some intermittent outage for a brief period of time in a limited geo on these cloud apps (O365, AWS, whatever). Your local sys admin would have just as much down time running Exchange on prem as any cloud service, likely more time. People have different levels of expectations for cloud vs on prem. If on prem goes down or has a planned outage for maitenance, that is just the way IT works. If a cloud service is ever down, it is front page news and goes to prove that cloud is no good. So definitely needs to be fixed, but put it in perspective.
Response:-
It is of course certain that premises IT will fail, or need to be maintained, or need to be upgraded.
I think the difference is keeping people in the loop. If a failure occurs then a good IT department will liaise with management to say what the problem is, how it is being dealt with, and when things are likely to be fully working again.
The mouthpiece imparting that news is the IT Department themselves, not a PR representative of the IT Department. The difference is that if the Client has a question to ask regarding such news, that the mouthpiece has the answers - whatever that question is - at his/her fingertips. So if the question is asked "are my messages safe? Nothing's going to be lost?" a sensible answer is given, rather than one which bows to the legal profession for its content.
O365 can not afford to liaise on a per user basis with its customers - economies of scale dictate that a canned response is all you're likely to receive. This doesn't help the situation where management screams at [what's left of] the IT Department with the very same questions mentioned higher up. I have encountered the situation where Management say they want an hourly update on the failure - with an on-premises failure it is easy to give a sincere response, even if it's not the news Management want to hear ("Bob's popped to the shops to get a coffee whilst the restore is happening"). O365 could have a Cortana response which, reading between the lines, is along the lines of "That's alright, say whatever you want if it makes you feel any better." The only way people can vent their frustrations in such circumstances is to take to Twitter, etc. and that is the reason that front page news in made.
-
Tuesday 23rd February 2016 18:08 GMT Anonymous Coward
Re: A message to all who thought Office 365 was a good idea
"I think the difference is keeping people in the loop. If a failure occurs then a good IT department will liaise with management to say what the problem is, how it is being dealt with, and when things are likely to be fully working again. The mouthpiece imparting that news is the IT Department themselves, not a PR representative of the IT Department. The difference is that if the Client has a question to ask regarding such news, that the mouthpiece has the answers - whatever that question is - at his/her fingertips. So if the question is asked "are my messages safe? Nothing's going to be lost?" a sensible answer is given, rather than one which bows to the legal profession for its content."
With all due respect, I have been in many of these crisis situations. In most cases, if something is really screwed up, it is not the IT department doing much of anything other than giving high level support/engineering from Microsoft, IBM, Oracle, EMC, pick your vendor access to systems and logs. If it gets to crisis mode where something is down hard, that means that IT hasn't been able to figure it out. The bottleneck is still the vendor and it is still on them to fix the problem. Often times the executives want to speak directly with IBM, EMC, etc for those hourly updates as IT is just relaying information they heard. Regardless of vendor, there would be some sort of hourly conference bridge open to provide an update. Generally the vendor assigns a resolution owner to be the point person to discuss all of those questions people have.
Now you do run into an issue with cloud as when something goes wrong it isn't one customer impacted but likely many... maybe even thousands. While I'm sure a vendor would be happy to sell you some premium service where you can get someone to explain things, I think the benefit of cloud is that, if there is a wide spread impact, you know that every person at that IT vendor from the CEO down is working on fixing that issue as soon as humanly possible, at any cost... which is why these cloud issues do get fixed quickly (although they really shouldn't happen at all). If your Exchange environment has issues, MS will certainly work with you to fix it but it isn't really a crisis for MS's management and you likely have the attention of one support engineer. It could linger on longer than required due to resources. This is not specific to MS, true of every vendor... just the way vendor support works. If Europe's Exchange environment has issues, you can rest assured that everyone will be working on it and it will be fixed even if it costs millions of dollars/pounds.
-
Tuesday 23rd February 2016 18:34 GMT Ken Moorhouse
Re: With all due respect
I rarely need to get support from vendors of on-prem hardware or software (including email servers). I supply stuff that, if it breaks I can fix without needing outside assistance. There's a lot of people out there selling stuff that they can not support by themselves (and this, by definition, includes The Cloud). This is one reason I do not like The Cloud: regardless of your resourcefulness at fixing On Prem nasties, you are up against a brick wall when trying to get to the root of a Cloud problem.
If the problem is the Broadband then that is a fault common to both Cloud and On-Prem. Ditto DNS issues.
-
Tuesday 23rd February 2016 21:46 GMT Anonymous Coward
Re: With all due respect
Yes, if there is a configuration issue, you can fix it on site... but you cannot fix everything on site. You are playing in the sandbox that Microsoft, IBM, EMC, Cisco, etc have created for admins. If there is a problem with the firmware or source code that is invisible to IT, then the vendor needs to apply a fix. Those generally are the nasty issues which are not quick fixes.
The argument that IT admins can out admin the cloud is asking people to make the bet that whoever they happened to hire as their IT admin can do a better job than mega tech companies with an army of Stanford/MIT/etc PhDs, access to all the source code, virtually unlimited funding to set up a public cloud environment. It is like saying you can build a better car than Mercedes by ordering parts to your garage. It is possible that is true, but not likely.
-
Tuesday 23rd February 2016 22:06 GMT Ken Moorhouse
Re: army of Stanford/MIT/etc PhDs
On the surface you are of course correct in what you say.
But if that were the case in practice, (historical example springs to mind) why did Cogswell and Russinovich feel the need to write utilities that Microsoft's army of coders could easily have written themselves (they having the source-code, after all)? As I understand it, Microsoft could not have had such tools in their tool-chest, otherwise why buy the company that concocted them (Sysinternals)?
My belief is that the "army" needs to have appropriate motivation to achieve things, and quite often there is no motivation to do things right when decisions are made by committee and there is no incentive to do things well without due recognition.
This is where the "small guy" can usurp the "big guy".
-
Thursday 25th February 2016 17:06 GMT Anonymous Coward
Re: army of Stanford/MIT/etc PhDs
We are talking apples and oranges. I can give you a better example than sys utilities. Microsoft did not anticipate the need for a hypervisor (despite all of their Unix competition's primary differentiation being a hypervisor) until VMware came around. Even so, we are talking features/functions vs the base use case. Everyone knows that job number one is keeping an environment up and running. That one hasn't escaped Microsoft. Now that everyone agrees on what needs to be done, who is best suited to make it happen? I think the cloud providers for the reasons stated above. Yes, software vendors, Microsoft or otherwise, often miss functionality or don't anticipate every use case... look at how many companies exist just to supplement SAP functionality... but there is a difference between a company not recognizing the need for a function and the ability to execute well on delivering that functionality or service after it is agreed that something is required. Everyone agrees that uptime is a requirement.
-
-
-
-
Tuesday 23rd February 2016 19:37 GMT Ken Moorhouse
Re: although they really shouldn't happen at all
I am sorry, but this is an impossibility.
You might be able to get 99.999% uptime, but no vendor can promise 99.9999999 (recurring)%. Arguably the more that a vendor aspires to that utopia, the less likely it is.
Why? Because failures don't always look like failures, and because working systems sometimes look like they are behaving abnormally. The consequences of not taking remedial action where needed, and taking remedial action where not needed leads to instability which, with a critically-damped control system is likely to cause an outage. In such circumstances the solution might be to "over-damp" which means a failure does occur because there is a built-in delay before taking remedial action.
-
-
Tuesday 23rd February 2016 19:23 GMT techlogik
Re: A message to all who thought Office 365 was a good idea
Yes, the typical O365 support response goes like this:
"Engineers have confirmed that a recent service update was not successful in updating the localization feature. To mitigate impact, engineers will begin deploying a fix starting this we…"
Or "An environment update had a negative impact on the blah blah..." No real answer, just hot air. Sadly, unless you are an enterprise customer, and willing to spend $30K+ for the support to get a dedicated representative to handle your issues (which, if you only pay for about 60-70 users, or $17K/yr for E3 licensing, how would that even be justified? Instead, you are left to create some online support request where some guy from overseas in Asia returns your call a day or two later and you go around in circles with no response for issues often.
This is the real downside with O365 I've found. As long as it is up/running and just works, don't care. But I ran our internal Exchange server, and still do in a hybrid setup, and I had only 30min downtime during working hours over a typical work year. Patching etc, all done off-hours when staff was not really using the server, and it would bounce pretty quick and people would rarely notice. Smaller operation like us, easy to handle/maintain internal Exchange, but space/backups etc..was all an issue, plus DR recovery/planning led us to O365.
As mentioned, the complete lack of information, channels to gain information of status/issues with the environment is non-existent and lacking severely. That is their biggest fail, amongst breaking little things daily. It is a rarity to log into the portal and not see a yellow "In Extended Recovery" notice for one of the cloud services. They can't seem to stop breaking stuff.
-
-
Tuesday 23rd February 2016 08:42 GMT pikey
Re: A message to all who thought Office 365 was a good idea
"Your local sys admin would have just as much down time running Exchange on prem as any cloud service"
Nope sorry not true. I had our on prem exchange running for the last 4 years (E2010 migration to E2013) since we split as a company and I had only one outage in that time.
And that was down to the data centre done looking at backups if they worked of not. I know keep an eye on it myself and kick them when it does not run.
Having said that I have probably Jinx it now :)
-
Tuesday 23rd February 2016 17:45 GMT Anonymous Coward
Re: A message to all who thought Office 365 was a good idea
"Nope sorry not true. I had our on prem exchange running for the last 4 years (E2010 migration to E2013) since we split as a company and I had only one outage in that time."
I meant it as an average around the globe. I'm sure there are some superstar outliers such as yourself who can beat cloud uptime. :)
-
-
-
-
-
Monday 22nd February 2016 18:13 GMT ratfox
I'm getting bored of these useless outage reports
What I would like is a web site reporting clear metrics, displaying a graph of how much of the time during the past year there was a good service for 90%, 99%, and 99.9% of users.
This would be much more informative than irregular articles stating "there's an issue affecting some users today".
-
Monday 22nd February 2016 18:52 GMT Mephistro
Re: I'm getting bored of these useless outage reports
'...irregular articles stating "there's an issue affecting some users today".'
Microsoft are the only ones with any chance of knowing the number of affected subscribers. I have a hunch that they won't make that data available. Or if they do, it will be total bullshit, as no one else can check their numbers.
-
Tuesday 23rd February 2016 07:27 GMT Anonymous Coward
Re: I'm getting bored of these useless outage reports
What I would like is a web site reporting clear metrics, displaying a graph of how much of the time during the past year there was a good service for 90%, 99%, and 99.9% of users.
LOL - you are asking MICROSOFT to go public with real figures? Actual facts?
They have been manipulating statistics for decades to the point that we started to save any sales presentation because digging out where they had been creative this time was far more interesting than crossword puzzles. They didn't so much as use every trick in the book as adding whole new chapters to it. When it comes to bamboozling an audience, Microsoft refined the art to a point that the most ardent conman can still learn from it. All of that is in my opinion, of course, but I would suggest that you never trust a graph in any MS originated presentation without checking date, source and data point selection, and even then I would second-source the data itself before I'd give it any credibility.
So no, thanks, I don't need statistics from Microsoft. I rather see those from someone independent who I can trust. Not that I will go near their new idea - once they think they have reached saturation the subscription price WILL go up. I've been there before in education - after all, that was what gave Gates his OBE (the ability to extract even more coin from the peasants' education and then return a fraction of it as pretend charity).
-
-
Monday 22nd February 2016 18:26 GMT All names Taken
Dear el Reg ...
That is not knews.
What is news is Boris, Dave and the EU.
So the future seems to be:
(a) minimal existentialism as the Duchy of London with Liz or her offspring having some (great?) titular significance (ask Premier about brand management for an example?)
or
(b) borged into an EU with equality and opportunity for all without armed conflict as a base for going forward?
Those with preference for (a) tend to abhor option (b) no?
Just saying thats all doodz
-
Monday 22nd February 2016 20:20 GMT Philip Teale
Incident Report
I realise its de rigueur to bash cloud providers when they have an outage, but if the recent VS login issue is anything to go by, MS will shortly publish a fairly detailed "warts'n'all" incident report - I for one have learned from their new-found openness and let's face it; we'll all have either written one, or be shortly about to...
-
Monday 22nd February 2016 22:45 GMT Will Godfrey
I must be one of the lucky ones.
I seem to be able to get by with a bog-standard email account, and everything else on my own (regularly backed up) home network.
Yes, yes. I know. I'm not running some huge complex office environment of, oh, say six people.
P.S. I do actually know of such a setup!
-
Tuesday 23rd February 2016 07:05 GMT Christian Berger
It's getting needlessly complex
Any sane person would just get a couple of large computers enable what ever kind of remote access (either remote desktop or ssh) and just run the already working software that way. Have your home directories stored on some file servers you regularly back up so you can switch to standby servers in short amounts of time.
You don't have to go all "cloud" to offer a centralized service. Offering computing services via remote lines is something that works, reliably, since the 1960s. Even Windows, which was one of the last large operating systems to do so, offers multi-user capability with networked terminals since about 2000. Just set up a hardened Windows 2000 box somewhere and let people log into it.
-
Tuesday 23rd February 2016 07:43 GMT Anonymous Coward
Re: It's getting needlessly complex
You don't have to go all "cloud" to offer a centralized service.
Ah, but you're approaching this from a technical angle.
The whole "Cloud" term was an absolute Godsend for any halfwit and politician (but I repeat myself) to start talking about IT as if they knew what they were talking about as it continued the outsourcing "no longer my problem" trend into "no longer my problem and I can sound knowledgeable when I try to convince other idiots".
From a technical perspective, all you have are a couple of stack changes but it's still a bunch of expensive room heaters that get cheaper when bought in volume, managed by people who get cheaper if you take advantage of nationals who have no actual rights in your country and wired up via a network that is by nature not terribly redundant so you can blame that when it all goes titsup.
From a management perspective, you have a black box with a fixed price that seems cheaper, and it's not going to be your problem if outages stop the company because it's someone else's fault and "everyone else is doing it too" so your decision is validated by all the other idiots out there. This is why politicians *love* the Cloud: all the press, none of the responsibilities and nobody can show you up for an incompetent idiot whose aura should have never been allowed near anything technical in the first place.
Damn, I just ran out of monthly rant allowance :).
-
-
-
Wednesday 24th February 2016 14:38 GMT Chika
What could possibly go wrong?
I can remember a certain bobcat asking that question. He's no longer around though.
May I simply state that the whole cloud thing introduces a bunch of points of failure that you are less likely to have to deal with otherwise, mostly linked with Internet connection and central system failure. It's not something that makes it unworkable but before rushing into O364.5 you need to decide whether the risks of using somebody else's machine is worth it.
In the ten years I spent at one employer using various versions of Exchange from version 5.5 onwards, we only had one major outage which was a server failure. It was back up over the course of a weekend on a new server. Of course that was before the days of virtualisation but one serious outage over ten years would seem to be better than what is currently happening with O364.99999.
But YMMV.
-
Biting the hand that feeds IT
