Global network outage hits Microsoft: Azure, Teams, Outlook all down Microsoft is currently rolling back a network change across its wide area network that it believes toppled over a raft of its cloud services this morning, perhaps in solidarity with the company's tumbling profits. Users all over the world are venting their anger on social media at not being able to send or receive email, …
Don't be immature there is absolutely nothing wrong with Microsoft 365, it's reliable and our staff depend upon it to help facilitate meetings talking about their pets. If it wasn't for M364 I'm sure the bosses would have more of the staff back in the office. The only way we got M363 is due to COVID but at this rate the business case for M360 is looking weaker each year.
Careful now! If the counter ever reaches 360, the shambling zombie corpse of Yahoo 360° will emerge from the depths…
(No, I had never heard of it either, until I once stumbled across a Wikpedia page about it)
Google apps on Mac or even Chromebook would probably be best. Apple office apps aren’t man enough, and Office 365 on Mac has some features missing. 365 on iPad has so many features missing they should call it Office Lite. Macs probably have the longest software support, though there’s some oddities; 2017 MacBook Air gone out of support a year later than a 2012 MacBook Pro.
Not in a month of Sundays. Microsoft is already sniffing around trying to see if it can get a spy global operation up like Google (remember the "we will scan for your Office 365 copies using the next security update " thing?), but it's merely doing what it always does: trying to imitate another company which is already making a mint doing exactly that, and that is Google.
Here is a hint: read the Terms you have to agree to. Plan for a lot of coffee, and make some time for it, don't try to do it all in one go (they're designed to tire you out before you get to the dodgy bits, which is admittedly standard contract practice but MS, Facebook and Google have elevated that to almost an art form).
Once you have read them and actually understand what you've agreed to you may never touch Google again, ever.
Back to MS: the good thing about Microsoft trying to imitate another company's services is that it invariably cocks it up, badly, and that effort then joing a long line of other attempts to broaden their revenue stream.
They have since long given up being innovative themselves. The only innovation MS has really come up with over the years is the various methods by which they lock in and then fleece their victims customers..
I was called by a colleague this morning to say that email, teams etc. were down. I pointed out that I was on my day off and actually didn’t care. They said it was Microsoft that had cocked things up to which I said so I can’t do anything about it. Let IT support worry about it, I said that MSFT would doubtless fix it at some point and I was going back to sleep.
We have some larger client on Google Workspace, and over the years Workspace has been a lot more resilient than MS365 (which our other clients are stuck with).
At the end of the day, it depends on what your company wants. MS365 has some great looking apps and a lot of features Workspace doesn't have, but the backend behind MS365 seems to be put together with toilet roll cores and chewing gum. So if you're dependent on Microsoft apps then MS365 is the only option as MS is killing all standalone versions that are still existing one by one. However, if the focus on just having a reliable messaging backend then Workspace is a good option, and even the often derided Google apps have long reached maturity for serious work. Plus, the Workspace admin settings aren't all spread across a number of completely different and constantly changing user interfaces (which all are at their own semi-constant rate of disfunction) like MS365 is.
As far as Apple is concerned, iCloud uses lots of Azure and reliability can be hit and miss (although it's still better than Microsoft's own services). Still, it's essentially a consumer grade service with limited capabilities in a business environment. Even for a very small shop (couple of users) I'd rather invest in Google Workspace than iCloud.
I tend to think of cloud services as a form of ransomware - you have to keep paying to have access to your data, and there's no guarantee that it is any more secure than it would have been when held locally. The only thing it seems to be good for is the profits of the cloud companies. I seem to remember when Bill Gates was in charge, one of his dreams was to have a recurring monthly income stream.
Yes but no good if you can't access anything.
I couldn't access anything (Windows Laptop and Android handset) thanks to the global corp using Microsoft Authenticator, not mentioned but this wasn't working either.
Lovely morning, because I couldn't auth with Microsoft MFA, my VPN would connect but refuse to auth further, I couldn't access the internet because the Proxy software used couldn't authenticate me.
Also as per usual the Microsoft status website showed no issues until after 8AM, couldn't get through to helldesk so I was very confused (Although that is becoming more normal these days).
Yet again a morning of whinging at yet another single point of failure.
While in my private life I stay clear of cloud stuff, I cannot not see the vast advantage for businesses to have everything in the cloud. Namely: every business in one, single cloud. When The Cloud's offline, every business is offline so no one is bothered that their counterparts are not responsive.
"Critical Business activities are impacted majorly this morning."
Yeah, that's the white streak of fear that I made damned sure Manglement knew was a possibility when we migrated to Office365 several years ago. That we could come in one day, and Microsoft could have blown itself out of the water during the night, and nothing would work for the entire day. It was felt that the possibility of that was more than offset by us not having to manage (or pay for) an on-prem mail system, so we went full speed ahead. But yeah, Microsoft soiling itself was, and remains, a major menace to the system they're selling.
"CEO Satya Nadella ascribed this to customers "optimizing" their cloud spend, and predicted that growth would resume once Azure can deliver the AI customers want"
The only AI I "want" is one that tells the Microsoft techs "If you push out this config change, you will likely break all Office365 operations across all Microsoft domains. Are you Really Sure you want to do this?". Too bad Microsoft hasn't figured out how to do that yet. They certainly have enough failure event data points to train the system with, though.
'F' the small print in their 100,000 page contracts.
Their ineptitude has cost businesses billions over recent weeks. Get that class action ready folks.
Start with $100B. Companies like MS need to be taught a lesson that while businesses depend on you, if you screw up then you should carry the can.
"...do you know how many days it failed to work?"
Yes but that's not the point of many comments regarding Lotus Notes.
The bloody program might have worked but with all the convoluted ins and outs did you manage to get any work done while struggling to get it to do what you wanted?
Depends on the setup, I managed 98% uptime at my worst, but then it was a few Windows servers and not the infrastructure I wanted (i.e Lotus Domino made it super simple to replicate everything to a backup server or a server dedicated for webmail. All could be controlled through Notes networks, which no-one I spoke to ever knew wtf it was, often setup for dialup and never updated). Also had to deal with fighting for resources on the SAN, in the end I found using the inbuilt NSF compression lowered I/O requirements and the Blade had more than enough CPU resources that actually made it faster to respond. Never did use the de duplication (I did find in testing that was a problem)
I really did like Notes 9 when released, but at the time it was bloated and wasn't Outlook and I can't make people like Notes just because I did.
We were working on a new project and the company had Lotus Notes, our company used Microsoft Exchange with all the trimmings.
And then the ILoveYou virus hit (which gives you some idea how long ago that was, it was even then already rubbish, and this was pre cloud, so on prem).
Our own company: about a 7 day outage (people working over the weekend included).
Company with Lotus Notes: not as much as a twitch during the original wave, and one infection about a week later when a secretary came back from holiday.
The key reason for the difference was because it was way, WAY harder to launch anything executable from Lotus Notes.
I recall another customer from those days who had two Lotus Notes setups. One they had running on a Windows setup, and the other one on Linux. The Linux guy spent most of his days drinking coffee watching the Windows people trying to keep the thing running - his ran just fine :).
I think they were running a pilot.
Given the time this happened I think someone from Microsoft had just been and convinced them to go all Microsoft and some supicious people decided to put that to the test.
And failed spectacularly, as usual when the rubber meets the road.
Before you even consider moving cloudwards, you need to know for many hours the service will be unavailable per year and broken down into planned & planned downtime as well as which geographical region, amongst other dimensions.
It's usually written into SLAs and the figure offered by the cloudy offerers will have to be higher than what you currently get with your on-premises hardware and their support staff.
you need to know for many hours the service will be unavailable per year
And how exactly are you going to get that data when it's all unplanned? Oh, wait, you'll ask Microsoft sales people on the golf course, of course, and naturally you even believe them..
Correct but if you are on-prem, the costs are (mostly) peaks & troughs not a nice monthly bill that looks smaller.
That is actually costs far more is irrelevant, the only thing that matters is the monthly cost looks small.
It is the same for so much that is purchased in recent years. The only figure people look at is the monthly cost, sod how long it is for or if you actually own anything.
Those deriding cloud seem to be missing a rather key issue - downtime happens onsite as well. And many organisations don't have the funds or expertise to maintain the same level of service as the big cloud players do.
Schools, for example. Running an on-prem email server is very difficult, vs "use Office 365". Having a fully resilient system is expensive - vs "use Office 365"...
Is it possible that the impact of a local server(s) failing temporarily has an impact on one business, but when Microsoft fuck up it impacts many businesses, and the total cost is vastly more than a single business issue ?
I wonder how much this costs the UK for the outage as it affects so many people.
But Office(insert number less than 365 here) is not resilient. What is the real cost of local support vs. cloud subscriptions? What is the real cost of downtime to an organisation? What about somewhere big, like the NHS, with Office offline?
When the cloud was first offered as a solution, it was assumed that your data would be available from anywhere at any time and it would be held in multiple locations to make it resilient in terms of access, security and backups. Everything is fine while it works, but one screw-up seems to be able to take it all down at once.
Schools don't exist in isolation, or at least they don't in the UK, they are part of a local authority which are massive organisations, for instance the LA I work for, has something like 25,000 users, so not too sure why any school would have it's own email server. Prior to the cloud their email server would have been in a council computer centre somewhere. And yes there can be local outages, but there is usually a backup server, or traffic can be rerouted, whatever, my point being there are options. With the cloud all that is out of your hands. It's like the difference between driving and flying, at least when you're driving you are in control.
LAs are not "in control" directly of most school IT. It is delegated and not centralised.
Trusts like Somerset, that used to run an email service for them, have abandoned that service - as the cost of running it is significant compared with Office 365 or Google.
The idea that a school or even an LA has the same level of support or capability as, say, Microsoft or Google is hilarious.
Not any more, most school now are part of Academy Trusts, business with huge teams of CEOs, Executives Heads and admin staff all sucking money out of the system. Pretty much everything schools use now is some sort of cloud service or SaaS provided by a "specialist company". For the latter look no further than the biometric stuff that is forced on pupils for the highly dangerous and complex task of paying for a meal. Apparently using a fingerprint means the system can bring up a photo so that they can see if the pupil matches.
That an ID card with a barcode/QR code could do the same thing is irrelevant.
Sorry, this one really pissed me off when my kids were at high school. It was SaaS sold to the trust for a huge subscription that promptly made all the queues 3 or 4 times as long because the stupid finger pint readers were incapable or reliably detecting a finger print.
Academy Trusts all sound fuzzy wuzzy warm and somehow dedicated to doing the best for the schools. They are just businesses taking tax payer's money and putting it into manglement's pockets.
... always did wondered how that would work for identical twins (or triplets, or more).
What would happen if they both (or all three) denied incurring the charge. Both deny responsibility with plausible deniability and no charge to either account? Probably some legalese that says "Tough luck, your bear it, you pay for it".
"That an ID card with a barcode/QR code could do the same thing is irrelevant."
And you want to deny the child lunch because they left their card at home/had it nicked by the school bully/fell down a drain?
Unfortunately, it's the kids that need a hot lunch the most that are likely to come from homes where keeping track of school stuff isn't a priority.
downtime happens onsite as well
Yes, but then you can at least DO something about it, divert facilities, come up with alternatives - a whole raft of options exist (and can be planned for) that you do not have with the cloudy stuff.
Of course, they will tell you THEY have all that, but MS has yet again proven that they don't.
The difference between on prem and cloud is that when on prem goes down it's your problem and you can act, in the cloud it's EVERYBODY'S problem and you can do sod all other than wait and moan a lot. On a global scale, the impact on productivity is so much greater. Oh, and you don't have a third party holding your data hostage either.
After the already negative impact of having to use Microsoft products in the first place, going cloudy merely exacerbates it..
That's what I thought originally, but when I tried working out how that would work I ended up with at best a relay holding all incoming messages in queue unless you have the two Exchange servers behind primary and secondary somehow synched. In the Unix world that is not hard, in the Microsoft world that amounts to setting yourself up for a f*ckfest of epic proportions - and that's just inbound. Outbound and stored email? Well, your second priority MX record won't help with that, unless the aforementioned sync exists that server will be empty and even if it weren't, good luck reconfiguring all devices to use the second Exchange server unless you planned ahead (wot, not believing Microsoft's promises? Surely not) and moved it all to a DNS A record that you can quickly change. Again, assuming you manage to get something to sync and not die as well the moment the cloudy part remembered that it was managed by Microsoft and thus failed.
In short, this again demonstrates that Microsoft went cloudy to pretend to have some support for service resilience in their products. And again failed spectacularly.
"don't have the funds or expertise"
Priorities. Pay the fuckin' money and get someone(s) good -- in-house, or contractor -- to do the work. It's cheaper than the cost to your organization of CloudyServiceX downtime.
Icon for anger at false economy -->
(Cloud is great for some things, but usually not for the things it's most-often used for.)
When I was at college we ran a 100 user time sharing system on mainframes. We achieved a 99% scheduled uptime counting every crash for any reason at least a 15 a minute downtime. The main cause of unreliability was total building/college power failure. Maybe not the 99.99999% uptime claimed by the clouds but still noteworthy in the 1970's. Every time you add another 9 you need an order of magnitude greater effort.
Hmm, does seem to be a fundamental flaw with the cloud model, if someone snips the cord, your cloud will float away. But what do I know, I'm just an old timer who's been in the industry since the early 80's, I take it these young'uns know what they're doing? Can't lie it seemed better when you could go in to the machine room, and add another sack of coal to old Betsie, the mainframe node, when things got a bit sketchy.
My grandad looked after a static steam engine that powered a large flour mill back in the early 60s before he retired. I have actually seen blokes shovel coal into a huge furnace to get the steam up (albeit at a young age).
Some change in a bit more than half a century. Humans could be really dangerous if they got organised.
Icon: obvious
If your OS is sourced from a single supplier then you're at risk of them having an off-day when they roll out an update (sure, you can mitigate by extensively testing their updates before allowing them - but that does rather sound like doing their QA for them). However, if you then decide to place your business' email and working files in a cloud owned by the same software vendor then you're multiplying your risk considerably: not only could their cloud connections cause issues, but their own internal rollout of their own update could take you down (question: when they release an update, do they extensively test it before rolling it out across their cloud? I think we should be told!)
Put simply: the more you rely on a single supplier, the greater your risk. Use the cloud if that suits your operational/financial processes. Rely one one vendor for your OS and mainstream business apps too, if have to. Just be aware that placing all your eggs in one basket doesn't usually work out as the best strategy if business continuity is your objective - but then, we all know that, right? Right?
The much more pressing future concern for Microsoft to presently consider is can they deliver the customers and services AI wants .... for that is what is required of their business in order for its programs to survive and prosper and lead in advanced fields of virtual endeavour and free market enterprise.
This post has been deleted by its author
I suspect the primary use of the AI for Microsoft is to come up with new and innovative excuses for why their products and services fail or are horrifically hard to secure. Hence the investment, apparently that is cheaper than actually fixing the code..
About a year ago I saw a van on the street that had the email address of the business painted on the side.
It said : @outlook.com.
I am a bit embarassed to note how slowly it took me to realise that that was a Microsoft address.
At least on vans, email addresses are mostly gmail nowadays and the outlook van was almost an historical curiosity.
"I feel bad for the guy who pulled the trigger on the change that has caused this firestorm."
Never be that guy. At a certain level, always have your changes signed off. By a technical and business team. It is funny to watch our Network Director say "it's been tested by 'networking company' in the lab and there should be no problems". Then, as a group, everyone in the Change Approval meeting said "No, we need a war room and business testers".
Once burnt, many times shy.
(Network Director has everyone take an instant dislike to him. It saved a lot of time.)
Considering we got an email from our CSP provider giving us the absolutely brilliant news that Microsoft are jacking up ALL their 365 CSP prices (Bar Azure..) by ~9% on April 1st 2023 in the UK (I WISH this was an April fool).
That'll be the second price increase in less than 6 months, makes it extremely hard for those MSP's selling month-to-month to give any kind of customer reassurance to pricing whatsoever, as Microsoft have also said they'll be re-evaluating their pricing based on USD/GBP fluctuation every 6 months and the cynic in me seriously doubts this means prices will drop at any point, even if the USD/GBP rate reflects this!
As usual there seems to have been a bit of hyperbole in the reporting. Reporting it as "down" implies that the whole lot and caboodle was down, bit that only some users couldn't sign in. One of may favourite reports I saw today reported this as if it were some sort of international disaster on the scale of the whole internet being down. Further down the article they started that 5000 users in the UK were impacted. That isn't a significant number at all. Turns out that their 5000 figure came from doendetector, hardly a reliable source, but even so all that hyperbole looked ridiculous if you read down far enough to find them reporting at only 5000 users were affected.
The only thing I noticed was the URI link authentication in emails was a bit slow. We run a lot of our servers on Azure and I didn't hear of any customer complaints. Certainly nothing like the flood we get when something big goes wrong.
It's still not a good look and I'd like to know why Europe keeps taking the brunt of these as other posters have wondered. But it seems like a it was a small storm in a large teacup.
There was a time that I would expect el Reg to break IT-related stories first.
That time appears to be past.
The Graun broke this so long ago now that it's not even on their front page any more.
https://www.theguardian.com/technology/2023/jan/25/microsoft-investigates-outage-affecting-teams-and-outlook-users-worldwide
That's date 09.53 GMT. El Reg picked it up at 11.30.
Massive fail.
-A.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
