Azure, Office 365 go super-secure: Multi-factor auth borked in Europe, Asia, USA Happy Monday, everyone! Azure Multi-Factor Authentication is struggling, meaning that some users with the functionality enabled are now super secure. And, er, locked out. Microsoft confirmed that there were problems from 04:39 UTC with a subset of customers in Europe, the Americas, and Asia-Pacific experiencing "difficulties …
Didn't say that.
But when it does, your data is sitting RIGHT THERE. Available to you.
Literally, pop down to PC World, buy a machine, access your backups, bam... data.
On-prem is no different to cloud or anything else - but you have your data in your possession. So even if it means "ARGH! Quick... install an exchange server quickly so we don't lose email on our domain", you can do that. Without having to wait for some cloud provider to switch you over. Or you can pull all your existing files and refer to them. Or you can literally make your own mini-network and get things running again.
While you're tied exclusively into "I have to log in to Azure and it's not working", you can't do a damn thing, even as an admin.
People with brains do on-prem AND cloud, so failure of one doesn't affect the other.
I speak as someone who only two months ago had 450KW of three-phase cable arced together at the main transformer station supplying the entire site. Our servers went down hard.
And I had a copy of the data... just there... literally in arm's reach. Given an absolute emergency, I could have used iSCSI and ANY MACHINE WHATSOEVER to power up some Hyper-V images direct off the storage and booted things up. As it was, it was a weekend and power came back on the Monday, so I just powered up the 50% of kit that wasn't damaged, checked data integrity and carried on as normal.
But if off-prem doesn't let you even log-in... you're stuffed.
"Because on-prem stuff never breaks, floods, gets stolen, overheats or loses power?"
Well said! You deserve a beer. And a topic. I hope you like it.
(I hope you weren't being snarky - which you probably were - but I assume that, on average, the likelihood of any of those things actually happening is pretty low compared to cloudy outages)
"the guys that were giving their passwords to scammers/spammers"
Now everyone's giving those passwords to microsoft, google, amazon, ... "Microsoft Logon" anyone?
If the cloud were a batch process to backup to it, rather than an ongoing bandwidth-intensive "required to do work" constant access, it would make sense. Cloudy "applications" are not just overrated, they're THE PROBLEM.
I've never fully trusted cloud-y things since a previous employer moved everyone to Sharepoint without investing in adequate bandwidth. This was a fairly large premises anyway, but operating at nearly double its initially scoped numbers following multiple rounds of collocations, hot-desking, floorplate adjustments, etc. We had >8,000 people working on a site with an IT backbone scoped for ~4,000 (and that's also 4,000 at the time of build - so far lower planning assumptions than modern services and applications require).
The fact that I'm old-fashioned and non-collaborative, and regularly make locally-saved working versions of documents means I'm actually able to get on with things today, instead of scratching around and asking "well, what do I do now?!"
The fact that I'm old-fashioned and non-collaborative, and regularly make locally-saved working versions of documents means I'm actually able to get on with things today, instead of scratching around and asking "well, what do I do now?!"
I did some work for a company where the (lone) Sysadmin managed to bork our entire network and consequently almost nobody could work. The director of the company came round and told people they should go home if they couldn't work and that they wouldn't be being paid for the lost time.
"I have local copies of my files," I promptly said and carried on typing. Others were a little more honest and left.
The company is no longer there, incidentally.
Common misconception. "Cloud" is not synonymous with distributed systems. It makes it easier, sure, but "Cloud" is just the buzz word your accountants use to justify spending less by "co-locating" your data with an "online service provider", "simplifying the support chain", and "reducing non-core business focuses".
Well, our external IT support suggested we move everything to the cloud, to which I said no, but hey this 2 factor authentication seems like a really good idea as someone brute forced a bunch of our 365 passwords and sent half a ton of spam.
Now sitting here hitting the refresh as I cannot change any email accounts or login to the portal to set up anyone.
Great job Microsoft, my F5 key is getting a workout. Still at least I can sit and read El Reg, gives me something to do.
Turns out it's only their MFA validation infrastructure that's flailing. If you whitelist your IP address (which we do for those of us with static IPs) here - https://account.activedirectory.windowsazure.com/UserManagement/MfaSettings.aspx?culture=en-GB&BrandContextID=O365 - then it works fine
Why can changes to Azure that break things not be rolled back almost instantly?
Why is their testing so poor that these issues cannot be identified?
Why is customer communication so poor about clearly mission critical systems?
Why do whole regions get affected when others don't?
What is your plan of action for improvement?
Until we hear full, credible answers to basic questions like these, MS cloud services should be considered too unreliable for business use.
We put MFA on all users with admin rights....I somehow was still logged in from Friday (laptop went into hibernate) and have been lucky to get back in again. Otherwise how would I see this super, helpful message from MS? Thank [insert your deity] we have not rolled this out to all users yet - as per my recommendations to enhance security!?!
Latest update from Admin Portal:
MO165510 - Unable to sign in to Microsoft 365 services
Status:Service degradation
User impact:Affected users may be unable to sign in using Multi-Factor Authorization (MFA).
Latest message:Title: Unable to sign in to Microsoft 365 services
User Impact: Affected users may be unable to sign in using Multi-Factor Authorization (MFA).
More info: Users may also be unable to carry out self-service password resets.
Current status: We're in the process of deploying the code update. For those users in the infrastructure that has received the update, we're seeing a drop in error rates and improvement in connectivity. We're monitoring the deployment to ensure the fix is effective.
Scope of impact: Impact may be experienced by users accessing Office 365 services via Multi-Factor Authentication.
Start time: Monday, November 19, 2018, at 4:39 AM UTC
Next update by: Monday, November 19, 2018, at 4:00 PM UTC
Updated:2018-11-19 14:45 (UTC)
Start time:2018-11-19 04:39 (UTC)
Microsoft has NOTHING to explain. They are Microsoft. They are a monopoly, there's noplace else you can go, there's nobody else available, it's ONLY them, you HAVE to accept it, and they don't care about YOU.
Microsoft don't have to explain NUTTIN to NOBODEEZ. Once you're assimilated, you'll understand...
[at least, that's the perception]
Not sure that word means what Microsoft think it means
mitigate
/ˈmɪtɪɡeɪt/
verb
gerund or present participle: mitigating
make (something bad) less severe, serious, or painful.
"drainage schemes have helped to mitigate this problem"
synonyms: alleviate, reduce, diminish, lessen, weaken, lighten, attenuate, take the edge off, allay, ease, assuage, palliate, cushion, damp, deaden, dull, appease, soothe, relieve, help, soften, temper, still, quell, quieten, quiet, tone down, blunt, dilute, moderate, modify, abate, lull, pacify, placate, mollify, sweeten, tranquillize, remit, extenuate, excuse, commute
"drugs which mitigated the worst symptoms of the disease"
antonyms: aggravate, increase, intensify
•lessen the gravity of (an offence or mistake).
"he would have faced a prison sentence but for mitigating circumstances"
synonyms: extenuating, exonerative, justificatory, justifying, vindicatory, vindicating, exculpatory, palliative, qualifying, moderating, modifying, tempering, lessening
"he would have faced a prison sentence but for mitigating circumstances"
I tried to access email from a certain local, very much on this side of the Atlantic, site, this morning. They had gone all in with Office365, including the imbecilic MFA which requires logging back in every 24 hours and using the MS Authenticator App on your iOS or Android device to re-auth. As they don't pay me enough to care, I had not logged in over the weekend. In the halcyon daze of yore, before they went all MFA, I could just log in, see any email which had accumulated over the weekend, ignore the bumf (which would be 95% of it) and drive on. This morning, of course, their 'secure' email is very secure in most ways. You see, they'd turned the MFA on on desktop/laptop email clients, and on web clients, but not on email clients running on actual iOS devices. Even they weren't so thick as to send MFA alerts to the device the users used to connect to the service requiring MFA. (Hi, Apple; they do exactly that. Exactly what Earthly purpose Apple MFA serves is beyond me.) This means that I can, oh, use MS Outlook on an iOS device and get mail. (One post was from support, telling the users that MFA is borked. Gee. Thanks. We could never have figured that out without you.) Apple Mail is blocked, even on the same iOS devices where MS Outlook works. Hmmm. I wonder why this might be... Web access, including web access to the employee web for everything, is still blocked. Numerous people use the employee web to do actual work; they're locked out as I type this.
I had posted to my line manager, the vp in charge of this division, and the IT manager back when they first set up this nonsense, saying that this was a disaster waiting to happen. For some reason the IT department is not talking to me this morning. How very sad. I know that certain elements at HR use iPads; I have posted to them indicating that I'll be looking at the next paycheck and I had better not be missing any pay because of this fiasco, copying the little notes I sent to the line manager, the VP, and the IT manager way back when. And, yes, I have been looking for a new job for quite some time, and have found one and will be starting in January.
Useless gits.
I'm in Canada on a US instance and our MFA is borked....so much for managing any clients instances today.
Since its unlikely they need to do code changes to fix something that was running fine I suspect they must have changed something and now are trying to fix whatever their eggheads broke.
2018-11-19 06:45 (UTC) Scope of impact: Impact is specific to a subset of users who are served through the affected infrastructure.
"It's just a small number of users"
2018-11-19 07:56 (UTC) Scope of impact: Impact is specific to a subset of users who are served through the affected infrastructure.
"No honestly, its just a small number of users, nothing to see here"
2018-11-19 08:54 (UTC) Scope of impact: Impact is specific to any user who is located in the Europe, Middle East and Africa (EMEA) or Asia Pacific (APAC) regions.
"It might be a few more users than we thought, but America is fine!"
2018-11-19 09:59 (UTC) Scope of impact: Impact is specific to any user who is located in the Europe, Middle East and Africa (EMEA) or Asia Pacific (APAC) regions.
"America is still fine!"
2018-11-19 12:24 (UTC) Scope of impact: Impact is specific to any user who is located in the North America (NA), Europe, Middle East and Africa (EMEA) or Asia Pacific (APAC) regions.
"But South America is still okay..."
2018-11-19 12:50 (UTC) Scope of impact: Impact may be experienced by users accessing Office 365 services via Multi-Factor Authentication.
"Okay, you got me, it's everyone"
Whitelist all IP addresses:, since 0.0.0.0 doesn't work use:
1.0.0.0/1
128.0.0.0/1
Which can be done here: https://account.activedirectory.windowsazure.com/usermanagement/mfasettings.aspx
Then once Microsoft sorts it's act out you can remove the whitelist entries and everyone is back to working normally without having to reconfigure individual users. Beware of course that this increases your risk profile hence why I'm posting this anonymously, I wouldn't advertise that you've done this anywhere
From MS https://status.office.com
Current status: Due to the complex nature of this problem, our investigation into the root cause of this issue may take an extended period of time. This incident is and will remain our highest priority until the underlying source of the problem has been identified.
I’m not looking forward to tomorrow morning...
Latest:
11/19
Issues connecting to Azure resources in Europe, Asia and the Americas regions using Multi-Factor Authentication
Summary of impact: Between 04:39 UTC and approximately 21:30 on 19 Nov 2018, customers in Europe, Asia-Pacific, and the American regions may have experienced difficulties signing into Azure resources, such as Azure Active Directory, when Multi-Factor Authentication (MFA) is required by policy.
Preliminary root cause: Requests from MFA servers to Redis Cache in Europe reached operational threshold causing latency and timeouts. After attempting to fail over traffic to North America this caused a secondary issue where servers became unhealthy and traffic was throttled to handle increased demand.
Mitigation: Engineers deployed a hotfix which eliminated the connection between Azure Identity Multi-Factor Authentication Service and a backend service. Secondly engineers cycled impacted servers which allowed authentication requests to succeed.
Next steps: Engineers will continue to investigate to establish the full root cause and prevent future occurrences. A full Root Cause Analysis will be published in approximately 72 hours. To stay informed on any issues, maintenance events, or advisories, https://www.aka.ms/ash-alerts and you will be notified via your preferred communication channel(s): email, SMS, webhook, etc.
Like many we were caught out with this MFA issue and find it difficult to understand the poor communication and duration it took to fix. Found a way around it using the Office365 admin Mobile app, sorry late in the day to be posting a work-around.
The app still functioned and remained logged in unaffected by the MFA issue for user admin/email tasks including creating new users. We created a new user and assigned Admin role, it is not possible to enable or disable MFA in the app. This is fine as once user created we could log in with the new Admin and turn off MFA for the existing Admins.
Although a bit slow, the Admin app is quite handy for lots of things and got us out of a hole yesterday.
Outlook (email) was also down completely for 4 hours the other night and nobody noticed. When we were acquired by our new parent company we were forced off the perfectly fine corp google mail service we were using that never had issues onto the Microsoft stuff and it's been crap ever since. I usually defend Microsoft quite a lot but you'd think they could manage to keep something so simple online for everybody, all the time.
now they know why i said id rather shave my giblets with a broken beer bottle, itll end in tears you tits!
cue 90% of global IT up the shitter today, heh, didnt think it would only take 10 days to prove my point.
TODAY was ace fun at work :P on-prem ftw! HA failover setup MS? Offered C&C use of some of my rackpsace servers twiddling their thumbs, oh your MFA`d ALL the ADM accs inc the master one, a good job done well my stateside brethren! <cue mic muting and pissing IT pants UKside>
Why Microturd are market leaders
Oh now I remember, they gave their shitware away for free in the early days to get market share and now we’re all fucking stuck with them, all businesses use their clusterfuck software, its a fucking shitshow
Someone need to tell them to fuck off and chew on a tailpipe
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
