User locked out of Microsoft account by MFA bug, complains of customer-hostile support Konstantin Gizdov, an IT professional, was locked out of his Microsoft account by a bug in the company's Multi-Factor Authentication (MFA), but says support refused to acknowledge the bug or recover his account. Gizdov is founder of KGE Consultancy Ltd in Edinburgh and an Arch Linux Trusted User. His problems began when he …
"Clicked on" in this case probably means copied the URL, inspected the URL, verified that it did in fact go to a Microsoft-owned domain which it did, verified that it was an expected domain name which it was, and one that a standard user couldn't edit which it wasn't, and then put it in a browser. Like we do all the time because people do send legitimate URLs in emails. They don't need to pad out that part of the description when it wasn't a malicious link, do they? Your assumption and the conclusion you imply, despite that conclusion having nothing to do with the problem reported, is not useful.
A genuine IT Professional would have (if sharing similar life experiences with me) been scarred for life by dealing with previous versions of Windows etc and would consider using a Microsoft designed and operated system that they have no administrative control over to be likely to be a personalised embodiment of hell, and frantically strive to avoid having anything whatsoever to do with the Microsoft Cloud for fear of the inevitable thunderstorms like this.
I do have a Microsoft account for the Volume License Service Centre; however I have all of the license numbers stored locally and if I lost access to it then while i'd have to get it sorted then it'd be a problem for Microsoft (allowing people to pirate software) rather than an operational problem for me because it wouldn't break anything.
We just made a service that runs on an Android "burner phone" to forward the MFA code bullsh*t to our IRC channel.
Not playing Microsoft's insecurity game. They are just slowing everyone down.
No-one should be storing sensitive data on a Microsoft service anyway so who cares? Just use it like you would a public toilet.
Please list any services you run so I know to avoid them. I'm guessing you do store sensitive information on that service, or you wouldn't have the account, and you have other security problems involving more important accounts. I'd like to make sure the information that gets leaked isn't mine.
I pay for Office365, and had an issue with photo image file integrity on OneDrive ( they change the metadata in the image file). Raised a support case, lots of back and forth supplying all the evidence, logs, etc. They promised to get back to me. Then it went quiet. My requests for an update on the case are ignored. Support Case number now in limbo. I will not be renewing my subscription.
You're right, I could have rejected the offer of my current job on the grounds they use MS services and I'd need an account. I could reject every other company with similar expectations.
Perhaps you could however help me with an outstanding query I have: Would I still be an IT professional if the resultant jobs available to me meant my primary user interactions were asking, "Would you like fries with that?"
Would I still be an IT professional if the resultant jobs available to me meant my primary user interactions were asking, "Would you like fries with that?"
The day will come when one of fries-guy's job requirements is to have a Microsoft account.
I think it goes without saying that if your employer gives you an MS account to be used as part of your job to log in to Office 365 etc... and non-MS websites via SSO then that is a different thing to choosing to have a personal MS account.
Paid for a developer account. Used a private email address. Login was only allowed from certain ASNs. Changed ISP and then could no longer log in.
Spent two days tried to find someone at the crack security team. Ended up speaking with someone wanting to flog domain names. Thankfully, I had logged in from a cloud machine and because the cookies were there from my original ISP, there was a history of this machine. Otherwise there would have been no way to log in again.
My first experience with MS support was in 1990, they were unhelpful, they told me that what I was trying to do couldn't be done. They were wrong. My second experience was earlier this year, their response was exactly the same as before and just as inaccurate. It's not a bug fix that's needed - it's a complete rewrite of the corporate culture.
My experience from the early 90s turned me from Window to a unix professional. Their support denied an issue that I had proven was with a MS driver for a plotter.
Since then the only time I deal with Windows is for my wife's gaming rig. I stay well away from anything that comes the diseased minds of Redmond as only madness and ruin lie in their domain.
I ran into something similar in the mid 90's (before the Internet got big). There was a bug in the VB docs for some Windows API call I was trying to make, seems like the docs gave the wrong constant for one of the variables. This was way back, when there wasn't much online to supplement what was in the printed books in front of me. After screwing around with it for a day or two, I finally called Microsoft's VB tech support (back when it only cost as much as a long-distance call) and was told the value was correct, even though I showed them the value was wrong. It didn't end well - younger me eventually lost his temper and said some Bad Words. I was asked to never call Microsoft Tech Support again. And I didn't for about 17 years. Even then, I felt the urge to use a pseudonym...
the stories you hear on r/realms or r/minecraft are horrific - minecraft players are being migrated to MS accounts, and in the case of issues, trying to get hold of a real person is a nightmare. In the case of realms which is subscription based, it is poor form - they are paying customers.
I certainly don't envy their support requests. When billions of nontechnical people use something, the support traffic must be nearly endless and mostly useless information. Open source operating systems get around this by not having billions of users and not offering general support, but if Microsoft decided not to support Windows anymore and everybody moved to a Linux of some sort, there would be a related wave of requests from new Linux users that I for one would want to run away from very fast.
That doesn't mean Microsoft's level of support is acceptable, as they have plenty of money to spend on improving it. I just don't want to be anywhere near that attempt. I support only close friends and family, and that's hard enough to do over the phone to a nontechnical user.
The IT equivalent of the dead parrot skit!
That reminds my of an incident with Dell support (not much better than MS).
This is back in the early 00's. A traveling employee is in my office and brings me his laptop. "It won't start". So I power it on and get an obvious hard disk failure. The drive is obviously dead and needs replacement. Not having a ready source for replacement drives I decide to contact Dell to "purchase" a replacement. The call goes like this.
Me: "I have a dell laptop model "XX" with a failed hard drive. It's out of warranty and I want to purchase a replacement drive"
Tech: (In moderately understandable Indian accent). Well, lets see if we can fix this. Can you please restart Windows.
Me: "I cannot restart windows as the primary hard drive is dead. I just need a new drive"
Tech: "I just need you to restart windows so I can try and fix your issue."
Me: "It's a dead hard drive, I cannot restart Windows because Windows is not running! The hard drive is DEAD!
Tech: "Let me talk to my manager and see what we can do.
wait.....
Tech: "My manager said that we need to try and restart Windows to resolve the issue."
Me: "Can we just get someone on this call who actually understands what I am saying, The hard drive is DEAD! It will not start, There is no Windows to restart because the computer WILL NOT BOOT because THE HARD DRIVE IS BROKEN!"
Tech: "Please hold..."
Wait..........
A pleasant American female voice gets on the phone.
Her: "How can I help you."
Me: "I have a Dell laptop model "XX" with a dead hard drive I need to purchase a replacement."
Her: "OK great, I can help you with that!"
> providing human support is expensive
This is what I find annoying. The size of the profits being made by these companies and they can't pay for support staff?
And when you do find that rare human, they have a script that says the company is perfect without bugs. Why are staff not allowed to use their own brains any more?
ARGH!
Don't get me started on scripts and alleged support people. A few years ago I spoke to the outsourced IT support at a company I worked at. There was an issue with something on Onedrive which was important but platform agnostic. When I phoned and explained the problem, the bloke said is this happening on a desktop, a laptop, an iPad or smartphone? I said desktop, laptop and Smartphone, I don't know about an iPad because I don't have one. He said I only have an option for one of those so could you please pick just one. Then he wanted to know the OS it was happening on. I said I assume some flavour of Windows as it's Onedrive which is a Microsoft product. He said no what is the OS on the machine I was using I said Windows but also Android. Not sure it matters though as the issue is with something on Onedrive itself. He asked if I could raise a ticket by email instead
Decades ago I had a conversation with a supposed support person at NTL (Analogue) Cable when the headend had failed for the channel I wanted to watch. I explained to the lady that their equipment had failed, that I had Engineer in my job title and worked for a broadcaster. However she said we need to check my box first and could I please perform the following using my remote control. I said I would like to put it on record that it was their headend and I suggested the make of their equipment that had failed. Then I did the tests and they proved it wasn't my box apparently, what a surprise. She said it looks like our equipment has failed. I said I told you that now when will it be fixed. 9 o'clock was the reply. "So I'll seee my film then?" "Erm no I mean 9am" I then enquired as to what I was paying for if the film channel could be out for 12+hrs.
When I was on Comcast, they did a network topology change one weekend (not an unusual occurrence), and I lost Internet. Did a little poking around and discovered the DHCP server was still handing out the gateway IP for the old segment, while handing out addresses on the new segment.
Called their help desk and told them the problem ("no Internet") and then told them to tell their technicians to check the DHCP server configuration. Which, of course, resulted in nothing happening. So, I took matters into my own hands, reconfigured my router to use a fixed IP (the one their DHCP server had issued me) and set the default gateway to what I guessed would be the new value (x.y.z.1). Bingo! For the remainder of the day, I had Internet, while the rest of my subnet did not. Eventually Comcast figured it out.
Trying to manage a large support organization is a lot like trying to manage a large restaurant chain.
You need consistency. You need ways to measure the quality of your product (customer support). You need ways to continually change and improve your product. Yes, this sounds like Six Sigma, TQM, Kata, and such.
In these organizations, a good employee is one who follows the standard procedures to deliver the standard product. A bad employee is one who does not. If that sounds like McDonalds, well...yeah.
I suggest you replace TQM by ISO9000.
Back in the distant past my then employer took to TQM. It had a mantra of "Get it right first time, every time". All the quality stuff led me to deciding quality is like sex, those who spend all their time talking about it aren't doing it. Anyway after spectacularly failing to get a relocation project off the ground, and without any admission that they hadn't got it right first time any time, the top team decided that ISO9000 and continuous improvement was the way to go. Nobody managed to answer my question of how, if we were getting it right first time every time with TQM, could we have scope to continuously improve.
What ISO 9000 wants is consistency. I quickly discovered that quality was a sliding scale and maintaining your position on it was more important than where that position was. I referred to it as the mediocrity management system.
To see the effect of consistency in practice take a look at Trustpilot reviews for banks. This is, of course, subject to selection bias as they're more likely to be the home for disgruntled reviews rather than praise. What you'll see is a lot of what the reviewer considers to be service failures plus a few where an employee actually owned the problem and dealt with it.
I have an awful suspicion that the banks don't really like these employees - they're providing inconsistent customer service. Being generous, this might be because providing dreadfully bad service is the only way they can be consistent.
(Not being generous I have an even more awful suspicion that by dis-empowering the branch staff they can make branches so bad that there's little push-back from customers when they close a few more.)
re: "how, if we were getting it right first time every time with TQM, could we have scope to continuously improve"
The business is constantly changing. What was right yesterday may be a little less right today, or flat out wrong. The change may come from new products, new markets, new customer segments, new customers, new regulations, new competitors, new.. well, businesses are constantly changing.
Then there's the "it's right, but that doesn't mean it's optimal" consideration. You ring me up, I spend an hour understanding your issue, help you resolve it, you're a happy customer. Everything went right for you. But maybe I can integrate this system over here, automatically measure that metric, run this test in the background.. spend only ten minutes with you and still resolve your problem. You're still a happy customer, everything's still right but now I'm helping 6 times as many customers.
It's also to an extent which of those methods to which you subscribe. What matters more is corporate culture, willingness to change and focus on the right outcomes (usually customer and/or financial, but intelligent organisations can translate customer outcomes into financial ones anyway).
Incidentally branch closures aren't encouraged through disempowering staff at the banks I've worked at. There are a number of factors, one of which is simply the dramatic reduction in footfall now people are using the web or mobile apps.
It's because they don't need to. Nobody will ever turn their back to Microsoft because of lousy support so why would Microsoft care ? Just look at this poor Microsoft customer's story. After all the abuse he endured, he will continue contributing to MS wealth.
I guess that proves my point.
of course they can, but they won't, because they calculated that the financial impact of people told 'have a nice day"! is probably nil, and most of them won't just go away, they'll fume on fb for a while, and they WILL come back to use those juicy, FREE services. That's set against very specific cost of support, so...
MS 2FA issues I have experienced are:
1-they do not seem to allow SMS to landline numbers - which works well in the UK for other 2FA
2-prohibit the use of certain ranges of "VoIP" numbers e.g. 020 3 - but do allow a ported to VoIP, 020 7 number
3-emailing to addresses that have a single character before the @ sign
Got caught out with each of these limitations during the last twelve months.
Very loosely linked but exactly the same concept...I was shopping and at the till the voucher printer flashed it's little lights and whirred for a while but nothing came out so I reported it and was told ' there are not many vouchers being given out today'. Well yes, that's because the printer is broken...DOH!
If someone tells you something is broken and this is the first report then what that means is this is the first report...no more, no less
"If someone tells you something is broken and this is the first report then what that means is this is the first report..."
If every report is met with "We have no reports of issues on our platform. There are no bugs.", followed by a refusal to log a ticket, then every report is the zeroth report.
Had a vaguely similar problem with a client who lost access to a Google Workspace account.
Getting in touch with them is practically impossible. In fact I would say actually impossible for a normal soul.
If it weren't for the fact that I had access to an unrelated reseller account where I could actually contact a human being, I doubt I would have been able to get the issue fixed.
There's many things Google do better than Microsoft but support isn't one of them.
The more lop-sided the size relationship is between you and the company you are trying to get customer service from, the less likely it is you will get any.
I do things such as maintain my domain names and web hosting providers separately. If my host becomes problematic, I'll switch out web sites in an hour or so. This is why I don't use proprietary design tools from the hosting company. Obviously it gets to be more difficult the larger your enterprise, but it's also more important to have escape routes should a service provider go away.
It's like having to recover data from a bad hard drive. Just one experience shows how much cheaper it is to have backups.
I got locked out of my Outlook account a few years ago. I couldn't get a human despite hours of trying to. To this day the account not accessible by me. I had to start from scratch, and his time using gmail.
Well, that should bullet proof me, shouldn't it?
"Not only that, but Microsoft by policy require a personal account in order to be able to back up MFA and sync between devices."
True, but not necessarily the issue here? You can enable authentication, including 'push' with Microsoft Authenticator, across multiple devices with a 365 account so to avoid reliance on one device - i.e. you can't lock yourself out of your account if you have multiple devices, which appears to have been the point here. You can add multiple authentication methods in the security section of your 365 account profile to achieve this and enable Authenticator on multiple, separate devices - they *all* then go 'ping' when you receive an authentication request. What you *can't* yet (?) do with a 365 account (but you can with a personal Microsoft account, I gather) is sync things like passwords and numerical authentication codes between devices. M$ need to pull their finger out on that bit for sure.
A/C (because I'm not admitting I'm using this stuff for real :-)
I have no idea why anybody links an account to a proprietary software authenticator on any device as it's just setting themselves up for failure. If you use TOTP 2FA and keep a copy of the secret key you can get back in with any other device if you need to and if the account provider doesn't allow TOTP as an option then it's probably best to look elsewhere.
... assuming you're savvy with understanding and safely storing your key(s), plus assuming your TOTP app lets you have access to that and is also not brain-deadingly attached to one device only.
PS: Microsoft at least gives you multiple ways to prove your identity to get back into your account, assuming you've had the forethought to set them up. If your auth app fails you can resort to SMS text code, or security questions, or a backup email address, etc. So, options, right? Easy ones too, for the average user to comprehend. Your org's 365 admin can also 'recover' your access for you to get you back in if you can't do it yourself.
Of course, if the proprietary *service* you're trying to access with your proprietary auth app is down, then it doesn't really matter anyway... :)
I'm in a reflective mood (and not personally affected, so easy to dish out such snippets of wisdom) so, I would think that 'Goodbye and have a nice day!' is a PERFECT description for the current / ongoing business model of the 'FREE!' internet. Perhaps, the MS 'support' (or ANY 'support') have it etched, in 24-karat gold, on a big plaque over the door, on their mugs, t-shirts, screen-savers, pens, etc.
I think this is the real point. This security theatre simply adds to the pretence that these cloud services are places you can unthinkingly store data that is of value to you.
By all means store copies, encrypted with your own keys, for relatively convenient remote access. But not your only copy. And without any reliance on the security offered by the vendor.
There are so many ways you can arbitrarily be denied access to your data either temporarily or permanently and other ways in which it may accidentally be leaked.
If your business "depends" on Service X, over which you have no control, what is your plan for the time when Service X is unavailable? Because that time will come.
and this proves the MS, Google, FB and Amazon approach WORKS. Once you become big enough, indispensable enough, you can 'prioritize' your customers, free and paying alike, to 'level 0', and they can twitter about till they're blue in the face.
Obviously, if you had a (purely theoretical) situation where business growth to the point of having near-monopolies trying to take on states and government is impossible/prohibited/blocked and you have lots of small ones instead, competing against each other and trying to win and maintain customers, this would make the whole game much more expensive (with much, much wider implications, some good, some bad perhaps). This would also inhibit 'progress', again, with wider implications, and also some good and some bad. But this is theory of course, and in reality human nature dictates the current status quo.
You have a point, but I see it as we are quickly moving to a corporitocricy in this world. With the Left internationally moving faster and faster not towards socialism or communism but towards the thing they "state" they most hate, Fascism! But not like the Fascism of the 1930's where Governments held influence over large corporations but a new Fascism, where mega-corporations hold sway and control over governments!
I activated 2FA.
Not long afterwards, I was unable to log in - the 2FA codes were not being accepted.
When this happens, you are advised to resync.
There's a resync page on the AWS site which did not work.
I was unable to resync, I was unable to log in.
I contacted support - or tried to - because when 2FA, the support offered is a form which lets a 2FA support team know you cannot log in. There's no way to actually send the a message. When you send this form, you get an no-reply email, with a phone number, saying "phone this number".
I don't keep a phone number. I can't phone that number. No 2FA support for me.
The basic problem is that the mechanism used to *initiate* 2FA on an account is *not* the mechanism used to *recover* 2FA unlike email/password based accounts, where they are the same. As such, it can be you can activate 2FA, but not recover when it goes wrong.
I tried contract normal AWS Support, which went as well as you'd expect. I was advised to make a new account.
Eventually, I found a *second* set of resync pages, which worked - and once in, I *instantly* disabled 2FA, since it was infinitely more dangerous than the threats it was there to protect against.
"I *instantly* disabled 2FA, since it was infinitely more dangerous than the threats it was there to protect against."
Sort of. Done badly it will lock out more legitimate users than miscreants, even without the inevitable user error. I have 2FA on a few accounts, one forced on me and the others where it seemed like a good idea. But in the back of my mind I am always worried about getting locked out. I do have backups. In my case having to create a new account (or a few) would not be the end of the world.
If I were a consultant with multiple services connected to a Microsoft account with 2FA, I would be *very* worried -- my customers are likely to think getting locked out is due to my incompetence rather than Microsoft's, and either way they just want me working on their problems. Creating a new account would not necessarily get me working, depending on how I access the customer's systems.
Commercial support isn't much better. You rarely get anyone who actually is experienced in the issue your requesting service about.
Case: We had an odd problem with the (relatively) new O365 Security and Compliance email retention policies. The procedure we had established to manage these policies suddenly started throwing errors. Not only that but with further investigation we notice that the policy deployments are also throwing errors that state "contact Microsoft support". Internet searches did not result in any resolutions.
Contacted MS support. Got individual who speaks very poor English. He asks for all the things I've already done. This technician has no clue about the Security and Compliance center and keep referring back to the legacy Exchange Retention Policies. Which I have to correct him on.
Needless to say the frustration continues, the back and forth of "do this and send me the results" that produces no progress. Finally I decide to do some further internet searching and find a Microsoft documentation article for the PowerShell command we are using updated 2 days ago, stating "not to add individual Exchange locations to the policy repeatedly as the policy will lock for deployment after the 1st entry, add them in batch with an array of UPNs". OK, that fixes the 1st issue.
A week goes by with more "do this and send the results", then finally "please try and redeploy the policies", I do this and the errors are gone.
So then the following final conversations via email (I could never communicate with this person via phone, his poor English and my hearing loss made it impossible)
Tech: "Have the errors stopped"
Me: "Yes, what was the issue, did you guys fix something?"
Tech: "You are not seeing any more errors? Good, can we close the ticket?"
Me: "Yes, I suppose you can. What was the issue, what was the resolution?"
Tech: "Thank you I will close the ticket!"
Take away:
1) It was obvious that MS made a change to their system that caused our established procedure to fail. There were no notifications no warning no communications about this at all. Not surprising as we have experiences several instances of MS making changes to O365/Azure without proper notification that broke things. It is my assumption that this ticket prompted the publication of the documentation article that explained this new "feature".
2) There were obviously errors within our policies that were not fixable by the end user (me). The message to "contact Microsoft Support" made that clear. It was clear that the tech had escalated this to a Security and Compliance engineer but we had no idea that was happening. They clearly found the problem and resolved it but we have no idea what was done to fix it and whether or not there is anything we can do to prevent this from happening again.
All in all just very frustrating. A technician, who can barely pronounce English words supporting customers in an English speaking country just made this hard. Not to mention he continued to call me when I'd asked him to use email as I cannot understand him. (No offense to the Indian people as most I've worked with here in the States are good technicians and damned fine people). The fact that we got no explained cause/resolution to the issue leaves us blind to how we could avoid it in the future.
Lastly, no one could tell us that issue #1 was caused by a change and that we just need to modify our process. The technician handling the issue did not have a clue and apparently those he escalated it to either also did not know or chose not to reveal it was caused by something they changed.
...think again.
Me: Can't login as it is saying account doesn't exist. If I try to recreate, it says account already in use.
G: You need to login and update your settings.
Me: Can't login as it is saying account doesn't exist.
G: Have you tried reset in the account
Me: Can't reset as it is saying account doesn't exist.
G: Try creating a new account with same credentials.
Me: Account already exists.
You know how this is going. It went through 4 teams of over 4 weeks "experts" before finally someone actually properly looked at the issue.
Oh did I mention this was a Paid for account?
I studiously avoid setting up 2FA with any personal web service for this very reason (well, any kind of problem with the "second factor", not just a bug), as I explained in an earlier comment.
I am infosec at an FI, I am horrified that management wants to "move to the cloud". I repeat it back to the boss as, okay so when the internet goes down you want to make sure we don't have access to anything?
I constantly give him examples of insecurity (many from the Reg) news about O365, Azure, and so on. He's still chasing the shiny floating magic cloud - because marketing masters lie about unknown unknowns in security.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
