Migrating an Exchange Server to the Cloud? What could possibly go wrong? The weekend is over so ease yourself into the working week with a few words guaranteed to strike fear into the bravest Who, Me? reader: "We were moving to Office 365..." Today's bang-up-to-date tale of migration mischief comes from a reader we'll call "Ben", who swears the following events were not his doing; he only dealt …
Forget the ‘We didn’t read the documentation’ or ‘Proof of concept. Nah, people do this every day it’ll be fine’.
I’d want to know why the ‘backup’ service account had enough permissions to start messing with the Administrator user accounts inside Active Directory. It should not have been able to go near any of these accounts.
You are using major's logic versus sergeant's logic.
Major: this is not being done according to the book.
Sergeant: Sir, if we do it according to the book there will be a royal screw-up.
This is the problem here: perfect security would have meant the users could not have been recovered. An unofficial backdoor is a vulnerability, but provided a way of keeping the show on the road after a TITSUP.
i am clearly using the wrong back-up software, i've had back-up tools that wouldn't work without full domain admin rights especially going cross domain.or work group to domain or backing up AD itself for that matter.
before some one asks we use enhanced security to Lock down(mfa, restrict RDP etc) any servers that use a full domain admin service account and the passwords for them are long complex and stored in a password vault with mfa.
better question is don't they have a back-up admin account without exchange rights for this sort of thing Or use a secondary accounts for admin rights.yes a pain for the Admin but better from a security prospective (and some creative scripting can minimise the pain)
personally don't get with more companies don't use hybrid mode and migrate straight from onprem into O365 sure you have to keep a small exchange box for Ad Sync but that's a free license (i heard that some even put it on a domain controller but that's not Ideal).
I left the world of System Administration many moons ago around the time VXA was becoming a thing and then promptly didn't. It might have just been customers of the business I was working for (the SME market) but it was rare they'd let us test restoring files because they were generally too busy and none of them wanted to pay for it.
We'd occasionally find the odd customer savvy enough to let us prove file backups were working advertised but no-one would let us do anything with the Exchange backups because they were all too scared of it going wrong.
It seemed like a very strange attitude to me but there you go
Windows restore files privilege is a "God Privilege" (any user with that privilege can gain system level access, much like the debug privilege also allows the gaining of full control).
IIRC the backup privilege doesn't give that much potential, but given the internal database of local account details can be backed up, you can brute force local account passwords offline.
(My coat's the on with Windows Internals in the pocket.)
Often the backups are taken automatically. The first problem is that people do not know how to restore, the second problem is that people do not know what to restore - and might restore just one file instead of a set - eg user data/transactional log data/configuration data which makes it even worse to recover!
I’d want to know why the ‘backup’ service account had enough permissions to start messing with the Administrator user accounts inside Active Directory. It should not have been able to go near any of these accounts.
Ummmmm, How do you expect Active Directory to be written to tape and read back from tape again? To do that it needs full read/write capability with active directory. Oh, I guess you don't care about backing up the actual user accounts.
I keep having this argument with Commvault engineers and they always say the only way to backup AD using their agent is to give the service account DA rights. It's not true.
Also, their current doco says they need Schema Admin rights to perform an install of the agent. a) Don't let them do push installs onto DCs. b) you need to ensure the account used for backup/restore has Reanimate Tombstones (as well as R/W to child objects in the domain), and to set a search flag value on UnicodePwd so that the password isn't stripped from the account on deletion (otherwise it gets restored without a password and is disabled by default, which I don't really care about anyway). So for the latter, yes, you need a schema admin to make the change. Not the commvault service account.
If an account is deleted, my first port of call is the AD Recycle Bin these days. Commvault is great for restoring attribute values, though (including such things as group memberships at a point-in-time).
This reminds my of my first encounter with that Exchange 2000 directory service sync tool, to bridge the gap from the days when Exchange was independent and AD didn't exist.
I remember my first migration/upgrade from 5.5 to 2000, I was a stand-in last minute consultant when my colleague was ill (or bottled it) and I was supposed to know what I was doing, the only problem been that I'd never done a 5.5 to 2000 upgrade before. I was desperately reading notes on the train, and trying to memorise the sequence. This wasn't a nice small client either, there were at least 200 people working locally, and they had 4 other sites, all with ISDN connections, remote BDC's and remote exchange servers.
Anyway, I went for it, and everything was going SO well. People were able to connect to their mailbox on the new Exchange 2000 server. Fine.
To achieve this you has to use a sync tool to hook up two products that had previously never communicated, Exchange 5.5 directory Service and this new shiny Active Directory.
In the sync tool were several options. The one most people wanted was 'create AD object for any directory service objects not in AD'. I'd guessed correct and been running that one for hours. Everything was going great. There was also a checkbox labelled "delete Exchange directory objects where there is no matching object in AD'. Having run it for several hours I couldn't see the harm in experimenting with that option, as in my mind AD now had everything that mattered. Little did I know how lethal that option is.
It instantly created two serious problems - firstly the CEO and board chairman only accessed his email using POP3 on his Psion and had no AD account, so he didn't get hooked up. Consequently his mailbox got deleted as there was no trace of it in AD.
Secondly, as I learned very fast, I hadn't yet migrated distribution groups, so running this option removed ALL distribution groups in Exchange 5.5. For all sites.
Luckily the IT department treated it as a huge joke and were able to restore the mailbox, and then phoned up every department head asking what staff they had. Somehow I kept my job.
Had something similar recently, but 100% not my fault.
I was doing a 365 migration to a new entity for a public sector body, and once I had moved the mailboxes they started disabling the user accounts on-premises without telling anyone, as the new entity was on a new AD and the 365 mailboxes were to be used as an archive. Of course whilst the directory sync was still in place this also disabled all the mailboxes on Office 365 as well. Had to have a word and politely suggest that they left the accounts live until the hybrid configuration and AD sync was removed.
Surely accounts removed from Exchange are only mail-enabled accounts. Which makes me wonder why admin accounts would have Exchange mailboxes.
In most orgs I've worked at, admins have separate privileged accounts (for obvious reasons) and they are not mail-enabled because they have their "normal" account for that.
Well, there are two types of admins : the ones who have the time and resources to do everything right in a secure manner, and the rest who have no time because they are constantly in fire-extinguisher mode because either they are incompetent or they are competent but do not have the resources to their job correctly because IT is a cost center.
Administrator@ is separated in Exchange 365 from the AD Administrator@.
However, the accounts on Exchange 2010 would still be linked to the AD so would be disabled at the point the Exchange 2010 was decommissioned.
There are several migration issues not looked at here that we've spotted since our Exchange was moved:
1. Until the hybrid exchange is de-linked email still needs to see the old exchange server. Which we discovered when our internet went down taking out our email at the same time.
2. Migrated accounts can't be set to use the Exchange 365 archiving through the web interface. You have to set it in the AD settings.
3. Since last week you now must set the email address in AD or reply addresses now revert to name@company.onmicrosoft.com (the migrated accounts are fine, this is just users created after migration).
4. If you're using Exchange 365 for the love of god turn on 2FA. Yes, 2FA is an abomination in the form that Microsoft has implemented (why have they made creation of "app passwords" such a difficult job and why is it Outlook on my PC requires an app password, but on my phone uses my regular login password?) but it's essential in order to prevent users being dumb enough to enter their login details for their email account into what is clearly not a Microsoft login page just because someone else they know has sent them a file. Instill in your users a sense of paranoia, make them aware that a) if your boss genuinely expects you to pay an invoice to an unknown supplier immediately, they're an idiot. And b) if any file "attachment" requests that you need to login to access it, it's 99% likely to be fake.
There's a few other issues as well, like passwords randomly deciding to stop syncing, but they're the main ones we've noticed.
So could the AD be restored from backup?
Could the users not authenticate to Azure AD in the cloud and then temporarily upgrade the account to allow back syncing to the on Premises AD (of course the cloud AD may already have be synced and had all the accounts deleted, but you should be able to recover those through the cloud portal)?
Why would you need to restore every machine on the network? There is no requirement for every machine to be in a certain synchronised state otherwise you couldn't restore any machine or re-image one or add a new machine used by the same user. I've restored ADs into test networks without issue previously.
I can see you've put it as a joke, but I don't understand it.
If you restore one of your domain controllers, it will resync with the other one so you don't lose any of the changes you made, and that will work.
In this scenario, you want to lose something, ie the deletions you made, and trying to roll the whole thing back in time, that probably isn't going to work.
On vacation, and just read this:
The team were sort of right, as Ben explained: "We did not know that removing the users from Exchange would automatically disable the users in Active Directory."
A M$ Partner made this happen?!!! WTF!!! I haven't used Exchange in 14 years, and I knew this! Might have been a better test to unplug the network cable on EXchange and test?
So glad Exchange was shown the door at our place...
If you delete a user in one place in Active Directory, you delete them everywhere. That's the whole point of Active Directory and the USP touted by Microsoft when they introduced it. The usual reason for deleting a user is because they have left the company, and you don't want ex-employees to still have access to certain parts of the system because you forgot to delete them everywhere.
I don't know how the hell these morons were allowed to be anywhere near Exchange. Given their apparent skill level, this should have been run under a strict AD split-permissions model.
There's nothing wrong with Exchange itself, since setting up the permissions appropriately is a well-known procedure. But, if you're an Exchange admin, how can you not know you can affect every single mail-enabled object in the directory with the default configuration?
I'd have thought this is the sort of operation that must happen fairly often, certainly one that Microsoft want to happen fairly often. OTOH it's probably one that individual admins only do rarely and in any case there's always a first time. And it's one that seems high risk.
Taking those three together why haven't Microsoft automated it?
Since Exchange 2000, it's been tightly integrated with AD. In fact, it was one of the primary drivers for the invention of AD. You don't have to maintain a separate x500 directory any more - it's baked in.
These idiots should not have been anywhere near Exchange if they did not understand the close connection between AD and Exchange user objects. A mailbox is simply a database object - nearly everything to do with it is configured as a user property in AD. Whether that mailbox is located on-prem or Exchange Online. So yes, deleting the user object from Exchange DELETES THE USER OBJECT in AD, assuming you're using the default OOB permissions model. Which, guess what, were developed in the late 90s to allow admins to manage mailboxes and users seamlessly from the same directory (since user admins almost by default became the Exchange admins - you put in AD to have Exchange!)
Now, there is such a thing as a split-permissions model that can be done in two basic flavours. 1. complete separation of AD admin from Exchange - you cannot alter AD user objects from the Exchange console. Rare, especially early on, when of course your Exchange and AD admins were the same people. There's also what MSFT calls the "RBAC" model, where certain objects can be managed by the Exchange console to varying degrees. Such as the ability to update distribution group memberships but not delete user objects.
It's quite flexible and granular and the product is actually pretty capable in that respect. IF YOU KNOW HOW IT WORKS.
The fact that these numpties didn't even know that the accounts synched to Azure AD (and the mailboxes in Exchange Online) are linked to the AD accounts is jaw-droppingly stupid.
Also, by the way, if you have an on-premises AD, you MUST HAVE ONE EXCHANGE SERVER IN HYBRID MODE ON PREM. I had this argument with MSFT a few months back, since we sync the accounts with AAD Connect (which purportedly syncs back user and group email attributes to on-prem AD), but there are some magical incantations you still need an Exchange server for, even if it's a tiny VM (and has no mailboxes or SMTP traffic). I'm not going to summarise why here, but just be aware.
Finally, the thing about the "Linux backup" being able to restore the AD accounts is a frigging ridiculous point. It doesn't matter what OS the backup runs on if it's AD-aware. Yes, AD is a customised LDAP database, and if the product can read the objects, schema and perms with its action account, big whoop-de-do. It worked as it should. Whether running on VMS, Linux, Solaris or a non-domain-joined Windows server.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
