back to article Migrating an Exchange Server to the Cloud? What could possibly go wrong?

The weekend is over so ease yourself into the working week with a few words guaranteed to strike fear into the bravest Who, Me? reader: "We were moving to Office 365..." Today's bang-up-to-date tale of migration mischief comes from a reader we'll call "Ben", who swears the following events were not his doing; he only dealt …

  1. Anonymous Coward
    Anonymous Coward

    Forget the ‘We didn’t read the documentation’ or ‘Proof of concept. Nah, people do this every day it’ll be fine’.

    I’d want to know why the ‘backup’ service account had enough permissions to start messing with the Administrator user accounts inside Active Directory. It should not have been able to go near any of these accounts.

    1. Anonymous Coward
      Anonymous Coward

      You are using major's logic versus sergeant's logic.

      Major: this is not being done according to the book.

      Sergeant: Sir, if we do it according to the book there will be a royal screw-up.

      This is the problem here: perfect security would have meant the users could not have been recovered. An unofficial backdoor is a vulnerability, but provided a way of keeping the show on the road after a TITSUP.

      1. Pascal Monett Silver badge

        In other words, if someone hadn't fucked up with permission security, the guy who fucked up with Exchange would have basically reset the company's IT network.

        What a weird way to save the day.

        1. Anonymous Coward
          Anonymous Coward

          The problem is that an "ideal" security setup assumes nobody ever makes a mistake. This is not a real world situation.

      2. Nick Kew

        That'll be why government needs its backdoor into all our communications.

        It's to save us from ourselves when we screw up!

      3. Mike Groombridge

        i am clearly using the wrong back-up software, i've had back-up tools that wouldn't work without full domain admin rights especially going cross domain.or work group to domain or backing up AD itself for that matter.

        before some one asks we use enhanced security to Lock down(mfa, restrict RDP etc) any servers that use a full domain admin service account and the passwords for them are long complex and stored in a password vault with mfa.

        better question is don't they have a back-up admin account without exchange rights for this sort of thing Or use a secondary accounts for admin rights.yes a pain for the Admin but better from a security prospective (and some creative scripting can minimise the pain)

        personally don't get with more companies don't use hybrid mode and migrate straight from onprem into O365 sure you have to keep a small exchange box for Ad Sync but that's a free license (i heard that some even put it on a domain controller but that's not Ideal).

    2. katrinab Silver badge

      Because it needs to do backups of the said (in)active directory?

      1. John Robson Silver badge

        A backup only needs read access, not write access.

        1. Fabrizio

          service account

          The backup account was probably a very old service account from back when Windows NT 3.51 would happily convert any kind of account to full administrator after popping up a dialog box:

          Convert account 'BackupUser'

          to administrator?

          Ok Cancel

        2. Anonymous Coward
          Anonymous Coward

          Sometimes you need to restore as well - there may well be a backup account that does restores.

          1. Evil Harry

            I left the world of System Administration many moons ago around the time VXA was becoming a thing and then promptly didn't. It might have just been customers of the business I was working for (the SME market) but it was rare they'd let us test restoring files because they were generally too busy and none of them wanted to pay for it.

            We'd occasionally find the odd customer savvy enough to let us prove file backups were working advertised but no-one would let us do anything with the Exchange backups because they were all too scared of it going wrong.

            It seemed like a very strange attitude to me but there you go

          2. richardcox13

            Windows restore files privilege is a "God Privilege" (any user with that privilege can gain system level access, much like the debug privilege also allows the gaining of full control).

            IIRC the backup privilege doesn't give that much potential, but given the internal database of local account details can be backed up, you can brute force local account passwords offline.

            (My coat's the on with Windows Internals in the pocket.)

        3. katrinab Silver badge

          On Windows, doesn't it need to set the archive bit on the file to say that it has been backed up?

          1. phuzz Silver badge

            That was the intended use for the archive bit, but I'm not sure if modern backup software uses it. These days it's more likely to use some kind of snapshot, so the data being backed up would be a read-only, point-in-time copy.

        4. ColinPa

          Often the backups are taken automatically. The first problem is that people do not know how to restore, the second problem is that people do not know what to restore - and might restore just one file instead of a set - eg user data/transactional log data/configuration data which makes it even worse to recover!

          1. Doctor Syntax Silver badge

            The third problem is not knowing what not to restore if there are some good up-to-date files undamaged.

        5. Scott 29

          Backups need write access

          Backups need write access - they write to metadata, for a few reasons. Read-only to data though.

    3. Filippo Silver badge

      It turns out that, sometimes, two wrongs to make a right.

    4. gnarlymarley

      I’d want to know why the ‘backup’ service account had enough permissions to start messing with the Administrator user accounts inside Active Directory. It should not have been able to go near any of these accounts.

      Ummmmm, How do you expect Active Directory to be written to tape and read back from tape again? To do that it needs full read/write capability with active directory. Oh, I guess you don't care about backing up the actual user accounts.

    5. Trixr

      I keep having this argument with Commvault engineers and they always say the only way to backup AD using their agent is to give the service account DA rights. It's not true.

      Also, their current doco says they need Schema Admin rights to perform an install of the agent. a) Don't let them do push installs onto DCs. b) you need to ensure the account used for backup/restore has Reanimate Tombstones (as well as R/W to child objects in the domain), and to set a search flag value on UnicodePwd so that the password isn't stripped from the account on deletion (otherwise it gets restored without a password and is disabled by default, which I don't really care about anyway). So for the latter, yes, you need a schema admin to make the change. Not the commvault service account.

      If an account is deleted, my first port of call is the AD Recycle Bin these days. Commvault is great for restoring attribute values, though (including such things as group memberships at a point-in-time).

  2. TonyJ

    In my experience...

    ...every IT partner/services company has underfunded, crappy IT internally that is supported by new-to-the-game (.i.e. cheap as chips) PFY types.

    If theyre lucky, they have an old hand in charge, but not often.

    The real technical talent are out on customer sites.

    1. Doctor Syntax Silver badge

      Re: In my experience...

      "The real technical talent are out on customer sites."

      Or long gone.

    2. Anonymous Coward
      Anonymous Coward

      Re: In my experience...

      >The real technical talent are out on customer sites.

      I've argued for years that the internal IT support are those not trusted to face customers....

  3. timhowarduk

    Ah, unintended consequences

    This reminds my of my first encounter with that Exchange 2000 directory service sync tool, to bridge the gap from the days when Exchange was independent and AD didn't exist.

    I remember my first migration/upgrade from 5.5 to 2000, I was a stand-in last minute consultant when my colleague was ill (or bottled it) and I was supposed to know what I was doing, the only problem been that I'd never done a 5.5 to 2000 upgrade before. I was desperately reading notes on the train, and trying to memorise the sequence. This wasn't a nice small client either, there were at least 200 people working locally, and they had 4 other sites, all with ISDN connections, remote BDC's and remote exchange servers.

    Anyway, I went for it, and everything was going SO well. People were able to connect to their mailbox on the new Exchange 2000 server. Fine.

    To achieve this you has to use a sync tool to hook up two products that had previously never communicated, Exchange 5.5 directory Service and this new shiny Active Directory.

    In the sync tool were several options. The one most people wanted was 'create AD object for any directory service objects not in AD'. I'd guessed correct and been running that one for hours. Everything was going great. There was also a checkbox labelled "delete Exchange directory objects where there is no matching object in AD'. Having run it for several hours I couldn't see the harm in experimenting with that option, as in my mind AD now had everything that mattered. Little did I know how lethal that option is.

    It instantly created two serious problems - firstly the CEO and board chairman only accessed his email using POP3 on his Psion and had no AD account, so he didn't get hooked up. Consequently his mailbox got deleted as there was no trace of it in AD.

    Secondly, as I learned very fast, I hadn't yet migrated distribution groups, so running this option removed ALL distribution groups in Exchange 5.5. For all sites.

    Luckily the IT department treated it as a huge joke and were able to restore the mailbox, and then phoned up every department head asking what staff they had. Somehow I kept my job.

    1. big_D Silver badge

      Re: Ah, unintended consequences

      And it is always the CEO/CFO who is the bloody exception to the rule. The one who breaks all the internally laid procedures and checks to ensure things "just work"...

      1. Anonymous South African Coward Bronze badge

        Re: Ah, unintended consequences

        CEO/CFO types are "speshul" and need "speshul" privileges/access/rights/whatever...


        1. Joe W Silver badge

          Re: Ah, unintended consequences

          As the BOFH remarked (rephrased, too lazy...) "special priority access and something beginning with z I'll think of..."

        2. itzman

          Re: Ah, unintended consequences

          in many companies they were the ONLY people running apple kit

  4. ColinPa

    Confusious, he say

    1) Delete is a 2 stage operation. Rename/move today, remove next month

    2) Try it on your manager first. He/She should be glad it didnt go enterprise wide if it fails.

  5. Franco

    Had something similar recently, but 100% not my fault.

    I was doing a 365 migration to a new entity for a public sector body, and once I had moved the mailboxes they started disabling the user accounts on-premises without telling anyone, as the new entity was on a new AD and the 365 mailboxes were to be used as an archive. Of course whilst the directory sync was still in place this also disabled all the mailboxes on Office 365 as well. Had to have a word and politely suggest that they left the accounts live until the hybrid configuration and AD sync was removed.

  6. Anonymous Coward
    Anonymous Coward

    Why are the admin accounts disabled by Exchange?

    Surely accounts removed from Exchange are only mail-enabled accounts. Which makes me wonder why admin accounts would have Exchange mailboxes.

    In most orgs I've worked at, admins have separate privileged accounts (for obvious reasons) and they are not mail-enabled because they have their "normal" account for that.

    1. Pascal Monett Silver badge

      Re: Why are the admin accounts disabled by Exchange?

      Well, there are two types of admins : the ones who have the time and resources to do everything right in a secure manner, and the rest who have no time because they are constantly in fire-extinguisher mode because either they are incompetent or they are competent but do not have the resources to their job correctly because IT is a cost center.

    2. tfewster

      Re: Why are the admin accounts disabled by Exchange?

      Further to your post, isn't there a built in Domain Administrator account, for setup and emergency use only? I'd be surprised if that had email enabled.

      1. Alien8n

        Re: Why are the admin accounts disabled by Exchange?

        Administrator@ is separated in Exchange 365 from the AD Administrator@.

        However, the accounts on Exchange 2010 would still be linked to the AD so would be disabled at the point the Exchange 2010 was decommissioned.

        There are several migration issues not looked at here that we've spotted since our Exchange was moved:

        1. Until the hybrid exchange is de-linked email still needs to see the old exchange server. Which we discovered when our internet went down taking out our email at the same time.

        2. Migrated accounts can't be set to use the Exchange 365 archiving through the web interface. You have to set it in the AD settings.

        3. Since last week you now must set the email address in AD or reply addresses now revert to (the migrated accounts are fine, this is just users created after migration).

        4. If you're using Exchange 365 for the love of god turn on 2FA. Yes, 2FA is an abomination in the form that Microsoft has implemented (why have they made creation of "app passwords" such a difficult job and why is it Outlook on my PC requires an app password, but on my phone uses my regular login password?) but it's essential in order to prevent users being dumb enough to enter their login details for their email account into what is clearly not a Microsoft login page just because someone else they know has sent them a file. Instill in your users a sense of paranoia, make them aware that a) if your boss genuinely expects you to pay an invoice to an unknown supplier immediately, they're an idiot. And b) if any file "attachment" requests that you need to login to access it, it's 99% likely to be fake.

        There's a few other issues as well, like passwords randomly deciding to stop syncing, but they're the main ones we've noticed.

        1. tfewster

          Re: Why are the admin accounts disabled by Exchange?

          @Alien8n - Thank you for that explanation - Potential nightmare scenarios, I almost wish I'd remained blissfully ignorant!

    3. phuzz Silver badge

      Re: Why are the admin accounts disabled by Exchange?

      "Which makes me wonder why admin accounts would have Exchange mailboxes."

      Because you can always trust people to find unnecessary ways of fucking things up...

  7. Anonymous Coward
    Anonymous Coward

    So could the AD be restored from backup?

    Could the users not authenticate to Azure AD in the cloud and then temporarily upgrade the account to allow back syncing to the on Premises AD (of course the cloud AD may already have be synced and had all the accounts deleted, but you should be able to recover those through the cloud portal)?

    1. katrinab Silver badge

      "So could the AD be restored from backup?"

      Not really, no.

      Only if you restore every other machine on the network from a backup taken at the exact same time, and even then, it isn't guaranteed to work.

      1. Anonymous Coward
        Anonymous Coward

        Why would you need to restore every machine on the network? There is no requirement for every machine to be in a certain synchronised state otherwise you couldn't restore any machine or re-image one or add a new machine used by the same user. I've restored ADs into test networks without issue previously.

        I can see you've put it as a joke, but I don't understand it.

        1. katrinab Silver badge

          If you restore one of your domain controllers, it will resync with the other one so you don't lose any of the changes you made, and that will work.

          In this scenario, you want to lose something, ie the deletions you made, and trying to roll the whole thing back in time, that probably isn't going to work.

  8. chivo243 Silver badge

    Late to the party...

    On vacation, and just read this:

    The team were sort of right, as Ben explained: "We did not know that removing the users from Exchange would automatically disable the users in Active Directory."

    A M$ Partner made this happen?!!! WTF!!! I haven't used Exchange in 14 years, and I knew this! Might have been a better test to unplug the network cable on EXchange and test?

    So glad Exchange was shown the door at our place...

    1. DougMac

      Re: Late to the party...

      Ditto. I find it questionable that a MS Partner wouldn't know that Exchange digs so deep into ActiveDirectory that touching anything in Exchange is the same as touching it in the main Active-Directory?

    2. katrinab Silver badge

      Re: Late to the party...

      If you delete a user in one place in Active Directory, you delete them everywhere. That's the whole point of Active Directory and the USP touted by Microsoft when they introduced it. The usual reason for deleting a user is because they have left the company, and you don't want ex-employees to still have access to certain parts of the system because you forgot to delete them everywhere.

    3. Trixr

      Re: Late to the party...

      I don't know how the hell these morons were allowed to be anywhere near Exchange. Given their apparent skill level, this should have been run under a strict AD split-permissions model.

      There's nothing wrong with Exchange itself, since setting up the permissions appropriately is a well-known procedure. But, if you're an Exchange admin, how can you not know you can affect every single mail-enabled object in the directory with the default configuration?

  9. Doctor Syntax Silver badge

    I'd have thought this is the sort of operation that must happen fairly often, certainly one that Microsoft want to happen fairly often. OTOH it's probably one that individual admins only do rarely and in any case there's always a first time. And it's one that seems high risk.

    Taking those three together why haven't Microsoft automated it?

  10. J. Cook Silver badge

    Well, to play devil’s advocate, the wording within exchange (both the ECP **and** EAC) are misleading. You only do it _once_ before writing down a sticky note or putting in a cheat sheet on how to do it correctly...

    1. Jou (Mxyzptlk) Silver badge

      Sooooo true. "Delete Mailbox" = AD delete user as well WHATFUUUUCKNO!.

      I made that "mistake" too. Once. And cursed Microsoft for using that wording.

      It is just like GPOs, where you have to "Activate" a "Do not Disable" function. Who makes such stupid expressions?

  11. M-AD

    Separate Accounts

    And that is one reason why you have your user account, and a separate Domain Administration account for Administrative tasks...

  12. Domquark

    Backup, Backup, Backup

    And this is the reason that you make sure you have a full and working backup before doing anything.

    Especially with Exchange.

  13. skuba*steve

    TL:DR - IT team implement major systems change without testing or a serviceable backout plan.

    Did they all get sacked? Because they should have all been sacked.

  14. HighTension

    One more reason to avoid exchange then?

    I can't even begin to imagine how that upgrade path was allowed. Glad I've never had to use it, and have always administrated standards-based IMAP and SMTP systems that just used AD, as, well, a directory!

  15. Trixr

    Maybe get someone who knows about the product

    Since Exchange 2000, it's been tightly integrated with AD. In fact, it was one of the primary drivers for the invention of AD. You don't have to maintain a separate x500 directory any more - it's baked in.

    These idiots should not have been anywhere near Exchange if they did not understand the close connection between AD and Exchange user objects. A mailbox is simply a database object - nearly everything to do with it is configured as a user property in AD. Whether that mailbox is located on-prem or Exchange Online. So yes, deleting the user object from Exchange DELETES THE USER OBJECT in AD, assuming you're using the default OOB permissions model. Which, guess what, were developed in the late 90s to allow admins to manage mailboxes and users seamlessly from the same directory (since user admins almost by default became the Exchange admins - you put in AD to have Exchange!)

    Now, there is such a thing as a split-permissions model that can be done in two basic flavours. 1. complete separation of AD admin from Exchange - you cannot alter AD user objects from the Exchange console. Rare, especially early on, when of course your Exchange and AD admins were the same people. There's also what MSFT calls the "RBAC" model, where certain objects can be managed by the Exchange console to varying degrees. Such as the ability to update distribution group memberships but not delete user objects.

    It's quite flexible and granular and the product is actually pretty capable in that respect. IF YOU KNOW HOW IT WORKS.

    The fact that these numpties didn't even know that the accounts synched to Azure AD (and the mailboxes in Exchange Online) are linked to the AD accounts is jaw-droppingly stupid.

    Also, by the way, if you have an on-premises AD, you MUST HAVE ONE EXCHANGE SERVER IN HYBRID MODE ON PREM. I had this argument with MSFT a few months back, since we sync the accounts with AAD Connect (which purportedly syncs back user and group email attributes to on-prem AD), but there are some magical incantations you still need an Exchange server for, even if it's a tiny VM (and has no mailboxes or SMTP traffic). I'm not going to summarise why here, but just be aware.

    Finally, the thing about the "Linux backup" being able to restore the AD accounts is a frigging ridiculous point. It doesn't matter what OS the backup runs on if it's AD-aware. Yes, AD is a customised LDAP database, and if the product can read the objects, schema and perms with its action account, big whoop-de-do. It worked as it should. Whether running on VMS, Linux, Solaris or a non-domain-joined Windows server.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like