Getting that syncing feeling after an Exchange restore It's Monday, and this week's column contains another reminder to check that those backups really have worked in an unfortunately synchronized episode of Who, Me? Our tale comes from a reader we'll call "John" (because that is not his name) and takes place in the glory days of Windows NT 4 and Microsoft Exchange Server 5.5. …
Much like Financial Controller of anonymous company "ACME Corp". New iphone fired up, restored from iPhone backup, sync with Exchange - killed every meeting in the calendar. Most contacts lost things like birth dates and email addresses.
Restore of the iPhone overwrote all the details in the Exchange calendar and contacts with its own copy. Meetings were still there, but without an attendence list to know who called it, who had accepted and who hadn't. No attachments either so documents were lost. Because they weren't on the file server. Also, company info, birthdates, addresses were all shorn from their owner. Business contacts of a lifetime that the Bean Counter could no longer send a birthday card too. Oh the humanity.
Cue CIO now racing to yours truly to do an urgent mailbox restore. To which I replied - you know that server that finance knocked back, that we could have restored the tape too? Yeah, that's where we would have space for the Exchange server - unless you want to take us offline for a week. Without it, we can't do it. 3 month lead time.
...was never easy to back up and restore. I remember when the first "bricks level" backup utilities arrived. Ah yes...years of a product before you had any easy way to restore a single email from backup.
I haven't worked hands on with Exchange for years, and even then the most recent was moving from on-prem to hosted online. And in my personal opinion, it's one service that is better looked after by someone else. :-)
Exchange is still no better.
The improvements have been in backup software (eg Veeam). I'm a particular fan of "instant recovery" where everything instantly runs from the backup storage with an option to migrate to the production drives allowing for an effectively instantaneous restore.
In Outlook, export each individual mail box to a pst file.
Do this before setting up any new servers
Connect to the new server, and make sure it is an actual new server with a new name.
Use Outlook to import the mail box.
May not be the most efficient way to do things, but it does actually work.
Don't export with Outlook do it on the Exchange server. Yes, the Exchange server can export a mailbox to .PST. Important detail: It includes deleted items which were not yet purged from the database, default 14 days, which is good for untrusty workers which just got fired and killed all their outlook content right before so noone can look what weird thing he's mailed around using the company address. I usually set this purge setting to 30 days, BTW. Mostly since people ask for mails deleted 15 days ago, I go to their Outlook and show the "recover deleted items" thing.
I just can't understand why did John lose the HD, did he reinstall Windows from scratch? If so, why? Replacing a motherboard for another one in most cases just requires some new drivers installed after booting from the windows recovery disk. I did several of those in my days as one-man IT support team.
Did a new motherboard not often need a new HAL?
Don't think changing the HAL was supported without a rebuild.
But then again there wasn't any way to convert a domain controller to a member server either but there was a lovely piece of software called UPromote that could do it.
Completely unsupportable by MS but solved a problem so all good.
"Alas. Turns out every single one of the 36 backup tapes were corrupt.""
At which point you just have to think about what you have done to piss off Sir John Cosmos* and wonder whether his plan of ruining your week will just stop at these corrupted backups or will he do his best to knobble your boiler on a cold morning as well.
* the universe - as it feels impolite to call the universe the universe without giving it it's proper title.
MSBackup that was installed with NT4 was overwritten by an "Exchange aware" version, on the local machine. It could cope with an online backup. But, backing up, say, a mapped network drive of the Exchange Server was a no-no. I saw that surprisingly often, as it was one of the ways that things could happen in those days.
Of course, the remote version of MSBackup would not have been able to cope.
Offline Backup was of course, the dogs unmentionables.
At which point you just have to think about what you have done to piss off Sir John Cosmos* and wonder whether his plan of ruining your week will just stop at these corrupted backups or will he do his best to knobble your boiler on a cold morning as well.
Amd just to prove that the perversity of the Universe tends to a maximum, my central heating recently jammed ON! Fortunately before the peak of the heatwave, but it was still excessively toasty around my place until the engineer came. And yes, I could have killed it entirely - but the hot water was on the same system!
I did a health check at a customer's site, with a standard list of questions
Q:Do you backup? A:yes..
Q:How do you restore? A:We get John to do it.
Q:Where's John? A: He's on vacation this week... Oh he's turned his phone off.
Q:Can you show me the evidence that the backup/restore has worked?
A:Yes look. Oh Shot. We've been backing up and test-restoring the prototype database - not production!
They had moved from an evaluation system to production, and had not changed the backup jobs.
I added another question to my checklist
Q:Are you backing up the correct things?
Long time retired so all I have to worry about are my own files. And, yes, in the last few weeks I have had to reclaim some from the Nextcloud server which is, effectively, the backup for my laptop. And this very day I was checking & decided it needs more disk but that can wait for a couple of weeks.
In the past distant past we had DR contracts with provision for rehearsals which included full backup restores.
Due to an unfortunate combination of access rights and fat fingers, in one prior job I was one of the only people to use a particular backup service - one we'd specially commissioned due to the value of the Actuarial data involved.
Because I knew how to request the backups, I was sometimes asked to raise a ticket for someone else when they needed it (which meant the restore ticket was in my name). This worked out pretty well - I knew exactly what the restore team needed to know, and that meant the tickets got done quickly, which made people happy.
So, anyway, one day the service manager turned up at my desk and informed me, in a very good natured fashion, that I'd 'cost' the company over 200k. How?
Because out of 20 uses of that backup system over 4 years, 18 had my name on the ticket, so the cost-breakdown sheet assigned it all to my user ID. The backup regime cost us that much because, well, it was 2009, and daily backups of hundreds of terabytes weren't trivial...
Back in the early eighties I proposed a new OS called WOMBATS.
It was a Write Only Memory Batch And Transaction System.
That was in the days when batches were common place and background compared to fancy new transaction systems.
Some of the unusual features were that you could use unary maths (another of my ideas that never took off) with a single key keyboard, it didn’t matter if you pressed the key or not. You also had a single pixel monitor and it didn’t matter if it was lit or not.
"Some of the unusual features were that you could use unary maths (another of my ideas that never took off) with a single key keyboard, it didn’t matter if you pressed the key or not. You also had a single pixel monitor and it didn’t matter if it was lit or not."
Apple did that with their mouse. So only the keyboard and screen to try now :-)
I once used a friends laptop to play stuff off my ipod. I don't remember exactly why this happened, but when she plugged in her ipod it wiped out all her music. I do remember that she didn't keep the actual music files on her computer, only on the ipod itself.
I continue to maintain that this was basically Apple's fault, but nonetheless apologies were made.
I was working in $BIG_COMPANY's European IT organisation that managed several hundreds of servers. One of those servers was in Germany (I was not) and needed rebuilding to swap the existing discs with newfangled arrays. It was an extremely important server too, of course (bad things don't happen to crash-and-burn boxes) and a shitstorm would ensue should anything untoward happen to it.
That Saturday morning everything looked fine, a customer engineer was onsite in Germany taking care of the hardware swaps. Diagnostics ran successfully, the result of the backup jobs had been checked, two full backups had been made, nothing could go wrong - famous last words.
Rewind to a couple of months earlier: I had suggested to my boss to implement backup verification because 'a backup that cannot be restored isn't a backup'. The suggestion was put in the freezer because that would be too time consuming, and perhaps we'd revisit that idea later. We did indeed.
The first restore failed. So did the second. Blood pressure was rising. My faithful onsite helper in Germany fetched tape set after tape set and in the wee wee hours of the morning we had a functioning box again, and I seem to remember that the data loss was marginal - if any.
On Monday I had a little chinwag with my boss and told him that either we implement backup verification NOW or this was the last box I'd ever rebuild. Shortly after that we had the process in place. Those were the days.
I remember Exchange Server 5.5 well. The BOFH and I lost so many weekends to that shitty piece of software back when I was a PFY. If you had the "regular" version, rather than the "super expensive" version, there was an artificial limit of 16GB for the mailbox database. As soon as it reached 16GB, it went offline, meaning that you couldn't even delete any emails to reduce it below the 16GB limit and make it work again.
There were only three ways to fix it. First we had to backup the server in case things went wrong which took a couple of hours. Then either run the "compact datastore" program which took many hours and usually didn't work. Or just truncate the file manually and then run the the "repair datastore" program, which also took many hours. Or restore from the last backup before it went down. Whichever option we tried invariably resulted in some data loss.
Why did it have to screw up so badly and generate data loss and necessitate hours of repair work when it hit this artificial limit? Couldn't it just have popped up a nice message saying "your datastore is full, please delete some emails"?
We tried so hard to find a free open source replacement for Exchange, but there was a nasty circular dependency that Outlook only works properly with Exchange and Exchange only works properly with Outlook. And none of the Open Source mail clients had the convenient calendaring and scheduling functionality of Outlook.
I remember helping people in my old organization to scoot around Exchange limits by setting up private files and... those bloody things crashed if they got any larger than 2GB, being that Win 98 era.
You could choose having just 16MB on the server, that you could see it anywhere, or... downloading everything into your machine and hope it doesn't ever go down.
Everybody chose PST files. Some people managed 5 or 6 PST files at once.
... and especially when the PST files are stored on a mapped network drive. And the end user is both a power user AND a packrat that insists on saving every email EVER, no matter how trivial or work related. And calls every other week for an hour plus long PST rebuild session because it's gotten corrupted from residing on the mapped drive when they undocked their laptop with Outlook open and trying to write to the PST file when the network connection goes from the wired connection of the dock to the laptop's wireless.
Definitely PTSD files.
In that time frame, there were *lots* of calendaring options.
It just that all of them sucked so bad nobody wanted to use them. The users all demanded the wonky-ass Outlook calendaring as the only option they'd ever consider, hell to all the others.
I never found the facination with Outlook. Still don't. Its like here, have a program that you'll beat yourself over the head with a brick with, and you'll *insist* its the only one you'll use.
Groupwise was.... Not Bad, during it's hayday.
Lotus Notes and Domino? I only ever used it at one place, and not for very long (I was a contractor assisting the permanent staff with system refreshes and the odd troubleshooting session), but what I saw of it wasn't a horror show that I've heard stories about. (at least the client end and not the server side of it.
I've yet seen anything lately that does what Outlook does that's not Lotus Notes/Domino (now HCL Domino).
I never found the facination with Outlook
Microsoft write “kinda sorta ok” software The company’s real skill is in marketing that software. To try to convince users that they could use anything other than an MS Office product in the 1990s was an exercise in self- flagellation.
We’re currently seeing the same thing with Microsoft Teams. It’s like an unflushable turd. Horrible, inefficient software that can take 30 seconds to struggle to launch on an i7 with SSD. But all the users just have to use it, because amicrosoft’s marketing people have brainwashed them so. The world will end if they don’t get MS Teams RIGHT NOW!
"And none of the Open Source mail clients had the convenient calendaring and scheduling functionality of Outlook."
That's the bit I never quite understand. Is this something that is demanded of a mail server or is it just something people have been managed into expecting by MS? Email and "calendaring" don't immediately spring to mind as instant happy bedfellows. It always feels a bit like shoe-horning database functions into a spreadsheet and treating that normal.
"If the purpose of email is mostly to organise meetings,"
That's quite an assumption. The corollary to that would be if one is spending so much time organising meetings, inviting to meetings and accepting invitations to meetings, then a dedicated meeting/calendaring system tuned specifically for that function would make more sense than shoe horning the function into another app. Clearly in the situation you describe, the email function is merely an addendum to a calendar :-)
It used to be MS Mail (meh) and Schedule+, so merging those products was indeed a good thing to do. Also, resource calendars are a handy thing that were facilitated by the merge.
While early Exchange was still pants, it got incrementally better over time, and I think Exchange 2016 was pretty great. At least before they started "encouraging" everyone into Exchange Online.
> Couldn't it just have popped up a nice message saying "your datastore is full, please delete some emails"?
Isn't there an evenlog entry for that? Cannot speak for Exchange 5.5, that was before my time. First contact was Exchange 2000, first real administration was exchange 2003. Ex2003 sends warnings to the eventlog about a "soon full" database.
I remember is setting it up for our head office, then luckily left it to some one else to deal with. At some time it went pear shaped and he had lost the original install media (or it was damaged), so sent him my copy. More fun and games as both DVDs were Exchange 5.5, but they were slightly different versions.....
(I worked in a small remote office, so we weren't allowed expensive software like that. Our email server was a piece of a shareware!)
Except the sync settings in the Palm software said 'take the most recent'.
Call me a dinosaur but I have never wanted to understand or use "sync" ing since it first started becoming foisted upon me when windows 95 started getting clever, wether it was mail , or files , or whatever.
I just stuck with , "Look I'll copy my shit from A to B myself thankyou ". I mean .. How does the machine know which pile of stuff is the "master" pile? Which leads to exactly the situation in this story - its my nightmare come true. Vindication!
I have recently been forced to embrace it , but the above is still a disaster waiting to happen in some form imho
I too must sync, but I still take my own copies. The company admins might be able to force me to save stuff to the cloud, but they can't stop me from making a third copy. When (not if) the cloud takes a dump, I do not plan to be without my critical data.
Seems to me you needed to unmount the DB to get a proper backup. I think I remember doing that like once a month and yea, restoring .PST files. (Bless M$'s little heart)
...and the tapes were probably fine, I mean how could they *all* be corrupt! Prolly a switch missed in the backup or restore settings.
'No worries mate, it was all useless shit anyway.'
I bailed on Exchange 2003, the amount of hand-holding (and hand wringing) got to be way too much, we moved on to a waaay better system, only to have it ripped out of our hands a few years later by a plan within plan... C Levels (does C stand for Clueless?) didn't like the system, was too difficult to use?!! It's freaking email, just like Exchange... Geesh! A focus group was assembled, all things were tried out against the heir apparent, GaFE! No matter how many people in the focus group liked the current system, the moderator? of the group pushed for GaFE, and his word was final. He left later that year too! Plans within plans...
Beer icon for the guy quoted above, and FAIL for going to GaFE. Glad I moved on...
University I knew was so idealogically motivated to leave Exchange that they went to Notes. That was so painful that they went back to Exchange.
18 months later they tried again with GaFE. GaFE wasn't as good as Exchange, but by that stage it had improved enough that it was usable.
Notes at the back end was great, good replication and easy to backup / restore.
The Notes client was a little bit of a pain, but the main problem was the Mail Template was ancient and never updated. If you did the work internally you could get a really good client, but not many did.
I got TUPE'd to IBM and we had to go back to Notes (we moved to MS, but kept a couple of regional servers with everyones old mail files on for archive reasons), which was not a real problem and I was interested ot see what they had done to the template. Nada, Zilch, Sweet FA. It was the old crap template.
I soon passed our template around to our "company" and we went back to what we had when we still used Notes in anger.
As for a restore. Easy, get the NSF back, tell the user where it was, let them open it alongside their current file, copy/paste what was needed
Back in the distant mists of time (late 80's) I was sent to a customer site to replace a failed hard drive in a Netware server. Easy job in those days: replace drive, restore bindery and restore data. Easy Peazy, right? Wrong. Everything went perfectly until the restore process skipped their customer database file. Retried multiple times from multiple tapes. Tried different versions of the tape software with no luck. Everything else restored perfectly except for that one database file. Called the tape drive/software support number and after their suggestions all failed, we were asked how large the db file was. Seems their software had a little "issue" with large files. Any file over 10Megabytes cannot be restored. WTF? Seriously? No luck at all restoring the most critical file on their system.
Luckily their database admin had made a copy to his local hard drive yesterday. We stopped selling that brand of backup drives that same day.
Good backups are one's that successfully test restore. I wonder if all the backups were totally incremental based but the original full backup was still needed and was lost in time. Maybe it ignored the weekly full. I also suspect circular logging would have helped the restore. it's something I have always enabled.
I've been around Exchange from the beginning and it still gives me the willies anytime I update it, however now it's virtualised so a test export makes life so much easier.
Been there at the weekend playing with eseutil and watching the painfully slow % bars, usually claiming back a few Gb of space
I must say that when you're in the situation of 'having the whole weekend', it does feel a lot better than having just eight hours while you're living it.
And once you've had a bit of experience (well, some years of it), you can use that whole weekend to better effect.
Even if the first time is a complete screw up.
Having worked for a Government department that used Banyan Vines ( because anything that is good enough for the US department of Defense must be secure enough for our piddly little Australian govvy department) I sympathise with you all. We had to get new versions of the software flown in (back in the days when the Internet was a glint in telecommunications company's eyes, and it was quicker to transport CDs than to try and download them) to get our server hardware upgraded at a remote site. They used Banyan Mail (character based but pretty cool in its day) and had a windows gui that hooked into it (running on Windows 3.11 Sharkmail?)
So come stupid o'clock on Monday Morning and we are trying to install the server and restore the data, and finish about 730 am on Monday morning, only to have some scientist haul me over the coals because they were unable to access their latest batch of spam emails. At that stage I had been running on coffee and chocolate for ~ 20 hours and was in no mood to be polite. So I told her that the mail would be fixed when it was fixed and she would know about it when it finished. I then went to try and get some sleep at the hotel room I had not seen that weekend, only to have the owner start mowing the lawn outside !
This is how accidents on the road happen, trying to drive when you have been awake for more than 24 hours and need to drive for another 3 hours to get home. At some point I think I pulled the ute I was driving into a truck bay and slept in the back for about an hour. Fortunately the parental home was a good hour closer so I went there and slept for 3 hours before driving home.
Got to the office the next day and was immediately requested to report to the IT manager. She asked why she was having to deal with a call from some senior scientist about why her staff were so rude. I then went to town on her with the whole sorry story, and resolved to start looking for a new job that day.
Before I left they had gone the whole Exchange route, with Banyan running on Windows NT for directory services. Some contractors made out like bandits on that contract!
.. I *hated* Exchange 5.5..
All of our network was Solaris/Sendmail except for one office that had mysteriously migrated to Exchange 5.5 overnight. Mail looked like it was still flowing so everyone relaxed.
Until the director of that office complained that he wasn't getting some emails. And it seemed to be a random assortment that he was missing. This was pre major search engines so I spent a lot of time trying to debug the issue. I finally found a found on an obscure forum where someone was complaining about failed SMTP pipelining.
Sendmail, on contacting the destination server, would parse the server capability header response and tailor the session accordingly. And Exchange 5.5, coming from that bastion of conformity to standards that late 1990's Microsoft was, proclaimed that it could use pipelinging (ie - one SMTP session could accept multiple emails). Which was fine except that it couldn't - or at least, not from sendmail.
Fortunately, with sendmail being an insanely complex beast, it could be configured on a per-destination server to over-ride the declared capabilities and, once we told it to *not* use pipelining with that server, all the mails started being received. It took more traffic though and, since we were all on frame relay, more cost.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2025
