back to article Getting that syncing feeling after an Exchange restore

It's Monday, and this week's column contains another reminder to check that those backups really have worked in an unfortunately synchronized episode of Who, Me? Our tale comes from a reader we'll call "John" (because that is not his name) and takes place in the glory days of Windows NT 4 and Microsoft Exchange Server 5.5. …

  1. Korev Silver badge
    Coat

    "Except the sync settings in the Palm software said 'take the most recent'."

    "What's newer than Friday's full mailbox? Today's empty one"

    Looks like a poor Outlook for John...

    1. ShadowSystems

      *FacePalmHappyGroan*

      I'd like to exchange that pun for a different one. Please server me up a better one? Thanks. =-)p

      1. Will Godfrey Silver badge

        Re: *FacePalmHappyGroan*

        Well it certainly left Windows of opportunity.

    2. Fred Daggy Silver badge
      FAIL

      Much like Financial Controller of anonymous company "ACME Corp". New iphone fired up, restored from iPhone backup, sync with Exchange - killed every meeting in the calendar. Most contacts lost things like birth dates and email addresses.

      Restore of the iPhone overwrote all the details in the Exchange calendar and contacts with its own copy. Meetings were still there, but without an attendence list to know who called it, who had accepted and who hadn't. No attachments either so documents were lost. Because they weren't on the file server. Also, company info, birthdates, addresses were all shorn from their owner. Business contacts of a lifetime that the Bean Counter could no longer send a birthday card too. Oh the humanity.

      Cue CIO now racing to yours truly to do an urgent mailbox restore. To which I replied - you know that server that finance knocked back, that we could have restored the tape too? Yeah, that's where we would have space for the Exchange server - unless you want to take us offline for a week. Without it, we can't do it. 3 month lead time.

      1. Doctor Syntax Silver badge

        It doesn't happen often, but when it does...

  2. TonyJ

    Exchange...

    ...was never easy to back up and restore. I remember when the first "bricks level" backup utilities arrived. Ah yes...years of a product before you had any easy way to restore a single email from backup.

    I haven't worked hands on with Exchange for years, and even then the most recent was moving from on-prem to hosted online. And in my personal opinion, it's one service that is better looked after by someone else. :-)

    1. Peter2 Silver badge

      Re: Exchange...

      Exchange is still no better.

      The improvements have been in backup software (eg Veeam). I'm a particular fan of "instant recovery" where everything instantly runs from the backup storage with an option to migrate to the production drives allowing for an effectively instantaneous restore.

      1. TonyJ

        Re: Exchange...

        As I say it has been a long time but you did remind me that before I did any work on an Exchange server that might cause any harm* I would insist on an offline backup being performed first.

        *Which was pretty much anything, really.

    2. katrinab Silver badge
      Paris Hilton

      Re: Exchange...

      In Outlook, export each individual mail box to a pst file.

      Do this before setting up any new servers

      Connect to the new server, and make sure it is an actual new server with a new name.

      Use Outlook to import the mail box.

      May not be the most efficient way to do things, but it does actually work.

      1. TonyJ

        Re: Exchange...

        I knocked up a powershell script back in the day to do just this - worked from Exchange 2007+ and it was a great last resort backup.

        There used to be limits in PST file size in Outlook though which made it difficult if the mailbox was over a certain size.

        1. Luiz Abdala
          FAIL

          Re: Exchange...

          The limit was exactly 2GB.

          And some people wouldn't even bother with the paltry 20MB on the server, they would go straight to a PST file on their machines, back in Windows 98 era...

      2. Jou (Mxyzptlk) Silver badge

        Re: Exchange...

        Don't export with Outlook do it on the Exchange server. Yes, the Exchange server can export a mailbox to .PST. Important detail: It includes deleted items which were not yet purged from the database, default 14 days, which is good for untrusty workers which just got fired and killed all their outlook content right before so noone can look what weird thing he's mailed around using the company address. I usually set this purge setting to 30 days, BTW. Mostly since people ask for mails deleted 15 days ago, I go to their Outlook and show the "recover deleted items" thing.

  3. MiguelC Silver badge

    I just can't understand why did John lose the HD, did he reinstall Windows from scratch? If so, why? Replacing a motherboard for another one in most cases just requires some new drivers installed after booting from the windows recovery disk. I did several of those in my days as one-man IT support team.

    1. Lazlo Woodbine

      If this was NT days, a new MB would usually mean a fresh install to keep things stable

      1. gryphon

        Did a new motherboard not often need a new HAL?

        Don't think changing the HAL was supported without a rebuild.

        But then again there wasn't any way to convert a domain controller to a member server either but there was a lovely piece of software called UPromote that could do it.

        Completely unsupportable by MS but solved a problem so all good.

  4. wolfetone Silver badge

    "Alas. Turns out every single one of the 36 backup tapes were corrupt.""

    At which point you just have to think about what you have done to piss off Sir John Cosmos* and wonder whether his plan of ruining your week will just stop at these corrupted backups or will he do his best to knobble your boiler on a cold morning as well.

    * the universe - as it feels impolite to call the universe the universe without giving it it's proper title.

    1. druck Silver badge

      A company I worked for in the 90s needed to restore their Exchange sever and found every single backup was corrupt too. It turns out Exchange kept it's database open and the backup software couldn't cope with that. So just like the article, it was start again from a clean sheet.

      1. Fred Daggy Silver badge

        MSBackup that was installed with NT4 was overwritten by an "Exchange aware" version, on the local machine. It could cope with an online backup. But, backing up, say, a mapped network drive of the Exchange Server was a no-no. I saw that surprisingly often, as it was one of the ways that things could happen in those days.

        Of course, the remote version of MSBackup would not have been able to cope.

        Offline Backup was of course, the dogs unmentionables.

      2. Jou (Mxyzptlk) Silver badge

        A sort-of-solution: Stop exchange information store, run backup, start information store. Later, when Snapshots were available you could do: Stop store, create snapshot, backup snapshot, and while the backup is running start exchange information store.

        Not elegant, but worked stable.

    2. Paul Cooper

      At which point you just have to think about what you have done to piss off Sir John Cosmos* and wonder whether his plan of ruining your week will just stop at these corrupted backups or will he do his best to knobble your boiler on a cold morning as well.

      Amd just to prove that the perversity of the Universe tends to a maximum, my central heating recently jammed ON! Fortunately before the peak of the heatwave, but it was still excessively toasty around my place until the engineer came. And yes, I could have killed it entirely - but the hot water was on the same system!

  5. ColinPa

    Of course the backups work

    I did a health check at a customer's site, with a standard list of questions

    Q:Do you backup? A:yes..

    Q:How do you restore? A:We get John to do it.

    Q:Where's John? A: He's on vacation this week... Oh he's turned his phone off.

    Q:Can you show me the evidence that the backup/restore has worked?

    A:Yes look. Oh Shot. We've been backing up and test-restoring the prototype database - not production!

    They had moved from an evaluation system to production, and had not changed the backup jobs.

    I added another question to my checklist

    Q:Are you backing up the correct things?

    1. Pascal Monett Silver badge

      One more question : have you actually tried restoring the backup ?

      1. Doctor Syntax Silver badge

        I think Q4 of the OP's list is better. "Have you tried?" can be answered by "Yes" even if it failed. It can be answered by "Yes" untruthfully even if no attempt was made. Asking for evidence it succeeded can only be ansered satisfactorily if the backup can be restored.

        1. Gerhard den Hollander

          When was the last time you tested your backups by doing a restore ?

          Tell me about that .....

          ....

          If I accidentally deleted a folder , how long would it take to restore ?

          Shall we try ?

          1. Doctor Syntax Silver badge

            Long time retired so all I have to worry about are my own files. And, yes, in the last few weeks I have had to reclaim some from the Nextcloud server which is, effectively, the backup for my laptop. And this very day I was checking & decided it needs more disk but that can wait for a couple of weeks.

            In the past distant past we had DR contracts with provision for rehearsals which included full backup restores.

          2. katrinab Silver badge
            Paris Hilton

            With volume snapshots, about 5 seconds plus disk io time.

            But that isn't a test of the backup.

          3. Killfalcon Silver badge

            Due to an unfortunate combination of access rights and fat fingers, in one prior job I was one of the only people to use a particular backup service - one we'd specially commissioned due to the value of the Actuarial data involved.

            Because I knew how to request the backups, I was sometimes asked to raise a ticket for someone else when they needed it (which meant the restore ticket was in my name). This worked out pretty well - I knew exactly what the restore team needed to know, and that meant the tickets got done quickly, which made people happy.

            So, anyway, one day the service manager turned up at my desk and informed me, in a very good natured fashion, that I'd 'cost' the company over 200k. How?

            Because out of 20 uses of that backup system over 4 years, 18 had my name on the ticket, so the cost-breakdown sheet assigned it all to my user ID. The backup regime cost us that much because, well, it was 2009, and daily backups of hundreds of terabytes weren't trivial...

            1. Jou (Mxyzptlk) Silver badge

              I hope you got him down to the ground, and maybe included his superior in this talk. I really really hate that type of premature conclusion type.

              1. Killfalcon Silver badge

                Oh, this was all good natured joshing. He was well aware of the reality, and thought I'd find the way the spreadsheets broke down the costs funny (and I did).

      2. Sgt_Oddball

        Schrodingers backup...

        You never really want to be the one to collapse the wave function...

        1. Eclectic Man Silver badge

          re: "Schrodingers backup..."

          Sgt_Oddball, I like your style. Reminds me of something a former work colleague said to me once:

          "Write only memory".

          (True, so no 'joke alert' icon.)

          1. TimMaher Silver badge
            Windows

            Re: “Write only memory”

            Back in the early eighties I proposed a new OS called WOMBATS.

            It was a Write Only Memory Batch And Transaction System.

            That was in the days when batches were common place and background compared to fancy new transaction systems.

            Some of the unusual features were that you could use unary maths (another of my ideas that never took off) with a single key keyboard, it didn’t matter if you pressed the key or not. You also had a single pixel monitor and it didn’t matter if it was lit or not.

            1. Doctor Syntax Silver badge

              Re: “Write only memory”

              "You also had a single pixel monitor and it didn’t matter if it was lit or not."

              Or even plugged in and switched on.

            2. John Brown (no body) Silver badge

              Re: “Write only memory”

              "Some of the unusual features were that you could use unary maths (another of my ideas that never took off) with a single key keyboard, it didn’t matter if you pressed the key or not. You also had a single pixel monitor and it didn’t matter if it was lit or not."

              Apple did that with their mouse. So only the keyboard and screen to try now :-)

          2. herman

            Re: re: "Schrodingers backup..."

            The Signetics WOM: https://www.baldengineer.com/signetics-write-memory-25120-datasheet.html

            1. Doctor Syntax Silver badge

              Re: re: "Schrodingers backup..."

              Signetics - now I remember them.

          3. Anonymous Coward
            Anonymous Coward

            Re: re: "Schrodingers backup..."

            I wonder if that might've been me. Unfortunately we'll never know.

            1. Eclectic Man Silver badge

              Re: re: "Schrodingers backup..."

              AC: "I wonder if that might've been me. Unfortunately we'll never know."

              My lips are sealed ;o)

  6. Andy The Hat Silver badge

    That syncing feeling ...

    Which way is it syncing, how is it set up where and when to sync? Sync is either great or a vision of impending doom - it never goes "a bit" wrong.

    1. oiseau
      Facepalm

      Re: That syncing feeling ...

      Sync is either great or a vision of impending doom - it never goes "a bit" wrong.

      Took the words right out of my mouth.

      Brought back bad memories.

      No, don't ask. 8^/

      O.

    2. nintendoeats

      Re: That syncing feeling ...

      I once used a friends laptop to play stuff off my ipod. I don't remember exactly why this happened, but when she plugged in her ipod it wiped out all her music. I do remember that she didn't keep the actual music files on her computer, only on the ipod itself.

      I continue to maintain that this was basically Apple's fault, but nonetheless apologies were made.

      1. jonathan keith

        Re: That syncing feeling ...

        Entirely Apple's fault.*

        * in this instance. Other megacorps are equally guilty, just under different circumstances.

    3. Doctor Syntax Silver badge

      Re: That syncing feeling ...

      Always back up the original before syncing.

  7. El Duderino
    Mushroom

    It was the second half of the 90s

    I was working in $BIG_COMPANY's European IT organisation that managed several hundreds of servers. One of those servers was in Germany (I was not) and needed rebuilding to swap the existing discs with newfangled arrays. It was an extremely important server too, of course (bad things don't happen to crash-and-burn boxes) and a shitstorm would ensue should anything untoward happen to it.

    That Saturday morning everything looked fine, a customer engineer was onsite in Germany taking care of the hardware swaps. Diagnostics ran successfully, the result of the backup jobs had been checked, two full backups had been made, nothing could go wrong - famous last words.

    Rewind to a couple of months earlier: I had suggested to my boss to implement backup verification because 'a backup that cannot be restored isn't a backup'. The suggestion was put in the freezer because that would be too time consuming, and perhaps we'd revisit that idea later. We did indeed.

    The first restore failed. So did the second. Blood pressure was rising. My faithful onsite helper in Germany fetched tape set after tape set and in the wee wee hours of the morning we had a functioning box again, and I seem to remember that the data loss was marginal - if any.

    On Monday I had a little chinwag with my boss and told him that either we implement backup verification NOW or this was the last box I'd ever rebuild. Shortly after that we had the process in place. Those were the days.

  8. Anonymous Coward
    Anonymous Coward

    I remember Exchange Server 5.5 well. The BOFH and I lost so many weekends to that shitty piece of software back when I was a PFY. If you had the "regular" version, rather than the "super expensive" version, there was an artificial limit of 16GB for the mailbox database. As soon as it reached 16GB, it went offline, meaning that you couldn't even delete any emails to reduce it below the 16GB limit and make it work again.

    There were only three ways to fix it. First we had to backup the server in case things went wrong which took a couple of hours. Then either run the "compact datastore" program which took many hours and usually didn't work. Or just truncate the file manually and then run the the "repair datastore" program, which also took many hours. Or restore from the last backup before it went down. Whichever option we tried invariably resulted in some data loss.

    Why did it have to screw up so badly and generate data loss and necessitate hours of repair work when it hit this artificial limit? Couldn't it just have popped up a nice message saying "your datastore is full, please delete some emails"?

    We tried so hard to find a free open source replacement for Exchange, but there was a nasty circular dependency that Outlook only works properly with Exchange and Exchange only works properly with Outlook. And none of the Open Source mail clients had the convenient calendaring and scheduling functionality of Outlook.

    1. Pascal Monett Silver badge
      Trollface

      Re: Couldn't it just have popped up a nice message

      Yeah, but that would be actually thinking about the issue instead of just coding a dead end.

      And, if you think about issues, then you actually start to try thinking about solutions.

      Borkzilla is not about solutions.

      1. Anonymous Coward
        Anonymous Coward

        Re: Couldn't it just have popped up a nice message

        Well it did pop up an error message... on that CRT monitor that you left turned off to prevent it burning the 'everything is fine' screen into the phosphor

    2. Luiz Abdala
      FAIL

      PST files. Or PTSD files?

      I remember helping people in my old organization to scoot around Exchange limits by setting up private files and... those bloody things crashed if they got any larger than 2GB, being that Win 98 era.

      You could choose having just 16MB on the server, that you could see it anywhere, or... downloading everything into your machine and hope it doesn't ever go down.

      Everybody chose PST files. Some people managed 5 or 6 PST files at once.

      1. J. Cook Silver badge

        Re: PST files. Or PTSD files?

        ... and especially when the PST files are stored on a mapped network drive. And the end user is both a power user AND a packrat that insists on saving every email EVER, no matter how trivial or work related. And calls every other week for an hour plus long PST rebuild session because it's gotten corrupted from residing on the mapped drive when they undocked their laptop with Outlook open and trying to write to the PST file when the network connection goes from the wired connection of the dock to the laptop's wireless.

        Definitely PTSD files.

    3. Anonymous Coward
      Anonymous Coward

      Why .... why oh why did you have to mention

      compact datatstore

      Now I need a lie down .... and some refreshing beverages of choice ....

    4. DougMac

      In that time frame, there were *lots* of calendaring options.

      It just that all of them sucked so bad nobody wanted to use them. The users all demanded the wonky-ass Outlook calendaring as the only option they'd ever consider, hell to all the others.

      I never found the facination with Outlook. Still don't. Its like here, have a program that you'll beat yourself over the head with a brick with, and you'll *insist* its the only one you'll use.

      1. J. Cook Silver badge

        Groupwise was.... Not Bad, during it's hayday.

        Lotus Notes and Domino? I only ever used it at one place, and not for very long (I was a contractor assisting the permanent staff with system refreshes and the odd troubleshooting session), but what I saw of it wasn't a horror show that I've heard stories about. (at least the client end and not the server side of it.

        I've yet seen anything lately that does what Outlook does that's not Lotus Notes/Domino (now HCL Domino).

        1. Anonymous Coward
          Anonymous Coward

          greifwise...............

      2. Shred

        I never found the facination with Outlook

        Microsoft write “kinda sorta ok” software The company’s real skill is in marketing that software. To try to convince users that they could use anything other than an MS Office product in the 1990s was an exercise in self- flagellation.

        We’re currently seeing the same thing with Microsoft Teams. It’s like an unflushable turd. Horrible, inefficient software that can take 30 seconds to struggle to launch on an i7 with SSD. But all the users just have to use it, because amicrosoft’s marketing people have brainwashed them so. The world will end if they don’t get MS Teams RIGHT NOW!

    5. John Brown (no body) Silver badge

      "And none of the Open Source mail clients had the convenient calendaring and scheduling functionality of Outlook."

      That's the bit I never quite understand. Is this something that is demanded of a mail server or is it just something people have been managed into expecting by MS? Email and "calendaring" don't immediately spring to mind as instant happy bedfellows. It always feels a bit like shoe-horning database functions into a spreadsheet and treating that normal.

      1. Tom Womack

        If the purpose of email is mostly to organise meetings, then being able to send emails which are instantly accepted or rejected into a calendar, and where the 'pick a time slot where the recipients are all free' is a function of the mail client is incredibly useful.

        1. John Brown (no body) Silver badge

          "If the purpose of email is mostly to organise meetings,"

          That's quite an assumption. The corollary to that would be if one is spending so much time organising meetings, inviting to meetings and accepting invitations to meetings, then a dedicated meeting/calendaring system tuned specifically for that function would make more sense than shoe horning the function into another app. Clearly in the situation you describe, the email function is merely an addendum to a calendar :-)

      2. Trixr

        It used to be MS Mail (meh) and Schedule+, so merging those products was indeed a good thing to do. Also, resource calendars are a handy thing that were facilitated by the merge.

        While early Exchange was still pants, it got incrementally better over time, and I think Exchange 2016 was pretty great. At least before they started "encouraging" everyone into Exchange Online.

    6. Anonymous Coward
      Anonymous Coward

      the original ransomeware

      I have a hazy memory that the 16gb limit could be removed by simply upgrading the exchange server enterprise edition, all you needed was a blank cheque and a full (but soon to be empty) company bank account.

    7. Jou (Mxyzptlk) Silver badge

      > Couldn't it just have popped up a nice message saying "your datastore is full, please delete some emails"?

      Isn't there an evenlog entry for that? Cannot speak for Exchange 5.5, that was before my time. First contact was Exchange 2000, first real administration was exchange 2003. Ex2003 sends warnings to the eventlog about a "soon full" database.

  9. Mayday
    Windows

    This should be called “Me too”

    Not “Who me?”

    Exchange 5.5/NT4 was why I packed the MS crap in, started on Cisco, got into networking and wound up doing my CCIE.

  10. J.G.Harston Silver badge

    Yes, I've been bitten by that. Lost email server, recovered it, local backup sync'd to empty server. ARGH!!!!!!

    Luckily, I whipped the power straight out, imaged the disk, and recovered about 99% of the data from the image.

    1. Chris J

      I had a RAID array fail and the backup software then sync the empty mount point to the backup.

      :Facepalm:

  11. Arbuthnot the Magnificent

    Exchange

    2000 was OK, 2003 was pretty decent, everything was downhill from there.

    Don't know how many SBS 2003 machines I put in then ripped out again a few years later, but it was a lot...

    1. DougMac

      Re: Exchange

      And anything before that was guaranteed downtime rebuilding all the time.

      My favorite was the _bug_ that made a reboot a 5 hour affair on an exchange server.

      That didn't get fixed for half a decade.

  12. darklord

    remeber those days well

    Yep I've had the same happen to me, no matter what we did nothing worked apart from going back to three months prior emails and slowly resored each tape form the full then the incremental, luckily it was on a side of the system and a server which wasnt used much,

  13. Anonymous Coward
    Anonymous Coward

    I remember is setting it up for our head office, then luckily left it to some one else to deal with. At some time it went pear shaped and he had lost the original install media (or it was damaged), so sent him my copy. More fun and games as both DVDs were Exchange 5.5, but they were slightly different versions.....

    (I worked in a small remote office, so we weren't allowed expensive software like that. Our email server was a piece of a shareware!)

  14. Prst. V.Jeltz Silver badge
    Windows

    The word sync gives me the chills

    Except the sync settings in the Palm software said 'take the most recent'.

    Call me a dinosaur but I have never wanted to understand or use "sync" ing since it first started becoming foisted upon me when windows 95 started getting clever, wether it was mail , or files , or whatever.

    I just stuck with , "Look I'll copy my shit from A to B myself thankyou ". I mean .. How does the machine know which pile of stuff is the "master" pile? Which leads to exactly the situation in this story - its my nightmare come true. Vindication!

    I have recently been forced to embrace it , but the above is still a disaster waiting to happen in some form imho

    1. Phil O'Sophical Silver badge

      Re: The word sync gives me the chills

      Oh, for the days when disks had write-protect switches...

    2. Anonymous Coward
      Anonymous Coward

      Re: The word sync gives me the chills

      I too must sync, but I still take my own copies. The company admins might be able to force me to save stuff to the cloud, but they can't stop me from making a third copy. When (not if) the cloud takes a dump, I do not plan to be without my critical data.

  15. Plest Silver badge
    Facepalm

    Adages...

    "It's not a backup until it's restored."

    If you can't restore it then it's just a nice way of testing your network throughput to the backup kit and then checking tape/disk write heads still work!

    1. Doctor Syntax Silver badge

      Re: Adages...

      Actually you don't even test that the write heads work, just that the backup drive accepts data.

  16. aregross
    Alert

    Seems to me you needed to unmount the DB to get a proper backup. I think I remember doing that like once a month and yea, restoring .PST files. (Bless M$'s little heart)

    ...and the tapes were probably fine, I mean how could they *all* be corrupt! Prolly a switch missed in the backup or restore settings.

  17. chivo243 Silver badge
    Pint

    Buy this guy a beer!

    'No worries mate, it was all useless shit anyway.'

    I bailed on Exchange 2003, the amount of hand-holding (and hand wringing) got to be way too much, we moved on to a waaay better system, only to have it ripped out of our hands a few years later by a plan within plan... C Levels (does C stand for Clueless?) didn't like the system, was too difficult to use?!! It's freaking email, just like Exchange... Geesh! A focus group was assembled, all things were tried out against the heir apparent, GaFE! No matter how many people in the focus group liked the current system, the moderator? of the group pushed for GaFE, and his word was final. He left later that year too! Plans within plans...

    Beer icon for the guy quoted above, and FAIL for going to GaFE. Glad I moved on...

    1. david 12 Silver badge

      Re: Buy this guy a beer!

      University I knew was so idealogically motivated to leave Exchange that they went to Notes. That was so painful that they went back to Exchange.

      18 months later they tried again with GaFE. GaFE wasn't as good as Exchange, but by that stage it had improved enough that it was usable.

      1. Julian 8 Silver badge

        Re: Buy this guy a beer!

        Notes at the back end was great, good replication and easy to backup / restore.

        The Notes client was a little bit of a pain, but the main problem was the Mail Template was ancient and never updated. If you did the work internally you could get a really good client, but not many did.

        I got TUPE'd to IBM and we had to go back to Notes (we moved to MS, but kept a couple of regional servers with everyones old mail files on for archive reasons), which was not a real problem and I was interested ot see what they had done to the template. Nada, Zilch, Sweet FA. It was the old crap template.

        I soon passed our template around to our "company" and we went back to what we had when we still used Notes in anger.

        As for a restore. Easy, get the NSF back, tell the user where it was, let them open it alongside their current file, copy/paste what was needed

  18. TooOldForThisSh*t

    Too Big to Restore ?

    Back in the distant mists of time (late 80's) I was sent to a customer site to replace a failed hard drive in a Netware server. Easy job in those days: replace drive, restore bindery and restore data. Easy Peazy, right? Wrong. Everything went perfectly until the restore process skipped their customer database file. Retried multiple times from multiple tapes. Tried different versions of the tape software with no luck. Everything else restored perfectly except for that one database file. Called the tape drive/software support number and after their suggestions all failed, we were asked how large the db file was. Seems their software had a little "issue" with large files. Any file over 10Megabytes cannot be restored. WTF? Seriously? No luck at all restoring the most critical file on their system.

    Luckily their database admin had made a copy to his local hard drive yesterday. We stopped selling that brand of backup drives that same day.

  19. terry 1

    Good backups are one's that successfully test restore. I wonder if all the backups were totally incremental based but the original full backup was still needed and was lost in time. Maybe it ignored the weekly full. I also suspect circular logging would have helped the restore. it's something I have always enabled.

    I've been around Exchange from the beginning and it still gives me the willies anytime I update it, however now it's virtualised so a test export makes life so much easier.

    Been there at the weekend playing with eseutil and watching the painfully slow % bars, usually claiming back a few Gb of space

  20. DS999 Silver badge
    Holmes

    I knew where this was going

    As soon as I read "backup" and "back it up again" with no mention of verifying that the backups were restorable.

    1. Boris the Cockroach Silver badge
      Unhappy

      Re: I knew where this was going

      Restorable backups...... oh god

      It was bad enough getting the backups done in the first place........ and then

      You werent there man.... you dont know what it was like.....

  21. Anonymous Coward
    Happy

    On The Other hand...

    I must say that when you're in the situation of 'having the whole weekend', it does feel a lot better than having just eight hours while you're living it.

    And once you've had a bit of experience (well, some years of it), you can use that whole weekend to better effect.

    Even if the first time is a complete screw up.

  22. Hazmoid
    Windows

    Try Banyan email for a doozy

    Having worked for a Government department that used Banyan Vines ( because anything that is good enough for the US department of Defense must be secure enough for our piddly little Australian govvy department) I sympathise with you all. We had to get new versions of the software flown in (back in the days when the Internet was a glint in telecommunications company's eyes, and it was quicker to transport CDs than to try and download them) to get our server hardware upgraded at a remote site. They used Banyan Mail (character based but pretty cool in its day) and had a windows gui that hooked into it (running on Windows 3.11 Sharkmail?)

    So come stupid o'clock on Monday Morning and we are trying to install the server and restore the data, and finish about 730 am on Monday morning, only to have some scientist haul me over the coals because they were unable to access their latest batch of spam emails. At that stage I had been running on coffee and chocolate for ~ 20 hours and was in no mood to be polite. So I told her that the mail would be fixed when it was fixed and she would know about it when it finished. I then went to try and get some sleep at the hotel room I had not seen that weekend, only to have the owner start mowing the lawn outside !

    This is how accidents on the road happen, trying to drive when you have been awake for more than 24 hours and need to drive for another 3 hours to get home. At some point I think I pulled the ute I was driving into a truck bay and slept in the back for about an hour. Fortunately the parental home was a good hour closer so I went there and slept for 3 hours before driving home.

    Got to the office the next day and was immediately requested to report to the IT manager. She asked why she was having to deal with a call from some senior scientist about why her staff were so rude. I then went to town on her with the whole sorry story, and resolved to start looking for a new job that day.

    Before I left they had gone the whole Exchange route, with Banyan running on Windows NT for directory services. Some contractors made out like bandits on that contract!

  23. CrazyOldCatMan Silver badge

    When I was a Solaris admin..

    .. I *hated* Exchange 5.5..

    All of our network was Solaris/Sendmail except for one office that had mysteriously migrated to Exchange 5.5 overnight. Mail looked like it was still flowing so everyone relaxed.

    Until the director of that office complained that he wasn't getting some emails. And it seemed to be a random assortment that he was missing. This was pre major search engines so I spent a lot of time trying to debug the issue. I finally found a found on an obscure forum where someone was complaining about failed SMTP pipelining.

    Sendmail, on contacting the destination server, would parse the server capability header response and tailor the session accordingly. And Exchange 5.5, coming from that bastion of conformity to standards that late 1990's Microsoft was, proclaimed that it could use pipelinging (ie - one SMTP session could accept multiple emails). Which was fine except that it couldn't - or at least, not from sendmail.

    Fortunately, with sendmail being an insanely complex beast, it could be configured on a per-destination server to over-ride the declared capabilities and, once we told it to *not* use pipelining with that server, all the mails started being received. It took more traffic though and, since we were all on frame relay, more cost.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like