back to article Scanning an Exchange server for a virus that spreads via email? What could go wrong?

Just like clockwork, another weekend is over and Monday is here again. To lighten the load, El Reg is offering you the latest instalment of Who, Me?, our weekly sysadmin confessional column. This time we meet "Romeo", who was working at a large music company in London at the time in question. It was his first job for a big …

  1. Giovani Tapini

    There is a little bit of me

    That thinks the actions could be close to doing the right thing?

    Although a company culturally can create a great dependence on email as a business tool for variously archiving, signoff approvals, support messages, and other unwise use-cases.

    At least the server was clean, and probably caught other things too.

    My worry would be how the messages got to this point without being cleaned already...

    1. Anonymous Coward
      Anonymous Coward

      Re: There is a little bit of me

      No, the server was not clean. It didn't clean the mail database, it just deleted it, so a previous copy had to be restored - it still contained all the bad emails the previous one contained but the last day.

      And why a company should not rely on email? Should they install pneumatic mail, and have people with carts going around desks bring paper documents to read and sign? Companies always relied on internal document shifting...

      1. Giovani Tapini

        Re: There is a little bit of me

        @LDS Fair call, but the article does not go into much detail on what was done with the restore. I assume some tool more suitable for cleaning mailboxes was deployed. Server was therefore clean albeit at the expense of deleting everything. The process did have an excess of collateral damage, but did remove the infection.

        Now about those carts, they used to be filled with snacks and coffee...

        1. Anonymous Coward
          Anonymous Coward

          Re: There is a little bit of me

          For what I read, it had big troubles restoring the mail database because in Exchange just replacing an older file is not enough - as database systems are usually picky when data files, logs and other things don't match. So, sure, he deleted the ILOVEYOU mails of that day - but whatever else was lurking there from previous days was still there.

          Cleaning such stuff exactly needs tools which are able to read the database correctly and clean infected messages one by one - but you usually need to have the mail database open and accessible to run them, because accessing the on-disk structure of such files - often undocumented - it's a very risky task.

          While in certain circumstance you may not have other options that wiping everything, running an AV against database files is usually a very bad idea - especially if the default actions is "delete".

          1. heyrick Silver badge

            Re: There is a little bit of me

            especially if the default actions is "delete"

            He must have been really new to virus scanning. People with experience know that there are such things as false positives which are more often than not important documents and critical Windows DLLs... as such the only sensible action is to quarantine suspect files to ensure you aren't about to nuke something important, and if it needs to go (of it's a real virus and not a "this code looks odd" heuristic), you know what to replace from the installation media before the machine gets itself into a bluescreen-at-boot state.

            Never ever give an antivirus program the ability to automatically delete stuff...

      2. phuzz Silver badge
        Thumb Up

        Re: There is a little bit of me

        "didn't clean the mail database, it just deleted it, so a previous copy had to be restored"

        That is cleaning it. The same way I clean my car, by sandblasting all the paint off, and then re-painting ;)

        1. Waseem Alkurdi

          Re: There is a little bit of me

          @phuzz

          That is cleaning it. The same way I clean my car, by sandblasting all the paint off, and then re-painting ;)

          In this case, the backup contained the virus.

          This is like re-painting with the same old paint melted into a liquid form or something.

        2. Anonymous Coward
          Anonymous Coward

          "by sandblasting all the paint off, and then re-painting ;)"

          But if you have a corpse in the trunk, after the sandblasting and repainting, you will still have a corpse in the trunk...

          1. amanfromMars 1 Silver badge

            Re: "by sandblasting all the paint off, and then re-painting ;)"

            But if you have a corpse in the trunk, after the sandblasting and repainting, you will still have a corpse in the trunk... ..... LDS

            The I Love You Virus a Corpse in a Trunk whenever Phantom Bodied? I Don't Think So, Mes Amis/Mon Brave.

            Are you Bitten and Smitten and Rooting for More LOVE Bugs to Display the Bounty of ITs Wares with Immaculate Temptations to Sate and Supply ie Fully Realise in COSMIC Great Order for SMARTR TerraPhorming Nations?

            Try Contemplation of IT as Advanced IntelAIgent Driver/Virtual Mentor and Practical Monitor.

            One of those Funky Clunky NEUKlearer HyperRadioProACTive IT AIdPrograms Perfectly Suited for Princes and Princesses with Visions in Peril? ...... Saudi Vision 2030

            And there be Kings and Queens, Princes and Princesses, Nymphs and Satyrs Everywhere. And that Convenience makes More LOVE Bugs AIdPrograms Astronomically Wealthy and Certainly Worthy.

            1. Anonymous Coward
              Anonymous Coward

              Re: "by sandblasting all the paint off, and then re-painting ;)"

              I have no idea what is going on anymore ...

          2. Adam 1

            Re: "by sandblasting all the paint off, and then re-painting ;)"

            > ... if you have a corpse in the trunk, after the sandblasting and repainting, you will still have a corpse in the trunk...

            Asking for a friend?

      3. chivo243 Silver badge
        Pint

        Re: There is a little bit of me

        Buttle or Tuttle? Tubes going everywhichway! Love it!

    2. David Knapman

      Re: There is a little bit of me

      It's not like this one organization would even have been in the minority here, ILOVEYOU was a major wake up call for many orgs to put more work into their email scanning.

      Many orgs, if they had incoming scanners at all were just using signature based checks so of no use against a rapidly spreading worm based on social engineering.

      1. JimboSmith Silver badge

        Re: There is a little bit of me

        At the time of that particular virus spreading I was working for a firm doing some tech related planning. We were using outlook and hit quite hard when one just person was sent it. Whilst IT were cleaning everything the rest of the place descended on the local public houses. We had an afternoon spent doing not much except trying different beers etc. Then returned to the office to collect our things at 5:30 whereupon some people went back to the pub. IT support told us that they had cleaned all our mailboxes/computers and beefed up the mail filter.

        Everything was fine after that except a few months later when we all started to get these emails again. IT were a bit annoyed that it had made it through the mail filter The culprit was the Intern mailbox which was only used when we had an intern. There wasn't an intern at the time so the mailbox never got cleaned. One of the staff left immediately for the pub when the first email appeared. He was called back before he could order anything to drink.

  2. Evil Auditor Silver badge

    Sweet memories...

    I remember well when ILOVEYOU broke out on our university campus. It was that morning when I found an e-mail in my inbox with this love confession from a girl in administration. As sweet as she was, I had some mixed feelings about this - can't remember her name since we cruelly only called her The Nose. For obvious reason. Needless to say, our love relationship wasn't meant to last. Not longer than a few seconds anyway.

    I was safe with my Linux environment. And the next thing was to start up the Windows sandbox - no virtual machine at that time - and investigate what this charming virus actually did.

    1. sandman

      Re: Sweet memories...

      Yep, that's rekindled a few grey cells. We thought we were on top of it, repeated warnings sent out, 24pt, bold, underlined, bright red, dire threats and all. Then, one person in PR decided to completely ignore all that... When asked, they said, "Oh, I never read messages from IT, you're always just sending out warnings."

      1. tiggity Silver badge

        Re: Sweet memories...

        Exactly what you (sadly) expect from some users

      2. Nick Kew

        Re: Sweet memories...

        "Oh, I never read messages from IT, you're always just sending out warnings."

        The boy who cried Wolf springs to mind.

        Can't comment on your individual situation, but warnings are more effective if you pick your cases with some care to avoid overloading users with esoterica that'll only baffle them.

        1. Aladdin Sane

          Re: Sweet memories...

          This would be just post Y2K, which people put a lot of effort into fixing only for their efforts to be dismissed as scare-mongering.

          1. katrinab Silver badge

            Re: Sweet memories...

            And while there was stuff that needed to be fixed, a lot of it was scaremongering, especially related to embeded systems. For example, the idea that a washing machine might, on 01/01/00, think it was 1900, and, that as it hadn't been invented yet, it ought to shake itself to bits and then spontaneously combust.

            1. Adam 1

              Re: Sweet memories...

              > For example, the idea that a washing machine might, on 01/01/00, think it was 1900, and, that as it hadn't been invented yet, it ought to shake itself to bits and then spontaneously combust.

              ... And I would have gotten away with it if not for you pesky kids.

              1. John Brown (no body) Silver badge

                Re: Sweet memories...

                "... And I would have gotten away with it if not for you pesky kids."

                Should've bought a Scooby! :-)

          2. steviebuk Silver badge

            Re: Sweet memories...

            That's what I always point out to people. I wasn't involved in any fixes, I'd just finished college. But the amount of people that say "That Y2K thing was bollocks wasn't it. Nothing happened". Especially so called comedians. Yeah nothing happened because people worked fucking hard to fix the issues before they could happen. Tits.

            1. Andrew Moore

              Re: Sweet memories...

              I had a journalist approach me in 1998 looking for scary Y2K story- I told her that nothing was going to happen because we'd been on top of it for a number of years now and everything should be in place by then. Needless to say, she completely ignored me and found a nutjob that gave the doomsday scenario that she was looking for.

              1. John Brown (no body) Silver badge

                Re: Sweet memories...

                "Needless to say, she completely ignored me and found a nutjob that gave the doomsday scenario that she was looking for."

                It would have been funny if all those media types touting the scare stories had their own orgs IT fall over, but sadly their own IT people were also on top of things. Shame the journos didn't interview their own IT bods and got the real story.

            2. Mike 16

              Re: Sweet memories...

              @steviebuck

              While I agree that a lot of conscientious people worked a lot of hours in the run-up to Y2K, IIRC a patch for Windows believing 2000 would be a leap year came out in something like November 1999. This despite earlier complaints from fin-tech people that computations of future value or the like were odd. The thing is,, sometimes you don't just need to know what day today is, but what day 60 or 180 days from now will be.

              1. katrinab Silver badge

                Re: Sweet memories...

                2000 was a leap year. 1900 and 2100 are not.

                Most computers at the time assumed any year divisible by 4 is a leap year, when in fact years divisible by 100, but not divisible by 400 are not leap years.

                Excel still thinks 29th February 1900 is a valid date, and most other spreadshhets copy this bug for compatibility.

          3. Anonymous Coward
            Anonymous Coward

            Re: Sweet memories...

            Y2K was scare mongering.

            Had to be to get all the CEO's and bean counters on board to allow all the time and money to be spent to check all the software for problems. In the end the CEO's and bean counters all knew they would get blamed by the stock holders, customers and little kids on the street if their company had a Y2K problem.

            1. Unicornpiss
              Meh

              Y2K was scare mongering..

              I remember people freaking out because they thought their cars wouldn't start on 1/1/00. As though cars (at least of that era) cared what date it was, with the exception of somehow knowing when they're 1 week out of warranty..

              1. katrinab Silver badge
                Trollface

                Re: Y2K was scare mongering..

                I suppose the software that says, "I'm one week out of warranty, I'm going to shake myself to bits and spontaneously combust" might not work, and it might think it has to keep going for another 100+ years.

      3. Mark 85

        Re: Sweet memories...

        Then, one person in PR decided to completely ignore all that... When asked, they said, "Oh, I never read messages from IT, you're always just sending out warnings."

        Those types should be strung up by the front door as example and with an email or group meeting explanation. The second one who does it should be drawn and quartered.

  3. Nick Kew

    Still baffled

    ... at how noone sued MS for damages at the time.

    The means by which this email evaded detection in a simple and sensible email scanner was MS's deliberate breaking of MIME standards dating back to 1992. And the RFC even contains an informational section under the heading of security implications explaining exactly why what MS subsequently did would leave their users wide open to attack.

    1. heyrick Silver badge

      Re: Still baffled

      "what MS subsequently did would leave their users wide open to attack"

      You mean like how the created user profile for users on a home installation of XP had admin rights by default, and how the restricted user profile was so restricted it was near useless for many (you couldn't even change the time FFS). There was, I believe, a tool to tweak what rights users had, in the enterprise version...

      It was pretty much a wide open door back then, just marginally less open than Win32 machines.

    2. jake Silver badge

      Re: Still baffled

      "at how noone sued MS for damages at the time."

      Read the fine print. MS' code isn't even guaranteed to work as advertised when used as intended. It's use at your own risk, at least according to MS's own EULA. You HAVE read the EULA, and fully understand it, right? And your corporate lawyers have vetted it as OK for use by your business, right?

      1. heyrick Silver badge

        Re: Still baffled

        Read the licence blurb?

        Eight years ago a company demonstrated how many people bother reading all that rubbish: https://www.geek.com/games/gamestation-eula-collects-7500-souls-from-unsuspecting-customers-1194091/

  4. adam payne

    Scanning an Exchange server for a virus that spreads via email? What could go wrong?

    It deletes the EDB file and then lots of people shout at you.

    I would say how did the emails get that far anyway but this was back in 2000 anyway so I suppose that answers that.

    I've never deleted an EDB file but I have had an old boss add an extra drive to the Exchange server and then proceed to import the RAID config. Oops.

    1. jake Silver badge

      "I would say how did the emails get that far anyway but this was back in 2000 anyway so I suppose that answers that."

      By 2000, real MTAs had been dropping malware long before it got anywhere near userspace for over a decade. Milters (introduced in 2000) and the like made it easier to admin. Toys like Exchange were never really considered an option by professionals.

  5. Anonymous Coward
    Anonymous Coward

    I deleted an EDB file once by accident - and no backups then.

    Oops.

    Learnt a hard lesson then, never, ever again. It was not a fun experience, and one I don't wish on any Exchange admin.

    Anon because.

    Since that incident I've started to use ntbackup to back up the mail store - and funnily enough, that issue was never repeated.

  6. Anonymous South African Coward Bronze badge

    Remote Desktop in the ILOVEYOU virus era?

    1. GlenP Silver badge

      PC Anywhere and the like have been around since the mid 1980s. I was certainly using it (via modems) in around 1988.

    2. Waseem Alkurdi

      The first version of RDP shipped with Windows NT Terminal Server 4.0.

      1. Freddellmeister

        Well there was NCD wincenter and a few other options before RDP..

        1. jake Silver badge

          Wincenter was OK, at least for the thin client set. Surprisingly, DESQview/X was a rather good option for remote GUI support of Windows boxen using *nix as the admin box. Spendy, though.

    3. Evil Auditor Silver badge

      @ASAC

      Indeed. My earliest recollection of a remote desktop solution dates back to the mid 90s. And it wasn't exactly avant-garde back then either. It might have been with NT 4.0.

    4. DJV Silver badge

      Yes

      Very common! A short while after the ILOVEYOU outbreak I was working for a "famous insurance company" that was (mainly) based in and named after the city I lived in (Norwich). We used remote access from our base in the city centre to the servers in the "lights out" data centre server farms four miles away out on the outskirts. It was fantastic!

      Well...

      ...only for a weird value of "fantastic" that meant that...

      ...the lights in the server farm were never actualy out because remote access was bloody slow and, for "security", sessions were set to always time out after 15 minutes which, due to the slowness of the network in general and remote access in particular, meant that you barely got more than 20 mouse clicks and 10 fields filled in remotely before the whole thing shut down on you (if it had managed to stay up for the full 15 minutes, itself a rare feat). So, it was often quicker to catch the regular company-provided shuttle bus to the data centre and go and access the server non-remotely (hence the lights never being out as the data centre was full of pissed off people all doing the same non-remote thing).

    5. jake Silver badge

      telnet was standardized in 1973. NCSA's version ran on DOS in 1986. Wall Data's Rumba was in wide use on corporate Windows desktops by 1990.

      Maybe not RemoteDesktop[tm], but certainly remote desktop capability.

  7. defiler

    Restoring EDBs...

    One of our clients accidentally started a restore in Exchange. I think it was a block-level restore of the database rather than of a mailbox or folder - it was a while ago and I (luckily) wasn't there. When she realised her mistake she pulled the power on the email server...

    My colleague had to regedit the hell out of it to force the database out of restore mode, and then restore a complete copy of the database from before the errant command. I don't think that database was quite right ever again.

    Still, after I'd left that job, my ex-line-manager managed to torpedo the server nicely in a different way, but that's a story for another Monday...

    1. Tom 7

      Re: Restoring EDBs...

      We had trouble with Exchange Server 4.5. I remember running some DB repair program that took 4 or 5 hours to scan the DB half a dozen times to get it into a state that Exchange Server was happy to load.

      I was pissing about with some VB for some web app and discovered VB would allow you to dump a whole exchange DB and read everyone's emails and I think I could have used that to rebuild a corrupted DB better than the MS repair program but never have the courage to try it in action. Some nice reading while repairing the DB mind.

  8. Anonymous Coward
    Anonymous Coward

    +1 for nom de guerre "Romeo"

    (n/t)

  9. Anonymous Coward
    Anonymous Coward

    Thing of the past, thank god!

    It's Stories like this that make me relieved to get out of the email game having just shifted 10,000 users to O365 in the last couple of months....

    Having lived through a corrupt EDB file recovery on our Exchange environment last year, i know how terrifying email issues can be. No matter how often you spout the line "Its a communication tool, not a file server" at people it never stops them from retaining everything ever sent since God was a boy.....

    Also, micromanaging mailbox sizes on-prem is becoming harder and harder, users just do not get why you need to limit their mail to 2Gb and exceptions pop up all over the place (largest user mailbox we had was 80Gb ffs...)

    It doesn't help these days that File Shares and collab SharePoint seems to be a dirty word at C-Level, where they just want to send 100Mb PowerPoint Decks and Business plans to each other...and who are IT to tell them how to do their job.

    1. Anonymous Coward
      Anonymous Coward

      Re: Thing of the past, thank god! -users just do not get why you need to limit their mail to 2Gb

      Oh, I remember that horror and the VIP who filled his mailbox (mostly with duplicate attachments) and wondered why he wasn't getting any email. Most of it had been deleted off the server.

      We set up a local mail server with him and a dummy account and then copied all of his emails, in blocks, over to the dummy account. After each block we ran one of those handy programs that removed all the attachments and saved them as files.

      When it was all done we deleted his account, created a new one, copied all the rescued emails back to it and passed over the attachments in a folder.

      It took a trainee technician three solid days. The VIP never did understand what took so long, surely just extracting all the attachments into a handy folder only took a couple of minutes?

      1. Anonymous Coward
        Anonymous Coward

        Re: Thing of the past, thank god! -users just do not get why you need to limit their mail to 2Gb

        A few years back, a TM asked me if I could help him clear some space in his email. He'd worked out that it was in the Calendar, people attaching documents to meeting invites, and he didn't fancy going through and zapping them by hand.

        At the time, I was working on some shoddy VBA automation to get stuff off emails to put in spreadsheets, so I said "sure", threw something together in an hour and sent it over. He mentioned it to other managers - managers being the sorts to have lots of meetings - and it spread around. Got put on the Yammer thing.

        Then one day the CFO just turned up at my desk as it didn't work for him. That was a surprise.

        1. defiler

          Re: Thing of the past, thank god! -users just do not get why you need to limit their mail to 2Gb

          ...or in a previous job a financial adviser who filled his mailbox with porn. I emailed him several times to ask him to trim it down and he ignored me.

          I got the (female) office manager to come with me to his desk, as she was above me in the org chart. He protested that he "needed" everything in his mailbox.

          <sort by size>

          Me: How about this? <opens PPT full of porn>

          Him: Ah - not that one, but I need the rest.

          Me: How about this one then? <opens a different PPT full of porn>

          Him: No, not that either.

          Me: What about this? <opens a pornographic movie>

          Him (by this time going very red): I'll have a little clear-out.

          Me: I think that would be a good idea.

          Office manager wasn't impressed with him.

          Besides which, I don't understand why people have this propensity to hoard porn - it's not like the internet is running out any time soon!

          1. Anonymous Coward
            Anonymous Coward

            Re: Thing of the past, thank god! -users just do not get why you need to limit their mail to 2Gb

            "Besides which, I don't understand why people have this propensity to hoard porn - it's not like the internet is running out any time soon!"

            I don't understand why people look at it and/or store it on their work computers.

            (unless they are pornographers)

            1. This post has been deleted by its author

            2. quxinot

              Re: Thing of the past, thank god! -users just do not get why you need to limit their mail to 2Gb

              >>I don't understand why people look at it and/or store it on their work computers.

              (unless they are pornographers)<<

              Maybe they don't have a pornograph at home?

          2. J.G.Harston Silver badge

            Re: Thing of the past, thank god! -users just do not get why you need to limit their mail to 2Gb

            And why on earth store it *in* *your* *mailbox*???? Extract it and store the file as a file in the file space for files.

      2. J. Cook Silver badge

        Re: Thing of the past, thank god! -users just do not get why you need to limit their mail to 2Gb

        My direct manager two bosses back (aka 'Turkey', whom I've ranted about before) burned his ~2 GB quota within three months of starting, because he didn't delete anything at all, and wanted to be on *every* group and list the rest of the team had, including some extremely chatty groups. (I've been here at [RedactedCo] for ~12 years and I've only gotten quota warnings once.)

        Fortunately, most of our users are reasonably decent about archiving old emails, and the few that actually do need open quotas are high enough up in the food chain that they get it. (especially the one that signs the paychecks, who is also the biggest space offender. :) )

    2. Anonymous Coward
      Anonymous Coward

      Re: Thing of the past, thank god!

      don't you just love users who know best. Back in Exchange 2003 days we didn't have any archive tools so some users used PST. We then got a archive tool but said users (senior management of course) still wanted to use PST, (despite all the >2Gb issues PST's had back in the day) Fast forward to exchange 2010 Enterprise with built in archive, (we migrated all the data from the old archive tool in) We rolled out Office2010 at the same time and I said LETS NOT enable PST archiving in Outlook so users will have to use the built in the exchange archive. Nope can't as 3 seniors want to use PST's Result 8 years later we still got users who think they're using Exchange archive when in fact they've been using PSTs and storing them locally, sometimes as the archive option (using PSTs) is the default you see in Outlook 2010.

  10. Lee D Silver badge

    Which is why you ALWAYS exclude any MS SQL, VHD(X)-holding-area, or Exchange database folder from any antivirus scan.

    Such "bad-string-search-programs" (as I like to call them) are too dumb to cope with such files half the time, and certainly you don't want the AV holding up or quarantining access to your main hypervisor's VHDX files that are constantly being read from / written to - for a start, just making some AV look inside a VHDX file which can be terabytes large is an incredibly stupid idea anyway, let alone when you're on a machine that has dozens of them. I don't debate that it's a good idea to have the core OS on a server (even a hypervisor) protected by an AV program, though.

    Modern software (usually) knows how to deal with such formats (famous last words), but I always put them on the exclusion lists anyway - you just know the one time that it doesn't, it'll take down your system, and any program that can sneak past the AV and plant its stuff in the MS SQL db folder is already a full system compromise anyway, and must have come via another entry point through which they would have been scanned anyway (as things tend not to download to that folder by default!). For me, there's a Sophos server config and a Sophos client config, and the server one excludes any of usual / default folders I store that stuff in, and certainly DOES NOT ever delete files - and the individual emails are handled via Puremessage anyway before they ever hit the Exchange database, and then the database is only scanned by a program that understands its format.

    It worries me that people manage systems by just slapping on some AV onto a server without for a second thinking of the potential consequences.

    1. LeahroyNake

      Puremessage Upgrade

      If you like Sophos Puremessage you may like their Mail scanning VM / SEA. I think it's included with every license that includes Puremessage so no additional cost. Just make sure you give the VM 3GB Ram as it can stall if you use the recommend minimum of 2GB. Well worth an hour to test and it restarts a lot quicker than exchange / less blips in email deliverability from external sources.

      The less installed extra stuff on an Exchange server the better IMHO.

  11. steviebuk Silver badge

    I like...

    ...the get out. Technically not his fault and surely why should you get into shit for that because of the virus. After all, he wouldn't of deleted the mail for that day if they hadn't been infected.

    1. Aladdin Sane
      Headmaster

      Re: I like...

      Wouldn't have

  12. Version 1.0 Silver badge

    Nice story

    It explains exactly the main reason that I do not, and never will, use Exchange.

    "Fire and brimstone coming down from the skies! Rivers and seas boiling! Forty years of darkness, earthquakes, volcanos! The dead rising from the grave! Human sacrifice, dogs and cats living together... mass hysteria!" -- The Ghostbusters explain why not to buy or use any Microsoft products (an old ASR sig).

    1. Trixr

      Re: Nice story

      Funnily enough, the worst c*ck-up I've ever encountered with email involved a Linux system. Shiny new Red Hat box, which had all the email from the university academics stored on it, recently transferred from the ancient VMS system. It was IMAP, but I don't recall whether or not it was Dovecot. I didn't administer it.

      One day, the email storage got hosed because of some issue with the SAN (twenty years ago; can't remember circumstances now). Oh dear, sorry academics, we'll have to restore everything from backup. In the meantime, they had dialtone mailboxes, so they were receiving new messages.

      Go to restore the backup... there is no backup. There was some arrangement where the mail storage was supposed to be backed up via another system mounting the mail storage volume, and this had never been put in place. The RHEL backup was only backing up local storage, not SAN-attached. Oh dear oh dear oh dear.

      How was it recovered? Recovering the VMS system, re-migrating the mailboxes to Dovecot(?) on RHEL, and then replaying the MTA logs to catch up the interval between the VMS migration and the loss of storage. Amazingly, it only took a week, although the boss was positively volcanic in demeanour that week.

      Conversely, the worst issue I've had in 20 years' of Exchange support was the smallish regional mail server that was happily receiving messages from the MTA and other Exchange servers, queuing them nicely in the SMTP message queues... and failing to deliver them to mailboxes. Since intra-Exchange and MTA delivery queues are different, and the server had plenty of storage, was not over subscribed, each of the email databases were happy and the messages were destined for different DBs, blah blah de blah, trying to find out what was going on was difficult.

      In the end, after inspecting logs, checking all services up, stopping/restarting services, unmounting/remounting databases, restarting SMTP, moving mail queues to different partitions... 6 hours later, I gave up and rebooted the box. Once it's back up, BAM, everything starts getting delivered as if nothing happend. All the mail was delivered within 10 mins. THANKS, MICROSOFT!

  13. muddysteve

    It was a Friday

    First alarm bell!

  14. Anonymous Coward
    Anonymous Coward

    Romeo left a loose end though

    Someone put the tape in the tape drive. Romeo will never be safe until that loose end is tied up.

    1. Waseem Alkurdi
      Mushroom

      Re: Romeo left a loose end though

      An accident involving a tape safe and X gallons of kerosene?

      1. Is It Me

        Re: Romeo left a loose end though

        No, an accident involving the person who put the tape in the drive.

        Otherwise they will always know...

  15. Anonymous Coward
    Anonymous Coward

    Small businesses did things differently back then.....

    A small retail shop of my acquaintance had an ethernet LAN with six or eight Windows 98 PCs on the LAN. Someone had set all the C: drives to be shared with everyone else. There was a cable modem on the LAN, and no firewalls anywhere. None of the machines was running any virus detection/protection.

    *

    One day one of these machines "caught" a virus, and thanks to the disk sharing, immediately they all had the virus.

    *

    Took a while to fix this mess!!! Retail sales were a bit slow for the following few days!!

  16. sisk

    Deleted Emails

    Many years ago I was responsible for deleting student AD accounts at the end of the school year. I did this by going to the OU in the third-party AD front-end we used, hitting select all, and hitting delete. I had to do this for each grade in 18 schools ranging from primary schools to high schools, somewhere around 100 OUs and 7000ish accounts in total. All was going well until about halfway through the task. I watched it clear the OU I was on and then, as it finished, realized that I was in a teacher OU.

    "Whoops" doesn't begin to cover it.

    While hiding my mistakes really isn't in my nature, fixing them before I tell the boss I screwed up is. I created new accounts for all the users I'd just deleted, restored the contents of their network drives, and a did a bit of hacking to recover the deleted Exchange mailboxes from the old accounts and connect them to the new ones (after being told by a Microsoft support tech that such a feat wouldn't be possible, I might add). Once all that was done I fired an email off to the boss explaining what had happened. In the end the only inconvenience to the deleted users was that they had to set new passwords for themselves when they came back a month later and I was spared any consequences by the fact that I'd already fixed it before anyone noticed anything.

    1. Trixr

      Re: Deleted Emails

      All I can say is thank god for the AD Recycle Bin these days, and "prevent accidental deletion" of OUs.

      Still, creating 7000 new accounts seems a bit knee-jerk - recover the accounts from brick-level backup if you have one or an authoritative restore from a DC backup. That shouldn't have been too difficult if it was done by deleting entire OUs. Recovering the accounts will also restore connectivity to the "orphaned" Exchange mailboxes because the mailbox attributes will also be restored.

      Also, for young players, TELL THE BOSS. Yes, develop some kind of basic recovery plan before you tell the boss, but TELL THE BOSS FIRST. A decent boss will fend off any upper management that starts whinging about missing accounts. A boss who is first informed of an issue that you're in the middle of p*ssing around with by the CEO, or (don't ask) a member of the public, is going to be spending much more time imitating a very hot blowdryer in your face rather than letting you get on with trying to keep your job.

      And no, if you're not in the US, you shouldn't be fired because of one c@ckup, if you recover the situation. However, doing the mushroom routine on the boss will not be great if it's bad enough and a PIR decides someone's head needs to be on the chopping block.

      As someone who has been the boss of an infrastructure team, I've had the good fortune not to encounter an issue that we couldn't recover from. But team members trying to fix serious issues themselves without putting their hands up (self-caused or not, although the former is worse) always make it more difficult for managers and team members to help them get it sorted. Not to mention making the manager look like a numpty in front of the real PHBs if they hear about it first - no-one likes being kept in the dark and made to look like a moron to their boss (I don't care what level you're working at).

      It also means that upper management lose confidence in the team as a whole if they perceive the manager as being clueless. Again, one incident like that shouldn't be too bad in the greater scheme, but if it keeps happening, in this day and age, it's the outsourcers next, not a new manager (and if you have a manager who genuinely wants to help you get on with your job, you want to keep them happy - mutual back-scratching is a good thing in this instance).

      1. sisk

        Re: Deleted Emails

        Still, creating 7000 new accounts seems a bit knee-jerk

        You misunderstand. I was supposed to delete 7000 accounts 20-30 at a time. The mistake was on one OU. Since it was a teacher OU instead of a student OU it was a little bigger, but still no more than 40. I also wasn't deleting the OUs themselves, just the accounts they contained.

        As for restoring, it was probably the first thing I tried (It's been several years, some details are lost to my memory), but at the time our backup system was both a major PITA and a bit unreliable. It only took around a couple hours worth of work to fix the whole mess once you discount the fruitless call to Microsoft tech support. Had it happened during the school year it would have been a much bigger problem, but as I mentioned all the teachers who were affected were out on summer break.

    2. defiler

      Re: Deleted Emails

      In the end the only inconvenience to the deleted users was that they had to set new passwords for themselves when they came back a month later

      Bwahahahaha!!

      Okay, first of all, well done for getting yourself back up and running - let's not consider taking that away from you. But a month? Gotta love academia... I've seen myself staring down the barrel of a figurative gun if the email server wasn't back up by the morning.

      Got any good jobs going?

    3. nick turner

      Re: Deleted Emails

      "In the end the only inconvenience to the deleted users was that they had to set new passwords for themselves when they came back a month later"

      I'm assuming this would have been prior to exchange 2000 as otherwise you would have had to create a custom x500 address for each of those new accounts to get round the issue of the new LegacyExchangeDN issue.

      A restore would have always been the more sensible option in every way possible!

  17. Anonymous Coward
    Anonymous Coward

    same thing only different

    I went looking for an email in someone else's account by adding their account to KMail. Of course, I set the account type to POP3 instead of IMAP so it all got deleted, after which I removed that account and then deleted the only copy off my machine. I got some of it back by scraping /dev/hda for email headers. Then I swore off reading other people's mail from behind the scenes, forever. If they were very unhappy, I don't remember them showing it. Of course later I could always ssh and grep -rni Maildir/ but that was 2003... and only one of my total pooch-screws.

    Another was spinning up a 40GB drive with the cover off-- I didn't know the spindle wasn't quite rigidly fixed to the drive body and it really depended on the top cover pressing down complete with torx screw to hold the axis perpendicular. The heads did plenty of damage... I copied out as much as possible, saved lots of C:\D&S, reinstalled XP on a new drive, and gave them back what I can only hope was all the things.

    You: Why the actual fuck would you do that to spinny rust, let alone with someone else's?

    Me: I... I don't know.

    1. Killfalcon Silver badge

      Re: same thing only different

      I won't lie, if I found a working hard-drive with the cover off, I'd give it a go to see what it looked like in motion.

      Well, I might skip it now you've told me it goes badly, but...

      1. Anonymous Coward
        Anonymous Coward

        Re: same thing only different

        But it was literally just that one time that it went badly. Almost always, you can get away with it-- the drive just gets some extra dust in it, which gets flung off the platters and caught in a little filter for the wind it stirs around in there. Maybe its useful life is slashed, it gets some unrecoverable read errors sooner, that's all, because 'usually' the whole spindle doesn't flop around. (yes, I did that more than a few times, yes it's fun to watch, at least one DIY thing was about replacing the lid with plexi so you could include it in case mods and it would [probably] not die)

  18. Potemkine! Silver badge

    Exchange is Hell

    Getting rid of it is good for sysadmins' life expectancy.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like