Scanning an Exchange server for a virus that spreads via email? What could go wrong? Just like clockwork, another weekend is over and Monday is here again. To lighten the load, El Reg is offering you the latest instalment of Who, Me?, our weekly sysadmin confessional column. This time we meet "Romeo", who was working at a large music company in London at the time in question. It was his first job for a big …
Oh, I remember that horror and the VIP who filled his mailbox (mostly with duplicate attachments) and wondered why he wasn't getting any email. Most of it had been deleted off the server.
We set up a local mail server with him and a dummy account and then copied all of his emails, in blocks, over to the dummy account. After each block we ran one of those handy programs that removed all the attachments and saved them as files.
When it was all done we deleted his account, created a new one, copied all the rescued emails back to it and passed over the attachments in a folder.
It took a trainee technician three solid days. The VIP never did understand what took so long, surely just extracting all the attachments into a handy folder only took a couple of minutes?
A few years back, a TM asked me if I could help him clear some space in his email. He'd worked out that it was in the Calendar, people attaching documents to meeting invites, and he didn't fancy going through and zapping them by hand.
At the time, I was working on some shoddy VBA automation to get stuff off emails to put in spreadsheets, so I said "sure", threw something together in an hour and sent it over. He mentioned it to other managers - managers being the sorts to have lots of meetings - and it spread around. Got put on the Yammer thing.
Then one day the CFO just turned up at my desk as it didn't work for him. That was a surprise.
...or in a previous job a financial adviser who filled his mailbox with porn. I emailed him several times to ask him to trim it down and he ignored me.
I got the (female) office manager to come with me to his desk, as she was above me in the org chart. He protested that he "needed" everything in his mailbox.
<sort by size>
Me: How about this? <opens PPT full of porn>
Him: Ah - not that one, but I need the rest.
Me: How about this one then? <opens a different PPT full of porn>
Him: No, not that either.
Me: What about this? <opens a pornographic movie>
Him (by this time going very red): I'll have a little clear-out.
Me: I think that would be a good idea.
Office manager wasn't impressed with him.
Besides which, I don't understand why people have this propensity to hoard porn - it's not like the internet is running out any time soon!
"Besides which, I don't understand why people have this propensity to hoard porn - it's not like the internet is running out any time soon!"
I don't understand why people look at it and/or store it on their work computers.
(unless they are pornographers)
This post has been deleted by its author
My direct manager two bosses back (aka 'Turkey', whom I've ranted about before) burned his ~2 GB quota within three months of starting, because he didn't delete anything at all, and wanted to be on *every* group and list the rest of the team had, including some extremely chatty groups. (I've been here at [RedactedCo] for ~12 years and I've only gotten quota warnings once.)
Fortunately, most of our users are reasonably decent about archiving old emails, and the few that actually do need open quotas are high enough up in the food chain that they get it. (especially the one that signs the paychecks, who is also the biggest space offender. :) )
don't you just love users who know best. Back in Exchange 2003 days we didn't have any archive tools so some users used PST. We then got a archive tool but said users (senior management of course) still wanted to use PST, (despite all the >2Gb issues PST's had back in the day) Fast forward to exchange 2010 Enterprise with built in archive, (we migrated all the data from the old archive tool in) We rolled out Office2010 at the same time and I said LETS NOT enable PST archiving in Outlook so users will have to use the built in the exchange archive. Nope can't as 3 seniors want to use PST's Result 8 years later we still got users who think they're using Exchange archive when in fact they've been using PSTs and storing them locally, sometimes as the archive option (using PSTs) is the default you see in Outlook 2010.
Which is why you ALWAYS exclude any MS SQL, VHD(X)-holding-area, or Exchange database folder from any antivirus scan.
Such "bad-string-search-programs" (as I like to call them) are too dumb to cope with such files half the time, and certainly you don't want the AV holding up or quarantining access to your main hypervisor's VHDX files that are constantly being read from / written to - for a start, just making some AV look inside a VHDX file which can be terabytes large is an incredibly stupid idea anyway, let alone when you're on a machine that has dozens of them. I don't debate that it's a good idea to have the core OS on a server (even a hypervisor) protected by an AV program, though.
Modern software (usually) knows how to deal with such formats (famous last words), but I always put them on the exclusion lists anyway - you just know the one time that it doesn't, it'll take down your system, and any program that can sneak past the AV and plant its stuff in the MS SQL db folder is already a full system compromise anyway, and must have come via another entry point through which they would have been scanned anyway (as things tend not to download to that folder by default!). For me, there's a Sophos server config and a Sophos client config, and the server one excludes any of usual / default folders I store that stuff in, and certainly DOES NOT ever delete files - and the individual emails are handled via Puremessage anyway before they ever hit the Exchange database, and then the database is only scanned by a program that understands its format.
It worries me that people manage systems by just slapping on some AV onto a server without for a second thinking of the potential consequences.
If you like Sophos Puremessage you may like their Mail scanning VM / SEA. I think it's included with every license that includes Puremessage so no additional cost. Just make sure you give the VM 3GB Ram as it can stall if you use the recommend minimum of 2GB. Well worth an hour to test and it restarts a lot quicker than exchange / less blips in email deliverability from external sources.
The less installed extra stuff on an Exchange server the better IMHO.
It explains exactly the main reason that I do not, and never will, use Exchange.
"Fire and brimstone coming down from the skies! Rivers and seas boiling! Forty years of darkness, earthquakes, volcanos! The dead rising from the grave! Human sacrifice, dogs and cats living together... mass hysteria!" -- The Ghostbusters explain why not to buy or use any Microsoft products (an old ASR sig).
Funnily enough, the worst c*ck-up I've ever encountered with email involved a Linux system. Shiny new Red Hat box, which had all the email from the university academics stored on it, recently transferred from the ancient VMS system. It was IMAP, but I don't recall whether or not it was Dovecot. I didn't administer it.
One day, the email storage got hosed because of some issue with the SAN (twenty years ago; can't remember circumstances now). Oh dear, sorry academics, we'll have to restore everything from backup. In the meantime, they had dialtone mailboxes, so they were receiving new messages.
Go to restore the backup... there is no backup. There was some arrangement where the mail storage was supposed to be backed up via another system mounting the mail storage volume, and this had never been put in place. The RHEL backup was only backing up local storage, not SAN-attached. Oh dear oh dear oh dear.
How was it recovered? Recovering the VMS system, re-migrating the mailboxes to Dovecot(?) on RHEL, and then replaying the MTA logs to catch up the interval between the VMS migration and the loss of storage. Amazingly, it only took a week, although the boss was positively volcanic in demeanour that week.
Conversely, the worst issue I've had in 20 years' of Exchange support was the smallish regional mail server that was happily receiving messages from the MTA and other Exchange servers, queuing them nicely in the SMTP message queues... and failing to deliver them to mailboxes. Since intra-Exchange and MTA delivery queues are different, and the server had plenty of storage, was not over subscribed, each of the email databases were happy and the messages were destined for different DBs, blah blah de blah, trying to find out what was going on was difficult.
In the end, after inspecting logs, checking all services up, stopping/restarting services, unmounting/remounting databases, restarting SMTP, moving mail queues to different partitions... 6 hours later, I gave up and rebooted the box. Once it's back up, BAM, everything starts getting delivered as if nothing happend. All the mail was delivered within 10 mins. THANKS, MICROSOFT!
A small retail shop of my acquaintance had an ethernet LAN with six or eight Windows 98 PCs on the LAN. Someone had set all the C: drives to be shared with everyone else. There was a cable modem on the LAN, and no firewalls anywhere. None of the machines was running any virus detection/protection.
*
One day one of these machines "caught" a virus, and thanks to the disk sharing, immediately they all had the virus.
*
Took a while to fix this mess!!! Retail sales were a bit slow for the following few days!!
Many years ago I was responsible for deleting student AD accounts at the end of the school year. I did this by going to the OU in the third-party AD front-end we used, hitting select all, and hitting delete. I had to do this for each grade in 18 schools ranging from primary schools to high schools, somewhere around 100 OUs and 7000ish accounts in total. All was going well until about halfway through the task. I watched it clear the OU I was on and then, as it finished, realized that I was in a teacher OU.
"Whoops" doesn't begin to cover it.
While hiding my mistakes really isn't in my nature, fixing them before I tell the boss I screwed up is. I created new accounts for all the users I'd just deleted, restored the contents of their network drives, and a did a bit of hacking to recover the deleted Exchange mailboxes from the old accounts and connect them to the new ones (after being told by a Microsoft support tech that such a feat wouldn't be possible, I might add). Once all that was done I fired an email off to the boss explaining what had happened. In the end the only inconvenience to the deleted users was that they had to set new passwords for themselves when they came back a month later and I was spared any consequences by the fact that I'd already fixed it before anyone noticed anything.
All I can say is thank god for the AD Recycle Bin these days, and "prevent accidental deletion" of OUs.
Still, creating 7000 new accounts seems a bit knee-jerk - recover the accounts from brick-level backup if you have one or an authoritative restore from a DC backup. That shouldn't have been too difficult if it was done by deleting entire OUs. Recovering the accounts will also restore connectivity to the "orphaned" Exchange mailboxes because the mailbox attributes will also be restored.
Also, for young players, TELL THE BOSS. Yes, develop some kind of basic recovery plan before you tell the boss, but TELL THE BOSS FIRST. A decent boss will fend off any upper management that starts whinging about missing accounts. A boss who is first informed of an issue that you're in the middle of p*ssing around with by the CEO, or (don't ask) a member of the public, is going to be spending much more time imitating a very hot blowdryer in your face rather than letting you get on with trying to keep your job.
And no, if you're not in the US, you shouldn't be fired because of one c@ckup, if you recover the situation. However, doing the mushroom routine on the boss will not be great if it's bad enough and a PIR decides someone's head needs to be on the chopping block.
As someone who has been the boss of an infrastructure team, I've had the good fortune not to encounter an issue that we couldn't recover from. But team members trying to fix serious issues themselves without putting their hands up (self-caused or not, although the former is worse) always make it more difficult for managers and team members to help them get it sorted. Not to mention making the manager look like a numpty in front of the real PHBs if they hear about it first - no-one likes being kept in the dark and made to look like a moron to their boss (I don't care what level you're working at).
It also means that upper management lose confidence in the team as a whole if they perceive the manager as being clueless. Again, one incident like that shouldn't be too bad in the greater scheme, but if it keeps happening, in this day and age, it's the outsourcers next, not a new manager (and if you have a manager who genuinely wants to help you get on with your job, you want to keep them happy - mutual back-scratching is a good thing in this instance).
Still, creating 7000 new accounts seems a bit knee-jerk
You misunderstand. I was supposed to delete 7000 accounts 20-30 at a time. The mistake was on one OU. Since it was a teacher OU instead of a student OU it was a little bigger, but still no more than 40. I also wasn't deleting the OUs themselves, just the accounts they contained.
As for restoring, it was probably the first thing I tried (It's been several years, some details are lost to my memory), but at the time our backup system was both a major PITA and a bit unreliable. It only took around a couple hours worth of work to fix the whole mess once you discount the fruitless call to Microsoft tech support. Had it happened during the school year it would have been a much bigger problem, but as I mentioned all the teachers who were affected were out on summer break.
In the end the only inconvenience to the deleted users was that they had to set new passwords for themselves when they came back a month later
Bwahahahaha!!
Okay, first of all, well done for getting yourself back up and running - let's not consider taking that away from you. But a month? Gotta love academia... I've seen myself staring down the barrel of a figurative gun if the email server wasn't back up by the morning.
Got any good jobs going?
"In the end the only inconvenience to the deleted users was that they had to set new passwords for themselves when they came back a month later"
I'm assuming this would have been prior to exchange 2000 as otherwise you would have had to create a custom x500 address for each of those new accounts to get round the issue of the new LegacyExchangeDN issue.
A restore would have always been the more sensible option in every way possible!
I went looking for an email in someone else's account by adding their account to KMail. Of course, I set the account type to POP3 instead of IMAP so it all got deleted, after which I removed that account and then deleted the only copy off my machine. I got some of it back by scraping /dev/hda for email headers. Then I swore off reading other people's mail from behind the scenes, forever. If they were very unhappy, I don't remember them showing it. Of course later I could always ssh and grep -rni Maildir/ but that was 2003... and only one of my total pooch-screws.
Another was spinning up a 40GB drive with the cover off-- I didn't know the spindle wasn't quite rigidly fixed to the drive body and it really depended on the top cover pressing down complete with torx screw to hold the axis perpendicular. The heads did plenty of damage... I copied out as much as possible, saved lots of C:\D&S, reinstalled XP on a new drive, and gave them back what I can only hope was all the things.
You: Why the actual fuck would you do that to spinny rust, let alone with someone else's?
Me: I... I don't know.
But it was literally just that one time that it went badly. Almost always, you can get away with it-- the drive just gets some extra dust in it, which gets flung off the platters and caught in a little filter for the wind it stirs around in there. Maybe its useful life is slashed, it gets some unrecoverable read errors sooner, that's all, because 'usually' the whole spindle doesn't flop around. (yes, I did that more than a few times, yes it's fun to watch, at least one DIY thing was about replacing the lid with plexi so you could include it in case mods and it would [probably] not die)
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
