Scanning an Exchange server for a virus that spreads via email? What could go wrong? Just like clockwork, another weekend is over and Monday is here again. To lighten the load, El Reg is offering you the latest instalment of Who, Me?, our weekly sysadmin confessional column. This time we meet "Romeo", who was working at a large music company in London at the time in question. It was his first job for a big …
That thinks the actions could be close to doing the right thing?
Although a company culturally can create a great dependence on email as a business tool for variously archiving, signoff approvals, support messages, and other unwise use-cases.
At least the server was clean, and probably caught other things too.
My worry would be how the messages got to this point without being cleaned already...
No, the server was not clean. It didn't clean the mail database, it just deleted it, so a previous copy had to be restored - it still contained all the bad emails the previous one contained but the last day.
And why a company should not rely on email? Should they install pneumatic mail, and have people with carts going around desks bring paper documents to read and sign? Companies always relied on internal document shifting...
@LDS Fair call, but the article does not go into much detail on what was done with the restore. I assume some tool more suitable for cleaning mailboxes was deployed. Server was therefore clean albeit at the expense of deleting everything. The process did have an excess of collateral damage, but did remove the infection.
Now about those carts, they used to be filled with snacks and coffee...
For what I read, it had big troubles restoring the mail database because in Exchange just replacing an older file is not enough - as database systems are usually picky when data files, logs and other things don't match. So, sure, he deleted the ILOVEYOU mails of that day - but whatever else was lurking there from previous days was still there.
Cleaning such stuff exactly needs tools which are able to read the database correctly and clean infected messages one by one - but you usually need to have the mail database open and accessible to run them, because accessing the on-disk structure of such files - often undocumented - it's a very risky task.
While in certain circumstance you may not have other options that wiping everything, running an AV against database files is usually a very bad idea - especially if the default actions is "delete".
especially if the default actions is "delete"
He must have been really new to virus scanning. People with experience know that there are such things as false positives which are more often than not important documents and critical Windows DLLs... as such the only sensible action is to quarantine suspect files to ensure you aren't about to nuke something important, and if it needs to go (of it's a real virus and not a "this code looks odd" heuristic), you know what to replace from the installation media before the machine gets itself into a bluescreen-at-boot state.
Never ever give an antivirus program the ability to automatically delete stuff...
But if you have a corpse in the trunk, after the sandblasting and repainting, you will still have a corpse in the trunk... ..... LDS
The I Love You Virus a Corpse in a Trunk whenever Phantom Bodied? I Don't Think So, Mes Amis/Mon Brave.
Are you Bitten and Smitten and Rooting for More LOVE Bugs to Display the Bounty of ITs Wares with Immaculate Temptations to Sate and Supply ie Fully Realise in COSMIC Great Order for SMARTR TerraPhorming Nations?
Try Contemplation of IT as Advanced IntelAIgent Driver/Virtual Mentor and Practical Monitor.
One of those Funky Clunky NEUKlearer HyperRadioProACTive IT AIdPrograms Perfectly Suited for Princes and Princesses with Visions in Peril? ...... Saudi Vision 2030
And there be Kings and Queens, Princes and Princesses, Nymphs and Satyrs Everywhere. And that Convenience makes More LOVE Bugs AIdPrograms Astronomically Wealthy and Certainly Worthy.
It's not like this one organization would even have been in the minority here, ILOVEYOU was a major wake up call for many orgs to put more work into their email scanning.
Many orgs, if they had incoming scanners at all were just using signature based checks so of no use against a rapidly spreading worm based on social engineering.
At the time of that particular virus spreading I was working for a firm doing some tech related planning. We were using outlook and hit quite hard when one just person was sent it. Whilst IT were cleaning everything the rest of the place descended on the local public houses. We had an afternoon spent doing not much except trying different beers etc. Then returned to the office to collect our things at 5:30 whereupon some people went back to the pub. IT support told us that they had cleaned all our mailboxes/computers and beefed up the mail filter.
Everything was fine after that except a few months later when we all started to get these emails again. IT were a bit annoyed that it had made it through the mail filter The culprit was the Intern mailbox which was only used when we had an intern. There wasn't an intern at the time so the mailbox never got cleaned. One of the staff left immediately for the pub when the first email appeared. He was called back before he could order anything to drink.
I remember well when ILOVEYOU broke out on our university campus. It was that morning when I found an e-mail in my inbox with this love confession from a girl in administration. As sweet as she was, I had some mixed feelings about this - can't remember her name since we cruelly only called her The Nose. For obvious reason. Needless to say, our love relationship wasn't meant to last. Not longer than a few seconds anyway.
I was safe with my Linux environment. And the next thing was to start up the Windows sandbox - no virtual machine at that time - and investigate what this charming virus actually did.
Yep, that's rekindled a few grey cells. We thought we were on top of it, repeated warnings sent out, 24pt, bold, underlined, bright red, dire threats and all. Then, one person in PR decided to completely ignore all that... When asked, they said, "Oh, I never read messages from IT, you're always just sending out warnings."
"Oh, I never read messages from IT, you're always just sending out warnings."
The boy who cried Wolf springs to mind.
Can't comment on your individual situation, but warnings are more effective if you pick your cases with some care to avoid overloading users with esoterica that'll only baffle them.
And while there was stuff that needed to be fixed, a lot of it was scaremongering, especially related to embeded systems. For example, the idea that a washing machine might, on 01/01/00, think it was 1900, and, that as it hadn't been invented yet, it ought to shake itself to bits and then spontaneously combust.
That's what I always point out to people. I wasn't involved in any fixes, I'd just finished college. But the amount of people that say "That Y2K thing was bollocks wasn't it. Nothing happened". Especially so called comedians. Yeah nothing happened because people worked fucking hard to fix the issues before they could happen. Tits.
I had a journalist approach me in 1998 looking for scary Y2K story- I told her that nothing was going to happen because we'd been on top of it for a number of years now and everything should be in place by then. Needless to say, she completely ignored me and found a nutjob that gave the doomsday scenario that she was looking for.
"Needless to say, she completely ignored me and found a nutjob that gave the doomsday scenario that she was looking for."
It would have been funny if all those media types touting the scare stories had their own orgs IT fall over, but sadly their own IT people were also on top of things. Shame the journos didn't interview their own IT bods and got the real story.
@steviebuck
While I agree that a lot of conscientious people worked a lot of hours in the run-up to Y2K, IIRC a patch for Windows believing 2000 would be a leap year came out in something like November 1999. This despite earlier complaints from fin-tech people that computations of future value or the like were odd. The thing is,, sometimes you don't just need to know what day today is, but what day 60 or 180 days from now will be.
2000 was a leap year. 1900 and 2100 are not.
Most computers at the time assumed any year divisible by 4 is a leap year, when in fact years divisible by 100, but not divisible by 400 are not leap years.
Excel still thinks 29th February 1900 is a valid date, and most other spreadshhets copy this bug for compatibility.
Y2K was scare mongering.
Had to be to get all the CEO's and bean counters on board to allow all the time and money to be spent to check all the software for problems. In the end the CEO's and bean counters all knew they would get blamed by the stock holders, customers and little kids on the street if their company had a Y2K problem.
Then, one person in PR decided to completely ignore all that... When asked, they said, "Oh, I never read messages from IT, you're always just sending out warnings."
Those types should be strung up by the front door as example and with an email or group meeting explanation. The second one who does it should be drawn and quartered.
... at how noone sued MS for damages at the time.
The means by which this email evaded detection in a simple and sensible email scanner was MS's deliberate breaking of MIME standards dating back to 1992. And the RFC even contains an informational section under the heading of security implications explaining exactly why what MS subsequently did would leave their users wide open to attack.
"what MS subsequently did would leave their users wide open to attack"
You mean like how the created user profile for users on a home installation of XP had admin rights by default, and how the restricted user profile was so restricted it was near useless for many (you couldn't even change the time FFS). There was, I believe, a tool to tweak what rights users had, in the enterprise version...
It was pretty much a wide open door back then, just marginally less open than Win32 machines.
"at how noone sued MS for damages at the time."
Read the fine print. MS' code isn't even guaranteed to work as advertised when used as intended. It's use at your own risk, at least according to MS's own EULA. You HAVE read the EULA, and fully understand it, right? And your corporate lawyers have vetted it as OK for use by your business, right?
Read the licence blurb?
Eight years ago a company demonstrated how many people bother reading all that rubbish: https://www.geek.com/games/gamestation-eula-collects-7500-souls-from-unsuspecting-customers-1194091/
Scanning an Exchange server for a virus that spreads via email? What could go wrong?
It deletes the EDB file and then lots of people shout at you.
I would say how did the emails get that far anyway but this was back in 2000 anyway so I suppose that answers that.
I've never deleted an EDB file but I have had an old boss add an extra drive to the Exchange server and then proceed to import the RAID config. Oops.
"I would say how did the emails get that far anyway but this was back in 2000 anyway so I suppose that answers that."
By 2000, real MTAs had been dropping malware long before it got anywhere near userspace for over a decade. Milters (introduced in 2000) and the like made it easier to admin. Toys like Exchange were never really considered an option by professionals.
I deleted an EDB file once by accident - and no backups then.
Oops.
Learnt a hard lesson then, never, ever again. It was not a fun experience, and one I don't wish on any Exchange admin.
Anon because.
Since that incident I've started to use ntbackup to back up the mail store - and funnily enough, that issue was never repeated.
Very common! A short while after the ILOVEYOU outbreak I was working for a "famous insurance company" that was (mainly) based in and named after the city I lived in (Norwich). We used remote access from our base in the city centre to the servers in the "lights out" data centre server farms four miles away out on the outskirts. It was fantastic!
Well...
...only for a weird value of "fantastic" that meant that...
...the lights in the server farm were never actualy out because remote access was bloody slow and, for "security", sessions were set to always time out after 15 minutes which, due to the slowness of the network in general and remote access in particular, meant that you barely got more than 20 mouse clicks and 10 fields filled in remotely before the whole thing shut down on you (if it had managed to stay up for the full 15 minutes, itself a rare feat). So, it was often quicker to catch the regular company-provided shuttle bus to the data centre and go and access the server non-remotely (hence the lights never being out as the data centre was full of pissed off people all doing the same non-remote thing).
One of our clients accidentally started a restore in Exchange. I think it was a block-level restore of the database rather than of a mailbox or folder - it was a while ago and I (luckily) wasn't there. When she realised her mistake she pulled the power on the email server...
My colleague had to regedit the hell out of it to force the database out of restore mode, and then restore a complete copy of the database from before the errant command. I don't think that database was quite right ever again.
Still, after I'd left that job, my ex-line-manager managed to torpedo the server nicely in a different way, but that's a story for another Monday...
We had trouble with Exchange Server 4.5. I remember running some DB repair program that took 4 or 5 hours to scan the DB half a dozen times to get it into a state that Exchange Server was happy to load.
I was pissing about with some VB for some web app and discovered VB would allow you to dump a whole exchange DB and read everyone's emails and I think I could have used that to rebuild a corrupted DB better than the MS repair program but never have the courage to try it in action. Some nice reading while repairing the DB mind.
It's Stories like this that make me relieved to get out of the email game having just shifted 10,000 users to O365 in the last couple of months....
Having lived through a corrupt EDB file recovery on our Exchange environment last year, i know how terrifying email issues can be. No matter how often you spout the line "Its a communication tool, not a file server" at people it never stops them from retaining everything ever sent since God was a boy.....
Also, micromanaging mailbox sizes on-prem is becoming harder and harder, users just do not get why you need to limit their mail to 2Gb and exceptions pop up all over the place (largest user mailbox we had was 80Gb ffs...)
It doesn't help these days that File Shares and collab SharePoint seems to be a dirty word at C-Level, where they just want to send 100Mb PowerPoint Decks and Business plans to each other...and who are IT to tell them how to do their job.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
