Reg HPC man relives 0-day rootkit GROUNDHOG DAY This is a difficult article to write, and I’ve put it off for way too long. But it’s time to bite the bullet and make an embarrassing admission to the Register audience. I’ve been hacked and hacked hard. Admitting this publicly to Reg readers is like chumming shark-infested waters with my own blood. Or like telling people that …
I know that systems are complex, and getting ever more so, but when a supposed expert is not able to identify anything about a compromised system, does this not indicate that they are getting too complex? Or maybe that proclaimed experts are not.
I've stared at the list of services and processes that are running on systems, and wondered what they all are. There appears to be nothing other than Google to try to identify the ones with unique names, or what they do, and this is just what you see, without the possibility of the kernel or standard shared libraries being subverted, or hidden loadable modules.
I'm not trying to pick on Windows here, because most Linux distributions are no better, but often in the list of running processes you see multiple things of the same name (can't give an example at the moment, don't have a Windows system running close to me!) I have no inkling of where on the filesystem the process was loaded from, or what it is associated with. I'm sure there are tools, which can dig this information out about a process on all OSs, but they are not always generally known about, much less shipped with the OS.
I doubt that anything can change at this point, I just wish we hadn't got here!
Edit. Hmm. Really should do research before posting. I should use tasklist on a standard XP system. Will have to give it a go when my wife next complains about her system being slow.
This is a problem with remote diagnosis.
Once an attacker has run a program with admin credentials, the system is theirs. They can alter any part of the OS. They can alter task manager so their processes don't appear on it, they can alter the filesystem libraries so their files don't appear, etc. etc.
If people had been there in person, they could boot up from an external disk and maybe see more of what's going on. But in nearly every case, they should be saying "yup, you done got hacked" and advising a complete reformat anyway. It's just too easy for something to be missed, which then acts as a source of re-infection.
I see your point about remote vs. keyboard access. But from looking at what they were doing on my system, it looked like they did do a boot from an external source into a proprietary shell o/s.
It didn't work, of course. Which sucked. But is reformatting the only solution here? That's going to take a lot of time - longer than it took me to reimage a spare drive. I get your point that there's every possibility for reinfection, but I would hope our current set of tools can do at least a decent job of ensuring that your existing files aren't compromised.
"....But is reformatting the only solution here...." Depends on your circumstances. With work kit the immediate answer from the business is 'we need to resume service to make money', so the pressure is to quarantine the kit and fire up a clean copy of the image on another system, something modern hypervisors and deployment software have made much easier. Analysis of the problem is secondary to getting back to making money, unless you have a reason to believe the hack is spreading and can't be contained by a quarantine. With home kit you have to ask yourself if there is much point in trying to dig around if the big boys of AV can't do it?
....but why would the rootkit / virus play sounds and give away it's presence?
All said, this post has got me thinking "Is my recovery process good enough". I do weekly images of my system but only keep enough to go back 2 or 3 weeks. Like the author I do everything humanly possible to secure my environment, but have to face the fact that it could simply be not enough.
Thanks for squashing any ego and sharing your experience!
NO! Not a stupid question at all! I spent hours wondering the same thing. I also closely checked network activity when the virus was active to see if it was busily sucking files out of my system or feverishly sending commands to a bot army. I didn't see much of any outbound network activity at all, and very little inbound traffic too.There also hasn't been any problems that are identity theft related either.
I think whoever penetrated me (whether I was targeted or it was random) probably used some code that was both sophisticated and amateurish at the same time. It was able to penetrate multiple layers of protection to get into my boxes and hide itself, but the payload didn't do much of anything other than annoy me.
On the back-up issue. I would heartily suggest that you keep more than three weeks of back-up images. One thing I didn't mention in the article (I didn't want to make it even longer and more tedious) was that one of the first things I tried was to use Windows System Restore to get back to an earlier version of the system. No luck on that, even with going back weeks in time.
So if I didn't have as many back-images as I did, I would have been even more screwed than I was. Storage is cheap these days, you can get a 3TB drive for less than $100. Get one and fill it up with back-ups, it's cheap insurance.
This post has been deleted by its author
Actually a dedicated drive for HDD images is not a bad idea, especially as you point out, when they are so cheap. Bang for buck and saved time I guess they would pay for themselves.
Here are some thoughts about weird nature of the infection
1 - Do you have a web cam attached to the network in anyway? Maybe they just wanted fun to mess with your head...
2 - Possibly using you as a lab rat, before deployment to their actual target?
....but what makes you think you ACTUALLY had a virus and not some random bug?
Did an extra process fire up when the audio kicked in? Did you notice more traffic?
Was it just a bizarre corruption? Possibly a background process such an alert, instead of playing a "ding" was playing some other random audio file.
Great question. Here's what I observed:
1) I couldn't seem to find a specific process that fired up when the audio was active. The process list (hundreds of processes and services) looked identical when the virus was 'sleeping' vs. when it was actively playing sound. I also didn't see any particular process taking more memory or CPU than normal.
2) The audio played even when there wasn't any other application open and without me doing anything that would prompt a system sound. There also weren't any system issues that would cause an alert - at least nothing I could find with deep hardware and software scans.
3) The audio was unfamiliar to me. It wasn't playing anything that I had ever heard before (nothing from my media library, for example). It sounded like it was current or cached versions of web broadcasts, but the snippets weren't long enough for me to figure out exactly what they were.
...was lost a long time ago. Not to say antivirus is useless, it'll prevent maybe 50% of rootkit infections - but there are advanced nasties out there, and most of them don't play random audio, so you're never going to know you've been compromised.
Prevention is better than cure these days - run noscript, don't open email attachments, reimage regularly etc.
Hold on, wait a minute. I didn't receive ANY benefit from their services. I thought I made this clear in the article, but maybe not.
The security company mainly just cost me time. They didn't tell me anything I didn't know. They weren't able to identify the virus, how it got into my system, or fix it. The did tell me that my current security status was top notch and exactly what they would recommend.
In their pitch to me, they said that they would certainly be able to fix my system without any intervention on my part. Was it an ironclad guarantee? Nope, but no one gives those types of guarantees on these types of services. The problem here is that they simply didn't have the experience or tools to handle my particular problem. Since the reason I purchased the subscription was to solve this problem, I feel that canceling it was fine ethically.
So this was major big-time forensics... which was ran remotely (ie, while infected system was booted and thus untrustworthy)... and they wanted a whopping $100 ? (yes, that's sarcasm....you've paid peanuts and wondering why you got monkeys)
Sound like a bunch of cowboys. Oh, and reflashing back to a few days won't get rid of a persistent rootkit. Sorry to say, but it sounds like this bunch are every bit as clueless as you are.
Oh, and your oh-so-special NAS boxes would have been for nothing had you malware that encrypted files and charged you for access.
Ok, here we go, this is more like it...lol. The charge was for $100 per month and it was with one of the biggest and most reputable security firms in the industry. So i wouldn't put them on par with monkeys - unless you know some really smart and experience monkeys.
I ended up reflashing back way more than a few days and that seems to have solved the problem, while losing valuable work at the same time, of course.
You're right that my NAS wouldn't have helped me if someone had come in and encrypted all the files on the server and NAS box. That sort of 'data kidnapping' scheme is pretty tough to protect against. How would you do it?
If it's something with admin rights then the only way to protect against it is to have an off-line backup that it can't touch.
Perhaps with the large disks available now, we need to go back to the VMS approach of versioning files, so that if I change a file, it keeps a copy of the previous one until I explicitly purge it. If that's built in to the file system then it makes it harder for someone to scramble all the files because it would only create new copies, the old ones would still be there. Provided there are several hoops to jump through to do the purge, it would be hard for the trojan to remove old copies.
As a side benefit, you could have an external audit device attached to which the filesystem would write a log entry time it changed a file so you'd be able to track back and see what changed. Being a write-only device from the perspective of the main system, and not being attached to the network in any other way, it would be helpful in forensic analysis if something bad did turn up. Obviously it can be defeated if a trojan can disturb the filesystem drivers, but even then there's a good chance that it would have to do that by overwriting the driver file on disk (which would create a record) and then forcing the system to reload it.
I guess it comes down to how paranoid you are, what performance penalty you'll accept (AV scanners do load Windows machines quite a bit) and how much you're prepared to pay for a bit more security.
"If it's something with admin rights then the only way to protect against it is to have an off-line backup that it can't touch."
It's an interesting point.
My backup regime for my MacBook has been to plug a spare USB3 HDD in when I remember and let Time Machine do it's thing.
I had been thinking it was about time to sort out a NAS both for home streaming/file sever and also as a Time Machine/rsync target so that if I forget/don't get around to plugging in for a couple of days it'll just do it's thing in the background anyway.
With the emergence of ransomware though, my offline drive is looking pretty good right now. It's only plugged in when I do a backup and I generally whip it out fairly shortly afterwards as it's the only thing that drive is used for and I need the port back (or I'm lounging and don't want the drive on my lap), which means it's relatively resistant to ransomware unless I'm incredibly unlucky and manage to pick up a bug whilst backing up (or the ransomware comes with a time delay, which is not beyond the realms of imagination). Okay, it effectively means I only have daily snapshots rather than hourly as Time Machine will do, but for home usage that's not generally of great significance.
I'm thinking if I got a NAS, rather than a simple in-box Raid 1/5 for redundancy (depending on the number of disks), might as well go for performance with Raid 0 and just have an offline USB HDD sat on top of the box that I plug in once a day to back up everything. Paranoia? Yes, a bit, but ransomware that mullers one machine is bad enough. To muller your network storage and backups as well just adds insult to injury.
Upvote for an interesting service, although when I said on/offline I meant a NAS available over the network permanently (or not, in the case of offline backup - unplugged USB drives that can't be infected until they're plugged into something).
Unfortunately as our rural cabinet is scheduled by OpenReach to receive fibre sometime after the heat death of the universe (if not later), anything "online" (as in, on the internet), is not going to do the business for images or whole disk backups.
Ahhh. gotcha! Sorry!
And yeah, your point makes sense. Also, I don't expect fibre to our rural exchange any sooner either!
As for tarsnap, I mainly use it for my remote servers, where it's impractical to pop around with a usb disk :-) although saying that, it's pretty good with dedup, deltas, and compression (much like rsync) but each backup 'appears' to exist as a full backup and can be treated as such (but as it really isn't, you don't pay for all the bandwidth and storage)
You're asking how to protect against your disk-backups being corrupted? How about a non-live backup that can't be live-edited? There's this stuff called tape y'know..... it had been quite popular over the years... There's also one-time snapshots, volume shadow copies, previous versions...
And self appointed "security auditors" There are genuine experts and there are "experts"
"Experts" do little more than run off the shelf commercial or open source utilities and then give their "recommendations" which are generic statements about how bad something could be (pasted from some security site) without even remotely understanding what they mean. They can't answer any questions about why something is dangerous. They don't know how to fix anything except by running the associated clean up utilities.
True experts can fire up a debugger, can chew up a linker symbol table, know what a disassembler is and how to use it, know a PE header from an ELF one, how dynamic code is loaded, know what RPC and CORBA are and how file compression works and go thru registry hex dumps as breakfast. They know how badly is what is being sold as security and confidence. They however charge much more than 100$ a month. They could write the malware themselves, but chose not to do so.
The drama is, for non experts it is impossible to tell what is a fake security expert from a true expert. Except perhaps because the latter category is very, very difficult to find and pretty expensive. But faking that is also easy, especially the high price point part.
True examples: a "Level 4" Unix expert assessing categorically that there is no way to tell which ports an application is using (hint: netstat). A big multinational firm presenting as evidence of a "security risk" a list of of files in a system that were world writable. The same security experts from same company claiming that Oracle listeners running on its standard port were a security risk.
There's one true fact in all what you heard from the high level expert: this is the world we live in.
Damn it! Where the hell were you when I was going through all of this??!! I could have really used your help and would have paid handsomely for it.
Just to keep things straight, I wasn't going through some random security auditors or 'experts'. I contracted with a major security firm to have them fix this box. From what I could tell from watching what they were doing on my screen, you're absolutely right, they were primarily using the same tools that I was using before I contacted them.
I didn't see any evidence of them using a debugger, or checking processes, or doing any sort of deep dive into figuring out how this virus functioned. To me, I figured it would be a matter of finding out what process is compromised and then eliminating it. But, according to them, it's not that easy these days. They said the virus code could have been buried inside other processes - which would make finding it more problematic.
On the other hand, there was SOME process that was pushing audio to my sound card, would it be impossible to trace that chain back and see what was issuing those commands?
<<But, according to them, it's not that easy these days. They said the virus code could have been buried inside other processes - which would make finding it more problematic.>>
As pointed out by other comments, rootkits can go to great lengths to hide themselves, either embedding themselves into executable images, obfuscating their code so that it is not recognized by scanners or altering low level disk read call to return the original data they overwrote. So the the worst ones (really the "best" made ones) would be difficult to detect using tools that run on the compromised machine. Any security expert would have suggested you first to image the disk booting the machine from a read only media, or better yet (BIOS can be compromised, although that is much more unlikely and difficult) plugging the boot disk on another computer
Even this could not be enough, in theory the on board drive controller can be compromised, but going to such lengths to play audio seems to use a lot of effort into something not very profitable, so I'd discard it.
Once you've imaged the disk you mount it on a known safe machine and take a MD5 hash of everything in the file system and compare it with other known good system. Not so easy as it sounds because of the many different patches and Service Packs around there. For some reason Microsoft ships close to entire code rewrites on it patches. But it can be done.
Even so, the thing can hide itself outside the reach of the file system by changing the boot loader itself and/or using the "system image" partition that is used to boot the installation process when it comes out of the factory. That is also easy to detect if you have the raw image of the disk.
<<there was SOME process that was pushing audio to my sound card, would it be impossible to trace that chain back and see what was issuing those commands?>>
Yes, it would be possible. It requires knowledge of media APIs, how the kernel handles multimedia devices and a lot of time waiting for the audio play to happen to be able to examine the system while it does that. Of course assuming that the rootkit in question has not placed any countermeasures to be debugged or that it can be circumvented.
I'm not a security expert, but I know that kernel hackers capable of doing such things exists. I know also that none of them would work over the phone helping customers.
I think what your experience summarizes best is what sorry state the security industry itself is. Each time I look at it says "scam" and "bubble" all over the place.
Huh?
Firstly, hashes are mainly used to secure the integrity of a file, whether encrypted or not (and this is the point of the article - protecting against maliciously altered files)
Secondly, even if you were talking about only protecting against, say, disk corruption, the extra entropy in sha256 can only help..
A rootkit cannot hide, while it is just data on a disk. In other words, trying to detect a rootkitted O/S using the same rootkitted OS is hopeless. You need to boot a standalone scanner (preferably off CD or DVD because they're not writeable after being checksum-protected and mastered).
Of course, this means some completely-downtime for the infected system.
Why does Windows go out of its way to make this form of security difficult or impossible?
In light of the fact that you suspect you may be getting targeted, and that you had to roll back several weeks and restore individual files, I would suggest a change now in backup strategy. Keep as much data as possible separate from the OS images and back these up separately from each other. The goal being that in the event this happens again you have all the data backed up as of now, and can roll back the OS as far as needed without hitting a problem of needing to do any (or at least many) data restores. The latter is possibly an ever greater risk to your business than the suspected rootkit - that is playing music (and who knows what else) but restoring new versions of old files massively increases the risk of missing some and then rolling forwards without them or with old versions, and likely never noticing until a long time has elapsed. That is genuine lost data.
I had a similar thought: keep your data separate from your operating system and apps, and keep the build well documented. It occurred to me that with the time the author spent finding a good image to restore, would it not maybe have been quicker (and safer?) to reinstall and rebuild the machines from scratch?
This sounds really bizarre. Why would a piece of malware literally toot its own horn? The whole purpose of a rootkit is to hide and be stealthy.
The second question is, what was it doing? The thing is, remote administration is not the way to go here. Admittedly, I'm one of those fellows who does know how to use a kernel debugger and a network sniffer, and I have a 16 port managed switch just for what's at my desk.
The first thing I would have done is, as Nigel 11 noted, run from a live CD and scan the drive. When a rootkit gets into the system, it then normally removes itself from the various process lists, or renames itself to something innocuous. The next thing I would have done is to look at the network traffic, using a different machine. OK, so I'm using switches that allow port mirroring, or else you'd have to keep a real hub handy. So I'd look at the network traffic. Today's malware usually wants to communicate on the network. So what's the traffic look like? Sending spam? Scanning? DDOS?
Something is fishy about just playing random tunes.
You make some great points above. And I was scared to death about what the rootkit might have been doing while it was distracting me with the audio. I did look at my network traffic on my switch (also 16 ports to support desktop computers and home infrastructure). I didn't see much, if any traffic coming out of the infected system, which at least sort of put my heart at ease.
I did boot the system from a recovery DVD and repair the O/S as part of my own troubleshooting. It didn't seem to work, or else the virus reinstalled itself upon boot.
I wish I had the skills to use a debugger and those kind of deep diagnostics you cite above. That would have come in very handy back then!
And you put all your 'stuff' back on the same disk and moved on. Good luck with that!
It appears that few are reading about today's rootkits, bootkits, BIOS kits, and router kits - or M$'s declaration that once infected, a computer can not be considered safe/clean EVER again - you may remove the (or a) vandal, but no one can account for what other apps may have been installed during the exposure which go undetected. [segway: And then, there's that whole problem of factory installed holes, ie Absolute Software's Computrace (anti-thief ware) installed during manufacturing at the BIOS, aux BIOS (requiring an external chip be physically cut from the mother board), or MBR level - which can not be removed and which are not secured by encrypted access.
As to the random sounds, I had this problem twice - well, I could be wrong, so maybe this IS a different problem.
There's a phenomenon (called audio rectification) where loose, twisted, curled wires will tune-in and pick up signals from a nearby broadcast radio station and with various results, play on a nearby speaker.
I had this happen once around a computer. And I had this once from a portable radio which was powered ON with volume muted. At the time of discovery, both cases were repeated to confirm and to amaze co-workers.
Yeah, you're right. I may not have a solution here at all. But so far, so good. The rest of your post scared me to the point where I almost peed my pants. But it's not anything that I haven't been wondering about.
Very interesting situation with your twisted wires picking up boradcasts. In my case, I could hear enough audio snippets to realize that they were coming from locations very far away from me - not local TV or radio stations.
You should charge admission to your demonstration of your audio problem. If it confounds and amazes co-workers, they should pay at least a little something for the experience, eh?
The laptop was not connected to the same speakers as the desktop. The audio on the laptop was playing through the laptop speakers and headphone jack. Hmm....I didn't check to see if both computers were playing the SAME audio at the same time - that was an oversight on my part, probably caused by panic..lol
This post has been deleted by its author
... from history and/or their mistakes.
People are creatures of habit and switching over to another OS seems to evade most people's logic.
While I agree that there is no such thing as a completely safe environment, some do have the advantage of anonymity since they are not as ubiquitous as Microsoft's Windows and they are built far more robust. While Apple's rendition of a certain BSD distro is better than any of Microsoft's lineage it too falls victim to an occasional nasty. Although not usually due to the inferiority of the underlying OS code which has happened, it more often is attributed to 3rd party wares like the horrendous code found in any of Adobe's products. Flash and Reader more specifically.
Windows on the other hand is just plain fubar when you look at the code and research what has been leaked from former MS employees who admit they don't go back and fix problems.
Fixing flawed code is frowned upon at MS and they usually just patch by addition which has contributed to their extremely bloated spaghetti code that resulted in the world having to upgrade their hardware to accommodate MS' screw ups.
As much as many readers do not want to hear or admit, certain Unix like OS' are far superior and extremely difficult to infect anything other than the users own files.
Sorry peeps, *nix variants are a completely different animal from which MS has been stealing ideas for years.
OpenBSD claims the highest security ratings and as for Linux Red-Hat / Novell has their stuff together fairly well. Albeit they do not support much of anything outside their own repos. So you won't find much if any Flash or other multimedia support that is notoriously to blame for a great many infections. It's when we venture outside the locked down vendor repo that we are taking all the risks into our own hands. We need to learn how to live without much of the vehicles that the bad guys are using to transport viruses and malware in.
Personally, I'd like to see all of the bad 3rd party crutches banned from the internet.
Flash? Really? It's time to die already.
People need to embrace HTML5 and CSS3 and ditch the legacy dung that is crippling all of us.
So did you or anyone actually run an offline virus scan using a boot cd/usb?
Like other say rootkits hide pretty well, so if your trying to fix it remotely/using a booted live os its a pretty impossible task, tools like unhackme work to a level and help dig out most things.
These days if i cant manually identify an infection and scrub it in less than 60 mins i pull the system and run an offline scan using at lest two different offline scanners, because the rootkit has much less of a chance to hide (its not actively running).
...the computer would randomly play the "you got mail" beep, even though no mail program was running. One or at most a few times per day, and only when I wasn't using the computer. I assumed it was some program which threw up an error, but then my laptop, which had zero in common with my stationary machine except being networked to it, got the same symptom. And when I searched the net, I found reports from other people which seemed identical.
I could never find the culprit, despite testing every antivirus and rootkit I could find (including from live CD), and hijackthis didn't turn up anything suspicious. I'm still not sure there was a culprit, but whatever caused it disappeared after I switched system HD and reinstalled my stationary, and put Linux on my portable.
In my case I *suspect* the system played the 'you got mail' sound when the malware (if it was) phoned home.
You can't reliably find a rootkit whilst running the OS which is rooted (they tend to trapdoor themselves form the running OS, that's the whole point of being a rootkit)
In other words you need to scan the infected machine's disks _with something else_, such as a bootable linux distro (or soemthing like kaspersky's rescue disk)
To rule out worst case scenarios (bios infections) the disks have to be removed and checked on another system.
Of course, if you're paranoid you generate checksums for all files and then use a IDS. Decent backup systems do this for you to, so you can see when the checksum for any particular file changed.
The more I've looked at looked at Malware infections the most important thing to understand how it got on there.
A rootkit at the level this sound like administrator privileges would be required, where you running with administrator privileges or was an exploit used to escalate privileges? On the opportunistic drive by infections these are rarely used from what I've seen, targeted attacks are more likely to go the escalation of privilege route as they have the money & resources to tailor this stuff.
A low level rootkit as mentioned previously would hide itself, so you wouldn't necessarily see any processes, I've also read about Malware with it's own integrated IP stack, so completely bypasses the Windows IP stack and so most methods of monitoring communications on a compromised workstation.
I'd also recommend the use of an offline recovery disk (I know it's a bit late now) such as the Kaspersky Rescue CD.
First things first, disconnect the machine from the network. Any and all tools either go on a CD, or on a blank usb key that will be formatted after use.
On windows, would have been to check out the Services for anything not signed by Microsoft Corporation and seeing if anything there looked suspect amongst the various services and drivers that are loaded there, especially if there is somthing with the same name as somthing officially Microsoft.
From there, System internal's Process Explorer is a great tool, and can not only show you the process names, but also their paths. Nothing looks like svchost.exe like another svchost.exe, but when you see that one of them is being run from somwhere that is not windows\system32 , you have probably found your culprit.
A quick look at the registry run and runonce keys can turn up the usual suspects, but as there are so many startup points, a tool like autoruns (systeminternals again) can list a lot of this.
Kill any strange processes you may see in process explorer, stop anything wierd in the services, stop from starting anything that seems nasty from the registry. You could also run somthing like cureit that is supposed to be an efficient standalone scanner that runs on a live system. It's free too, so it's worth a shot and let it do it's magic. See if it finds anything, and clean it. You may not have to go any further.
While this is running, from a known clean system, download a live CD / Live usb antivirus solution, burn to a CD or copy this to another USB key (do not reuse the one used to load cureit & systeminternal's tools as it's potentially tainted now). If your are not happy with your live system check, then shut down the computer, then restart and boot on the CD/Key and let that antivirus go over the "dead" system - At the very least it will be able to clean a bootsector virus... and after that, using the live CD before rebooting, you can backup any wanted files from My Documents, your address book, your email app's data files and files on the desktop to an external drive. Disconnect the drive, and reboot...
If this solved your problem, you are golden. If it did not, then you have the reinstall route - just run the files you backed up after the liveCD AV scan through an antivirus on your clean system. You should have most if not all of your most recent data to hand, you now just have to spend a day on reinstalling windows, installing the updates & patches, reinstalling your tools & apps, patching them, then restoring your data. At worse, it would be a full weekend job...
1. Buy from e-bay or similar a USB to SATA adaptor, then remove your hard disk and connect it to a second (clean) computer with this adaptor. Now run your scans and tests without any root kit being able to easily hide itself.
2. Put in your diary a reminder, every two weeks to run backup. Use the Microsoft "Backup" (Create System Image) - to snapshot your C: drive, along with a daily or real-time data backup program.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
