Problem solve Get help with specific problems with your technologies, process and projects.

Windows rootkit detection tools and tactics

A reader asks Tom Chmielarski how to find and remove a rootkit on his Windows machine.

Send Tom your security questions

Have a security question about risk management? Windows security? Mobile devices? Send them to Tom, and he'll answer them in a future tip.

A reader asks our expert Tom Chmielarski, "I'm concerned that my Windows machine has a rootkit - what is the best way to detect and remove it?"

A typical virus or worm does not exactly advertise its presence, but its components are there to find if you know what to look for.

A rootkit, on the other hand, is an item of malware that is particularly stealthy; it hooks itself into the operating system and will not be shown by typical system utilities. It is often used to mask the presence of other malware on the system. A rootkit will usually be configured to hide additional malware, too, such as a keylogger or botnet-node. Keep in mind: rootkits exist on every major operating system platform and are certainly not unique to Windows.

I'm going to answer this question backwards and address removal first. Since the inherent nature of the rootkit is to hide itself, detection and removal of a rootkit is a tricky process. You can't simply use Regedit, the Microsoft Registry Editor, to find the registry keys that the malware is using; the rootkit will hide them.

When dealing with any malware that has had the opportunity to execute itself, modify your system, and then download other malware from the Internet, the best response is to back up your data and reinstall the operating system and applications from known-good media. While this scorched-earth type response may seem a bit extreme, it is the only way to really ensure you've removed it. The malware may have added users, changed system files, modified your OS's kernel, and performed any number of other unwanted changes. You may remove part of the compromise, but it is difficult to ensure that you've really corrected the damage. Even if you have removed the malicious component, the system changes themselves may linger and result in instability or unexpected system behavior.

More "Ask the Expert" responses

A reader asks Tom Chmielarski, "What do you consider to be the trade-offs of going with a best-of-breed product over all-in-one security software like Forefront?"

 Now let's move on to detecting a rootkit. Many antivirus products will detect rootkits with various levels of success. Generally speaking, you shouldn't expect the "scan now" antivirus to detect a rootkit. The AV product, however, may identify a dropper, or installer program, that wants to set up that rootkit. If the rootkit gets installed through a vulnerability -- perhaps using an unpatched Web browser flaw that is exploited when you visit a website that serves malicious advertisements -- then the antivirus tool probably won't help.

Since the primary objective of the rootkit is to hide itself, a good way to find a rootkit is to enumerate your system's contents and boot up using a known-good operating system. A bootable CD is an excellent choice for this and I've previously discussed creating a Bootable Windows CD. The default UBCD comes with RootKitty, an application for detecting rootkits by comparing the contents found when booted normally and when booted via UBCD. Microsoft also offers a free tool, RootkitRevealer, to do this type of search.

Another means to detect a rootkit, or other malware, is the use of a packet sniffer, such as WinDump, or a network firewall to detect the traffic emanating from your computer. If you start the suspect computer and, without direct user interaction, that computer tries to connect to IP addresses outside of your company (particularly universities and home Internet services), then the chances are good that it has something unwanted.

Windows rootkit detection and prevention is a complicated topic, and I'm only skimming the surface here. As a final thought on this topic I'd like to remind you of the infamous Sony rootkit from 2005. This rootkit was installed as a DRM component on computers that autoplayed certain Sony CDs. (Note: Mark Russinovich, author of the Microsoft RootkitRevealer, discovered the rootkit program). The removal of this corporate-designed software was tricky and potentially harmful to the computer, and this saga may be interesting for anyone who would like to know more about rootkits.

Tom Chmielarski is a senior consultant with GlassHouse Technologies, Inc.

Send Tom your security questions.

Join us on LinkedIn.

Dig Deeper on Microsoft security threat management

Start the conversation

Send me notifications when other members comment.

Please create a username to comment.