The root of the problem

A rootkit is a set of programs that work to subvert control of an operating system from its legitimate operators. Usually, a rootkit will obscure its installation and attempt to prevent its removal through a subversion of standard system security.

Initially rootkits began replacing files on disk that belonged to the operating system itself, but today the threat is more advanced. “Around the year 2000, rootkits started moving to alter memory of the operating system, and today we're still seeing that evolution and trend,” says Jamie Butler, Principal Software Engineer at Mandiant and author of Rootkits: Subverting The Windows Kernel. “They aren't a new phenomenon. They just haven't attracted a lot of public attention until the last two or three years.”

So how have rootkits changed over the past decade? “To begin with, you would replace certain files on a file system,” Butler recalls. “On the Unix system, the login process could be replaced. In your subverted login process, the rootkit would log passwords as people logged in. That was one of the first examples of a rootkit. After that protection software caught up and processes did checksums and so forth, ensuring files in the file system did not change.”

Hooking became the next attack vector. Attackers knew if they altered the file system itself they would be detected. The rootkit then evolved when attackers learned that they could accomplish the same result by hooking operating system functions in memory in order to hide the attackers’ presence or to steal data from the user such as passwords.

After hooking technology, rootkits continue to advance; altering data structures in memory that the operating system relies upon. It’s notoriously hard to see data change in memory, because it’s unlikely you would know what dynamic memory was supposed to look like before it was changed. Rootkits continue to evolve and embed deeper into the OS itself.

Future threat

An attacker only has to beat the protection software it is up against, so attacks only evolve as quickly as protection does. Once the attacker sees the detection and prevention software, he then works out ways around it. “Attacks used to use more of a shotgun approach: worms spreading throughout the internet, for example, aimed at everyone and no one all at the same time,” says Butler. “Now attacks are focused and methodical, and understand the network as well as the system admin setting behind the computers. They understand what the firewall is letting out; they understand what version of the antivirus software is running, and what won't be detected.” They do this by probing the systems and moving slowly from host to host, eventually targeting the data they want to steal.

Better living through programming

To arm against rootkits, Butler points to Second Generation Digital Weaponry and Offensive Aspects of Rootkit Technology, both training courses that are designed to take an offensive stand against rootkits. “If you're a software developer working for a large antivirus company or operating systems company, the only way to learn how to defend against these things is to understand how they work at the code level. We demonstrate the actual code that rootkits use.

“For instance, if we are looking at the example of rootkits that want to hide a process or a registry key then we go through the exact steps the rootkit must take in order to do that level of subversion. Vendors and developers that come to our classes can learn how to improve their software and fend off these attacks.” Beyond this, Butler mentions how newer releases of the Windows operating system (Vista) have PatchGuard, a program that looks for changes to code in memory. This evolution took at least seven years, but at the very least, protection mechanisms are changing to meet these new dangers.

Implementing change

To ensure you are sufficiently protected from rootkits, the decision to address the threat has to come from the CIO. “This will best manage the threat, because the problem of rootkits is so pervasive or subversive to policy in multiple layers. We've seen that there is no single mandate you can put on an operating system to stop rootkits. It’s more of a policy issue where multiple things must be in place to mitigate the attack.” This begins with firewalls and IDSs, but those are old technologies that Butler has seen attackers get past.

Good host-based detection in the virus software, or a host-based intrusion prevention or detection system is invaluable; however, experience has taught that these solutions are often poorly implemented or hard to manage at the enterprise level. Since an attacker might get through all these security layers eventually, a logical, methodical, reactionary posture is necessary to prepare before your security has failed. Your enterprise should have processes in place to quickly identify exactly where the attacker is and how they got in. Then you can begin tightening up your system to avoid there being a ‘next time’.