VM Rootkits: The Next Big Threat?

By Ryan Naraine March 10, 2006
Lab rats at Microsoft Research and the University of Michigan have teamed up to create prototypes for virtual machine-based rootkits that significantly push the envelope for hiding malware and that can maintain control of a target operating system. The proof-of-concept rootkit, called SubVirt, exploits known security flaws and drops a VMM (virtual machine monitor) underneath a Windows or Linux installation.     Once the target operating system is hoisted into a virtual machine, the rootkit becomes impossible to detect because its state cannot be accessed by security software running in the target system, according to documentation seen by eWEEK. The prototype, which will be presented at the IEEE Symposium on Security and Privacy later in 2006, is the brainchild of Microsoft's Cybersecurity and Systems Management Research Group, the Redmond, Wash., unit responsible for the Strider GhostBuster anti-rootkit scanner and the Strider HoneyMonkey exploit detection patrol. Today, anti-rootkit clean-up tools compare registry and file system API discrepancies to check for the presence of user-mode or kernel-mode rootkits, but this tactic is useless if the rootkit stores malware in a place that cannot be scanned. "We used our proof-of concept [rootkits] to subvert Windows XP and Linux target systems and implemented four example malicious services," the researchers wrote in a technical paper describing the attack scenario.   "[We] assume the perspective of the attacker, who is trying to run malicious software and avoid detection. By assuming this perspective, we hope to help defenders understand and defend against the threat posed by a new class of rootkits," said the paper, which is co-written by researchers from the University of Michigan. Stealth rootkits are bombarding Windows XP SP2 systems. Click here to read more. A virtual machine is one instance of an operating system running between the hardware and the "guest" operating system. Because the VM sits on the lower layer of the operating system, it is able to control the upper layers in a stealthy way.   "[T]he side that controls the lower layer in the system has a fundamental advantage in the arms race between attackers and defenders," the researchers said. "If the defender's security service occupies a lower layer than the malware, then that security service should be able to detect, contain and remove the malware. Conversely, if the malware occupies a lower layer than the security service, then the malware should be able to evade the security service and manipulate its execution." The group said the SubVirt project implemented VM-based rootkits on two platforms—Linux/VMWare and Windows/VirtualPC—and was able to write malicious services without detection. The paper describes how easy it is to get the VM-based malware on a target system. For example, a code execution flaw could be exploited to gain root or administrator rights to manipulate the system boot sequence. Once the rootkit is installed, it can use a separate attack operating system to deploy malware that is invisible from the perspective of the target operating system. "Any code running within an attack OS is effectively invisible. The ability to run invisible malicious services in an attack OS gives intruders the freedom to use user-mode code with less fear of detection," the researchers said. The group used the prototype rootkits to develop four malicious services—a phishing Web server, a keystroke logger, a service that scans the target file system for sensitive information and a defense countermeasure to defeat existing VM-detection systems. The researchers also used the VM-based rootkits to control the way the target reboots. It could also be used to emulate system shutdowns and system sleep states.   While the prototype rootkits are theoretically offensive in nature, the researchers also discussed ways to defend against malicious use of VM. Where are rootkits coming from? Read more here. The group suggests that hardware detection is one way to gain control over the lower layer to detect VM-based rootkits, pointing out that chip makers Intel and AMD have proposed hardware that can be used to develop and deploy low-layer security software that would run beneath a VM-based rootkit.   Another defense technique the researchers proposed is to boot from a safe medium such as a CD-ROM, USB drive or network boot server to gain control below the rootkit. A secure VMM can also be used to gain control of a system before the operating system boots. It can also be used to retain control as the system runs and to add a check to stop a VM-based rootkit from modifying the boot sequence. Ziff Davis Media eSeminars invite: Learn how to proactively shield your organizations against threats at all tiers of the network, Symantec will show you how, live on March 21 at 4 p.m. ET. Sponsored by Symantec. "We believe the VM-based rootkits are a viable and likely threat," the research team said. "Virtual-machine monitors are available from both the open-source community and commercial vendors ... On today's x86 systems, [VM-based rootkits] are capable of running a target OS with few visual differences or performance effects that would alert the user to the presence of a rootkit." The threat is so real, the group said, that during the creation of SubVirt, one of the authors accidentally used a machine that had been infected by the proof-of-concept rootkit without realizing that he was using a compromised system.