Security

Caught in the Act

Putting malware (safely) through its paces with dynamic analysis is often the fastest and most effective way to rid company computers of malicious code.

As noted in the March 2006 issue of BizTech (“Home Wreckers”), malware includes many types of undesirable, malicious code — such as viruses, worms, spyware and rootkits — that infect computers through unpatched software vulnerabilities, peer-to-peer file-sharing networks, e-mail attachments, specially crafted bogus Web pages and other means. While the previous article explored strategies and tools for finding malware and performing analysis using safer “static analysis” techniques, a more advanced approach, known as dynamic malware analysis, can be faster and more revealing, though potentially dangerous, if not done properly.

Unlike static analysis, which is similar to performing an autopsy or dissection — the files associated with a program are examined without running them, and the code is “dead” during the analysis — dynamic analysis involves the study of a program as it runs. Using system monitoring tools to observe the behavior of the malware at a very low level, dynamic analysis poses potential risks as well as rewards in rooting out the most malicious and surreptitious code.

Dynamic analysis requires the creation of a safe and isolated environment. Running live malware and doing so in an uncontrolled manner could cause serious damage to a computer or network. In the past, establishing a controlled environment for malware analysis involved setting up an entire network of machines that was physically separated from operational networks. Thankfully, the development of effective virtualization platforms such as VMware Workstation and Microsoft Virtual PC has eliminated this requirement.

In general, performing dynamic malware analysis in a virtual machine environment greatly reduces the risks involved with deliberately running malware. Virtual machines can easily be “reverted” to a pristine state and any malicious actions performed by the malware can be reversed with the press of a button. Any virtual machine used for dynamic malware analysis should be configured not to communicate with other hosts on the network, and any advanced features that allow communication between the virtual machine and the host computer should be disabled (such as the ability to share the data between a virtual machine and the host operating system using the system clipboard).

Once a virtual machine for testing has been created and the pre-malware image of the system has been established, dynamic analysis generally includes the following steps:

Turn on one or more system monitoring tools;
Run or install the suspect program;
Pause or stop collection of data by the system monitoring tools;
Analyze the results;
Refine the collection techniques or program invocation, and
Revert the virtual machine to the previously known “good” state and repeat to step 1 until you have completed your analysis.

Effective dynamic analysis requires tools that let the malware sleuth monitor, at minimum, these system variables: data written to or read from the hard drive(s), data read from or sent to the network and changes made to the system registry. Here are the tools and techniques needed to monitor each of these aspects of program behavior.

File Monitoring

Filemon is a simple and free program from SysInternals.com that monitors data being read from or written to disk by any program running on the system. In its default configuration, Filemon collects all file activity from every process running on a system. In general, this results in far too much data for efficient review. Thankfully, Filemon provides the ability to filter the data collected by monitoring only specific processes or particular types of transactions. For example, when analyzing a suspected spyware program named “s.exe,” the filter feature of Filemon can limit collection to information about files written to disk only by that program. Data can also be exported from Filemon into a delimited text file, which allows for easier analysis in a spreadsheet or database program.

Network Monitoring

Although there are many tools for monitoring and analyzing network traffic, a few simple tools and the open-source Ethereal network protocol analyzer are perfect for malware analysis. Ethereal (http://www.ethereal.com) can be used to collect any network traffic that is generated when the suspect program runs. If the malware attempts and fails to create a network connection to another machine, it might be necessary to set up a simulated network environment in order to explore the full functionality of the suspicious program. Using VMware, it is possible to simulate a complex network of machines using a single powerful workstation. Network services can be simulated by setting up additional virtual machines running services like a DNS server or Web server. Although establishing and setting up these fake network services is beyond the scope of this article, keep in mind that it is easy to set up listening network services using Netcat—the “Swiss Army Knife” of simple network utilities. Even if the suspicious code does not appear to generate any network traffic, programs like netstat (distributed with Windows), fport (www.foundstone.com), or openports (www.diamondcs.com.au) can help determine whether the malware has opened any listening network sockets.

Registry Monitoring

Changes to the System Registry can easily be observed using Regmon, another tool from SysInternals.com. Like its sibling Filemon, Regmon can easily produce enough output to overwhelm even the most experienced malware analyst. The key to effective use of Regmon is defining filters that limit collection of irrelevant data without throwing away important clues. Some of the things to watch for in Regmon output include:

Keys associated with the addition of programs that run automatically, such as the run key in registry entry
HKLM\Software\Microsoft\Windows\CurrentVersion\Run;

Keys associated with the creation of new drivers or services, such as those under the entries
HKLM\SYSTEM\CurrentControlSet\Services and HKLM\SYSTEM\CurrentControlSet\Enum\Root.

Some malware applications are distributed using an executable file that installs the malicious program in the same fashion as legitimate programs. Tools like InCtrl5 from PC Magazine (http://www.pcmag.com/article2/0,4149,9882,00.asp) and InstallWatch from EpsilonSquared (www.epsilonsquared.com) provide an easy way to monitor changes made to both files and the registry during the installation of a program.

Beyond Basics

If the steps above have not provided a sufficiently detailed understanding of the program being examined, a good debugger can save the day. Debuggers are powerful applications that give a user complete control over the execution of a program. A debugger will actually step through a program one instruction at a time. A debugger can also be used to examine program memory and temporary values that may not be available via any other method. Two free debuggers that are commonly used for malware analysis are OllyDbg (www.ollydbg.de) and Microsoft’s WinDbg (www.microsoft.com/whdc/devtools/debugging/default.mspx). Each debugger has its own set of quirks and esoteric command set. Although effective use of a debugger is a skill that may take a long time to develop, a debugger under the control of an experienced analyst is the single most powerful tool available for malware analysis.

Dynamic analysis of malware is a very powerful technique that can be used to clean up affected machines and develop host and network signatures for malware’s activity. These signatures can be used by IT professionals to find other computers that are affected by the same program, and to tune host and network defenses to catch and stop similar types of malware in the future. In most cases, a combination of static and dynamic analysis provides the best clues about how malware found its way onto the infected machine, what it does, and how it can be safely removed and protected against in the future.

Kris Kendall is principal engineer at Mandiant in New York City, where he leads the company's computer forensics and reverse engineering practices. He is a recognized expert on malware analysis and has instructed government special agents on the application of data-mining techniques for criminal and counterintelligence investigations.