Security

Secure Testing Policies

Putting fuzz security testing into the development life cycle.

You can never test quality or security into an application. If the app is written in an insecure manner with poor coding practices or has a large attack surface, no amount of testing will make it secure.

Although known to only a few developers, non-security experts can conduct effective security testing, most notably by fuzz testing. Fuzz testing includes penetration testing, run-time verification, re-reviewing threat models and re-evaluating the attack surface. By fuzz testing all the file formats your app consumes and the “parser per” format and network protocols with man-in-the-middle techniques, you’ll provide a “sanity check” of code before release.

Fuzz Testing

Originally developed to find reliability bugs, fuzz testing is an effective way to find certain classes of security bugs as well. Fuzzing means creating malformed data and having the app under test consume the data to see how it reacts. If the app fails unexpectedly, a bug will be found. The bug is a reliability bug and, possibly, a security bug. Fuzzing is aimed at exercising code that analyzes data structures — loosely referred to as parsers. There are three broad classes of parsers:

File format parsers

Examples include code that manipulates graphic images (JPEG, BMP, WMF, TIFF) or document files (DOC, PDF, ELF).

Network protocol parsers

Examples include SMB, TCP/IP, NFS, SSL/TLS, RPC, and AppleTalk. You also can fuzz the order of network operations — for example, by performing a response before a request.

APIs and miscellaneous parsers

Examples include browser-pluggable protocol handlers, such as callto.

File Fuzzing

Fuzzing file formats means building malformed files to be consumed by your app. For example, if your app parses and displays TIFF files, you could build a malformed TIFF and have the app read the file. Of course, you don’t create just one malformed file; SDL mandates that you create and test at least 100,000 malformed files for every file format and parser you support.

A Generic File-fuzzing Process

The process for fuzzing files is simple. It consists of the following steps:

Identify all the file formats your app supports.
Collect a library of valid files your app supports.
Malform (fuzz) a file.
Have the app consume the malformed file, and then observe the app.

The following paragraphs address each of these steps in detail.

Identify all valid file formats

The first step in the file-fuzzing process is to identify all file formats your app reads and handles.

Your app should fail gracefully if faced with a file format it does not render or understand. Similarly, any component you have developed should fail gracefully and, just as important, bubble errors up to the next level of code.

Collect a library of valid files

You should gather as many valid files from as many trusted sources as possible — and the files should represent a broad spectrum of content. Aim for at least 100 files of each supported file type to get reasonable coverage. For example, if you manufacture digital photography equipment, you’ll need representative files from every camera and scanner you build.

You should continue to build on this library over time as you define new formats or new format variants.

Malform a file

The work really starts when you begin malforming a file. You need to build or use a tool that chooses a file at random, malforms the file and then passes the file to the software under test (van Sprundel 2005, Sutton and Greene 2005, Oehlert 2005).

The two broad classes of file fuzzing are smart fuzzing and dumb fuzzing. Smart fuzzing is when you know the data structure of the file format and you change specific values within the file. Dumbfuzzing is when you change the data at random. For example, PNG files start with a well-known signature followed by a series of blocks, named chunks (Milano 1999). The signature is 8 bytes long and must have the following value: 0x89 0x50 0x4E 0x47 0x0D 0x0A 0x1A 0x0A.

Each chunk has the following format:

4-byte length

The number of bytes in the data field

4-byte type

The name of the chunk (such as IHDR or IDAT)

n-byte data

The data, the format of which depends on the chunk type

4-byte CRC (cyclical redundancy check)

A CRC-32 calculated from the data

The IHDR chunk type always follows the signature, specifies image dimensions and color information, and has the following format:

4-byte width

Image width in pixels

4-byte height

Image height in pixels

1-byte bit depth

1, 2, 4, 8, or 16 bits per pixel

1-byte color type

0 for grayscale, 2 for RGB, 3 for palette, 4 for gray with alpha channel and 6 for RGB and alpha

1-byte compression mode

Always zero

1-byte filter mode

Always zero

1-byte interlace mode

0 for none and 1 for Adam-7 format

It’s important to know also that PNG files structure multibyte integers with the most significant byte first.

Knowing the basic PNG format and the IHDR chunk type, you can be very specific about how you corrupt a PNG file. We’ll give file corruption examples later.

Dumb fuzzing is a shotgun approach: you take a valid file and randomly corrupt it. It really is that simple. You can smart fuzz or dumb fuzz a file in many ways, including these:

Making the file smaller than normal
Filling the entire file with random data
Filling portions of the file with random data
Searching for null-terminated strings (in ASCII and Unicode) and setting the trailing null to non-null
Setting numeric data types to negative values
Exchanging adjacent bytes
Setting numeric data types to zero
Toggling, setting, or clearing high bits (0x80, 0x8000, and so on)
Doing an exclusive OR (XOR) operation on all bits in a byte, one bit at a time
Setting numeric data types to 2^N +/− 1

Looking back at the PNG format, you could be very specific and smart fuzz a file by using the following techniques:

Set the chunk length to a bogus value.
Create random chunk names. (They are case-sensitive, and the case has specific meaning.)
Build a file with no IHDR chunk.
Build a file with more than one IHDR chunk.
Set the width, height or color depth to invalid values (0, negative values, 2^N +/− 1, little endian, and so on).
Set invalid compression, filter or interlace modes.
Set an invalid color type.

In the PNG example, you would also need to build a valid CRC for each malformed file; otherwise, a CRC failure would prevent most of the parsing code from being exercised.

Consume the file and observe the application

Finally, have the application consume the file — for example, via a command-line argument that includes the file name. As the app runs, monitor for failures such as access violations or core dumps, and watch for spiked CPU usage. In Microsoft Windows, you can set the fuzzing tool to be a mini-debugger by using debugging APIs (Robbins 2003), and then you can write failure information to a log file. The sample fuzzer, MiniFuzz, shows how to do this.

Michael Howard is a senior security program manager at Microsoft and author of The Security Development Lifecycle (Microsoft Press, 2006). The publication includes a simple file fuzzer named MiniFuzz (written in C++), which demonstrates the malforming process.