Make x509 and asn1 fuzzer reproducible

[thirdparty/openssl.git] / fuzz / README.md

# I Can Haz Fuzz?

LibFuzzer
=========

Or, how to fuzz OpenSSL with [libfuzzer](http://llvm.org/docs/LibFuzzer.html).

Starting from a vanilla+OpenSSH server Ubuntu install.

Use Chrome's handy recent build of clang. Older versions may also work.

    $ sudo apt-get install git
    $ mkdir git-work
    $ git clone https://chromium.googlesource.com/chromium/src/tools/clang
    $ clang/scripts/update.py

You may want to git pull and re-run the update from time to time.

Update your path:

    $ PATH=~/third_party/llvm-build/Release+Asserts/bin/:$PATH

Get and build libFuzzer (there is a git mirror at
https://github.com/llvm-mirror/llvm/tree/master/lib/Fuzzer if you prefer):

    $ cd
    $ sudo apt-get install subversion
    $ mkdir svn-work
    $ cd svn-work
    $ svn co http://llvm.org/svn/llvm-project/llvm/trunk/lib/Fuzzer
    $ cd Fuzzer
    $ clang++ -c -g -O2 -std=c++11 *.cpp
    $ ar r libFuzzer.a *.o
    $ ranlib libFuzzer.a

Configure for fuzzing:

    $ CC=clang ./config enable-fuzz-libfuzzer \
            --with-fuzzer-include=../../svn-work/Fuzzer \
            --with-fuzzer-lib=../../svn-work/Fuzzer/libFuzzer \
            -DPEDANTIC enable-asan enable-ubsan no-shared \
            -DFUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION \
            -fsanitize-coverage=trace-pc-guard,indirect-calls,trace-cmp \
            enable-ec_nistp_64_gcc_128 -fno-sanitize=alignment enable-tls1_3 \
            enable-weak-ssl-ciphers enable-rc5 enable-md2 \
            enable-ssl3 enable-ssl3-method enable-nextprotoneg \
            --debug
    $ sudo apt-get install make
    $ LDCMD=clang++ make -j
    $ fuzz/helper.py $FUZZER

Where $FUZZER is one of the executables in `fuzz/`.

If you get a crash, you should find a corresponding input file in
`fuzz/corpora/$FUZZER-crash/`.

AFL
===

Configure for fuzzing:

    $ sudo apt-get install afl-clang
    $ CC=afl-clang-fast ./config enable-fuzz-afl no-shared -DPEDANTIC \
        enable-tls1_3 enable-weak-ssl-ciphers enable-rc5 enable-md2 \
        enable-ssl3 enable-ssl3-method enable-nextprotoneg \
        enable-ec_nistp_64_gcc_128 -fno-sanitize=alignment \
        --debug
    $ make

The following options can also be enabled: enable-asan, enable-ubsan, enable-msan

Run one of the fuzzers:

    $ afl-fuzz -i fuzz/corpora/$FUZZER -o fuzz/corpora/$FUZZER/out fuzz/$FUZZER

Where $FUZZER is one of the executables in `fuzz/`.

Reproducing issues
==================

If a fuzzer generates a reproducible error, you can reproduce the problem using
the fuzz/*-test binaries and the file generated by the fuzzer. They binaries
don't need to be build for fuzzing, there is no need to set CC or the call
config with enable-fuzz-* or -fsanitize-coverage, but some of the other options
above might be needed. For instance the enable-asan or enable-ubsan option might
be useful to show you when the problem happens. For the client and server fuzzer
it might be needed to use -DFUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION to
reproduce the generated random numbers.

To reproduce the crash you can run:

    $ fuzz/$FUZZER-test $file

Random numbers
==============

The client and server fuzzer normally generate random numbers as part of the TLS
connection setup. This results in the coverage of the fuzzing corpus changing
depending on the random numbers. This also has an effect for coverage of the
rest of the test suite and you see the coverage change for each commit even when
no code has been modified.

Since we want to maximize the coverage of the fuzzing corpus, the client and
server fuzzer will use predictable numbers instead of the random numbers. This
is controlled by the FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION define.

The coverage depends on the way the numbers are generated. We don't disable any
check of hashes, but the corpus has the correct hash in it for the random
numbers that were generated. For instance the client fuzzer will always generate
the same client hello with the same random number in it, and so the server, as
emulated by the file, can be generated for that client hello.

Coverage changes
================

Since the corpus depends on the default behaviour of the client and the server,
changes in what they send by default will have an impact on the coverage. The
corpus will need to be updated in that case.