Hoedur Experiments

This repository contains all data required to reproduce our target binaries and configurations and to re-run the experiments from the paper HOEDUR: Embedded Firmware Fuzzing using Multi-Stream Inputs published at USENIX Security 2023. For the fuzzer implementation itself, please refer to the hoedur repository.

The repository contains scripts and data to do the following:

1. Rerun all of our experiments with configurable CPU requirements.
2. Rebuild our new firmware target binaries and apply binary patches for the established data set.
3. Reproduce our auto-conversion of configurations and bug detection hooks between Hoedur and Fuzzware.

Repository Structure

Path	Description
generate_host_run_config.py	Script to generate experiment run configurations based on available hosts configured in available_hosts.txt
run_experiment.py	Script to execute an experiment run configuration generated by generate_host_run_config.py.
reproduce_targets_and_configs.py	Script to re-create and verify the targets and auto-converted configurations that we provid in this repository. While this is not required to run the experiments, it can be used to verify our builds and configurations.
experiment-config	Scripts that enable you to choose experiment durations and prepare running the experiments in a distributed way.
01-bug-finding-ability	Our first experiment which measures bug detection timings on the Fuzzware CVE target set. See Section "6.2 Bug Finding Ability" in the paper.
02-coverage-est-data-set	Our second experiment which measures code coverage on the established P2IM/uEmu/Fuzzware target set. See Section "6.3 Code Coverage: Established Data Set" in the paper.
03-advanced-mutations	Our third experiment which measures code coverage when using additional dictionary configurations on a subset of the established P2IM/uEmu/Fuzzware target set. See Section "6.4 Advanced Mutations via Dictionaries" in the paper.
04-prev-unknown-vulns	Our fourth experiment where we tested additional targets using Hoedur. See Section "6.5 Finding Unknown Vulnerabilities" in the paper.
targets	Pre-built samples and configurations of target firmware used in the experiments.

Getting Started

We designed our scripts to require python3, sudo, rsync, make, and docker. Make sure that these are installed on your system and available to your user. Then run:

# Install
./install.py
# Set up host for related work Fuzzware reproduction
sudo ./scripts/fuzzware/set_limits_and_prepare_afl.sh

And to kick off a test version of the experiments, run

# Set an experiment configuration that is feasible to run on a single host
echo "test-run" > experiment-config/active_profile.txt
./generate_host_run_config.py
# Run the experiment
./run_experiment.py experiment-config/host-run-configs/localhost.yml

Handling Reproducing the Experiments

Our experiments have been run on a set of 10 hosts running Ubuntu 22.04.1 LTS and using the following hardware: two Intel Xeon Gold 5320 CPUs @ 2.20GHz (52 physical cores in total), 256 GB of RAM, and SSD memory for storage. The original experiments consume a considerable amount of computation power. For a rough breakdown of the CPU requirements for the original experiment, consider the following experiments:

1. 01-bug-finding-ability: 12 targets, 15 days per run, 5 repetitions, 3 Fuzzers, 4 cores per run. Total number of core days: 10800 core days (29.5 CPU years).
2. 02-coverage-est-data-set: 20 targets, 1 day per run, 10 repetitions, 3 Fuzzers. Total CPU time: 600 core days (1.5 CPU years).
3. 03-advanced-mutations: 4 targets, 1 day per run, 10 repetitions, 2 (additional) Fuzzers: 80 CPU days.
4. 04-prev-unknown-vulns: 14 targets, 1 day per run, 52 repetitions, 1 Fuzzer: 728 CPU days (2 CPU years).

To easen the process of reproducing our experiments, we take two steps:

1. experiment profiles: We provide pre-set and adjustable configurations that specify different fuzzing durations and repetition counts to make the experiments less CPU intensive. To reproduce the main results with heavily reduced CPU requirements, we recommend the default shortened-eval profile which is pre-configured in experiment-config/active_profile.txt. For more details about the different experiment profiles, see experiment-config.
2. config autogeneration: Based on the experiment profiles and a configuration of available hosts, generate_host_run_config.py assists in splitting workloads among available hosts and provides configurations to run on each host. To configure available hosts, see experiment-config/README.md.

More information on configuring these options is available in the experiment-config directory.

Workflow to re-run the Experiments using remote Hosts

Take the following steps to re-run the experiment distributed to multiple machines. We split the workflow into these steps to make the overall process transparent:

1. Choose the hosts to use for running the experiments and configure them in experiment-config/available_hosts.txt. Set up an SSH config (~/.ssh/config) to be able to ssh into each host via ssh my-host.
2. Install and set up each host via ./install.py and ./scripts/fuzzware/set_limits_and_prepare_afl.sh.
3. Generate experiment run configurations by running ./generate_host_run_config.py. For information on how to customize these configs, see experiment-config.
4. Upload the generated configurations that will be located under experiment-config/host-run-configs to the respective experiment hosts. This can be done either manually or by running ./generate_host_run_config.py --upload.
5. Run the experiments for the uploaded experiment configurations (in tmux or similar) on the respective experiment servers via ./run_experiment.py experiment-config/host-run-configs/<hostname>.yml. Note that we do not fully automate this step to allow you to check the environment on each host, clean up any possible pre-existing results, and to deal with other additional requirements that are specific to your experiment hosts.
6. After running the experiments, sync the results via rsync or by using the provided utility ./sync_experiment_data.py. Note that depending on your network stability, you may need to run the syncing script multiple times.
7. Compute the experiment metrics (coverage, bug detection timings, ...) using ./compute_metrics.py.
8. Inspect the results in the experiment directories 01-bug-finding-ability, 02-coverage-est-data-set, 03-advanced-mutations, and 04-prev-unknown-vulns. The respective directories document the available data, how to interpret it, and what the expected outcomes are.

To make the experiment hosts available via SSH, entries in ~/.ssh/config might look like this:

Host my-host-1
    Hostname 12.34.56.78
    User user
    IdentityFile ~/.ssh/my_privkey.key
    AddKeysToAgent yes

Reproducing the Target Binaries and Configs

We provide copies of all pre-built binaries and configurations in this repository. As such, you can rerun the experiments without rebuilding (see run_experiment.py). We provide these scripts to allow understanding, reproducing, adapting, and extending our binaries, configurations, and bug detection scripts.

With that said, to install the required python packages, first run

pip install -r 04-prev-unknown-vulns/building/requirements.txt

For the reproduction, run:

./reproduce_targets_and_configs.py

The reproduction contains the following steps:

1. Fuzzware to Hoedur config conversion: Automatically convert all previously published Fuzzware configurations into their Hoedur equivalents and verify that the configurations match the pre-built ones. For example, the Hoedur configuration config.yml is automatically generated from the corresponding config_fuzzware.yml.
2. Established data set binary patching: Apply the commented binary patches to the original binaries from the established firmware data set and verify that the patched binaries match the patched binary versions.
3. Bug detection hook conversion: Convert the Hoedur bug detection hooks into Fuzzware equivalents and verify that the Fuzzware detection scripts match the pre-built ones. For example, the Hoedur configuration file hook-bugs.rn is automatically converted into the Fuzzware equivalent hook_bugs_fuzzware.py.
4. New target building: Create build environments and re-build the new targets in which Hoedur found bugs with CVEs assigned. Verify that the newly-built binaries match the pre-built ones in targets/arm/Hoedur.

Manually Testing Firmware in Hoedur

In this repository we provide scripts ready to reproduce all of our paper evaluation experiments. However, you can also test a provided (or your own) firmware target with Hoedur directly.

Running Hoedur on a given Target

You can run Hoedur on a preconfigured firmware target (in this example, CVE-2022-39274) like so:

scripts/run_in_docker.sh /home/user/hoedur/scripts/fuzz-local.py --runs 2 --duration 20m --log --name 'my-test-target' --targets Hoedur/loramac-node/CVE-2022-39274

After this fuzzing is completed, corpus archives can can be found in ./corpus/my-test-target. Given this corpus, we can perform a further analysis such as generate the fuzzing coverage:

scripts/run_in_docker.sh /home/user/hoedur/scripts/fuzz-coverage-list.py corpus/my-test-target-hoedur/ --no-basic-block-filter --targets Hoedur/loramac-node/CVE-2022-39274

The resulting coverage can now be found in ./corpus/my-test-target/_bb.

Creating a Configuration for a new Target Firmware

For a given ARM Cortex-M0/3/4 firmware binary, we need to first create a config.yml Hoedur configuration file. One way to do this is to manually create such a configuration file. For different configurations, you can check some of the Hoedur configurations in this repository. You can refer to the configuration of the P2IM CNC target firmware as an example.

For a more automated approach, you can use the fuzzware genconfig script in conjunction with our configuration conversion tool to automatically generate a configuration for a given firmware ELF file. For example if we wanted to create a new copy of an ELF and configure it automatically to run in Hoedur, we could take the following steps:

# Prepare new targets dir and copy ELF
mkdir -p targets/arm/my-new-targets/WYCINWYC
cp targets/arm/WYCINWYC/XML_Parser/XML_Parser.elf targets/arm/my-new-targets/WYCINWYC

# Generate the Fuzzware configuration
./scripts/fuzzware/run_fuzzware_docker.sh fuzzware genconfig targets/arm/my-new-targets/WYCINWYC/XML_Parser.elf
mv targets/arm/my-new-targets/WYCINWYC/config.yml targets/arm/my-new-targets/WYCINWYC/config_fuzzware.yml

# Convert the Fuzzware configuration into the Hoedur format
./scripts/run_in_docker.sh hoedur-convert-fuzzware-config targets/arm/my-new-targets/WYCINWYC/config_fuzzware.yml targets/arm/my-new-targets/WYCINWYC/config.yml

You can now find a the generated Hoedur configuration file in targets/arm/my-new-targets/WYCINWYC/config.yml. You can now fuzz test this target as seen before. Here the command-line is the following:

scripts/run_in_docker.sh /home/user/hoedur/scripts/fuzz-local.py --runs 2 --duration 2m --log --name 'my-newly-configured-target' --targets my-new-targets/WYCINWYC

Citing our Paper

To cite our paper, you may use the following BibTeX entry:

@inproceedings {scharnowski2023hoedur,
    title = {Hoedur: Embedded Firmware Fuzzing using Multi-Stream Inputs},
    booktitle = {32nd USENIX Security Symposium (USENIX Security 23)},
    year = {2023},
    address = {Boston, MA},
    url = {https://www.usenix.org/conference/usenixsecurity23/presentation/scharnowski},
    publisher = {USENIX Association},
    author={Scharnowski, Tobias and Woerner, Simon and Buchmann, Felix and Bars, Nils and Schloegel, Moritz and Holz, Thorsten},
    month = aug,
}