Comparing AFLplusplus sanitizers

[sheepbelldoor]

Hello fuzzbench team.

I am requesing an experiment because I want to measure the performance of each AFLplusplus sanitizers.
When your schedule permits, would it be possible for you to carry out an experiment using the specified --experiment-name and --fuzzers?
Thank you very much for considering my request and for your dedication, even amidst your busy schedule.

--experiment-name 2023-08-31-sanitizers
--fuzzers aflplusplus_afluse_asan aflplusplus_afluse_msan aflplusplus_afluse_ubsan aflplusplus_sanflags_asan aflplusplus_sanflags_msan aflplusplus_sanflags_ubsan aflplusplus_afluse_asan_ubsan aflplusplus_nothing

If there is a problem with my PR, please let me know.
Have a good day :D

[google-cla]

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

[sheepbelldoor]

Hi, fuzzbench team!

I was wondering if there are any issues with my recent pull request. If there are any concerns or problems, please do inform me so that I can address them promptly.

Additionally, I wanted to inquire if there is a predefined schedule for conducting Fuzzbench experiments. Knowing the schedule would greatly assist us in making timely requests for fuzzbench experiments.

I understand that you all have busy schedules, so please feel free to respond when it is convenient for you. I wouldn't want to disrupt your work in any way.

Thank you.
Have a wonderful day :D

[jonathanmetzman]

I can run it soon. Could you share why you want to run this experiment btw?

[sheepbelldoor]

Thank you for running soon :D
We are doing research on improvimg fuzzing performance. Especially, we have a conjecture that using address sanitizer all the time of a funzzing campaing is not efficient and decrease the coverage of the fuzzing campaign. The experiment we submitted is to validate our conjecture is true or not by running various fuzzing targets with and without address sanitizer. We will extend the possible configuration option from the address sanitizer to various other options that improve bug detection capability with the sacrifice of execution time.

[DonggeLiu]

Hi @sheepbelldoor, thanks for the explanation.
If I understand correctly, I presume this experiment seeks to compare the code-coverage ability of fuzzers?
If so, I can proceed with coverage-based benchmarks; Otherwise, I will use bug-based benchmarks.

Also, would you mind if I split the experiment into two, with 4 fuzzers in each?
The main reason is to prevent us from being bottlenecked at measurement, which happens when we have many fuzzers in a single experiment.
Thanks!

[DonggeLiu]

(A note for my future self) The commands to use are:

gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-10-05-sanitizers-1 --fuzzers aflplusplus_afluse_asan aflplusplus_afluse_msan aflplusplus_afluse_ubsan aflplusplus_sanflags_asan

gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-10-05-sanitizers-2 --fuzzers aflplusplus_sanflags_msan aflplusplus_sanflags_ubsan aflplusplus_afluse_asan_ubsan aflplusplus_nothing

[sheepbelldoor]

Your comprehension is accurate.
I request that you proceed with a coverage-based benchmark.
It is also okay to run the experiment twice with 4 fuzzers in each.
Thank you. Have a nice day!

[DonggeLiu]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-10-06-sanitizers-1 --fuzzers aflplusplus_afluse_asan aflplusplus_afluse_msan aflplusplus_afluse_ubsan aflplusplus_sanflags_asan

[DonggeLiu]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-10-06-sanitizers-2 --fuzzers aflplusplus_sanflags_msan aflplusplus_sanflags_ubsan aflplusplus_afluse_asan_ubsan aflplusplus_nothing

[DonggeLiu]

Experiment 2023-10-06-sanitizers-1 data and results will be available later at:
The experiment data.
The experiment report.

Experiment 2023-10-06-sanitizers-2 data and results will be available later at:
The experiment data.
The experiment report.

At the end of the report is a link to download the raw data (data.csv.gz), which can be used to re-generate the report with selected fuzzers and benchmarks.

[sheepbelldoor]

Hi fuzzbench team!

After this experiments started, I couldn't find any experiment data and results.
I found the following error looks like a network error in the build-logs dir in the experiment data

Step #5 - "aflplusplus_sanflags_asan-harfbuzz_hb-shape-fuzzer-intermediate-runner": #4 7.898   404  Not Found [IP: 185.125.190.36 80]
Step #5 - "aflplusplus_sanflags_asan-harfbuzz_hb-shape-fuzzer-intermediate-runner": #4 7.904 Fetched 27.2 MB in 5s (4954 kB/s)
Step #5 - "aflplusplus_sanflags_asan-harfbuzz_hb-shape-fuzzer-intermediate-runner": #4 7.904 E: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/libx/libx11/libx11-data_1.6.9-2ubuntu1.5_all.deb  404  Not Found [IP: 185.125.190.36 80]
Step #5 - "aflplusplus_sanflags_asan-harfbuzz_hb-shape-fuzzer-intermediate-runner": #4 7.904 E: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/libx/libx11/libx11-6_1.6.9-2ubuntu1.5_amd64.deb  404  Not Found [IP: 185.125.190.36 80]
Step #5 - "aflplusplus_sanflags_asan-harfbuzz_hb-shape-fuzzer-intermediate-runner": #4 7.904 E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/main/g/glibc/libc6-dbg_2.31-0ubuntu9.9_amd64.deb  404  Not Found [IP: 185.125.190.36 80]
Step #5 - "aflplusplus_sanflags_asan-harfbuzz_hb-shape-fuzzer-intermediate-runner": #4 7.904 E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
Step #5 - "aflplusplus_sanflags_asan-harfbuzz_hb-shape-fuzzer-intermediate-runner": #4 ERROR: executor failed running [/bin/sh -c apt install -y unzip git gdb joe]: exit code: 100
Step #5 - "aflplusplus_sanflags_asan-harfbuzz_hb-shape-fuzzer-intermediate-runner": ------
Step #5 - "aflplusplus_sanflags_asan-harfbuzz_hb-shape-fuzzer-intermediate-runner":  > importing cache manifest from gcr.io/fuzzbench/runners/aflplusplus_sanflags_asan/harfbuzz_hb-shape-fuzzer-intermediate:
Step #5 - "aflplusplus_sanflags_asan-harfbuzz_hb-shape-fuzzer-intermediate-runner": ------
Step #5 - "aflplusplus_sanflags_asan-harfbuzz_hb-shape-fuzzer-intermediate-runner": ------
Step #5 - "aflplusplus_sanflags_asan-harfbuzz_hb-shape-fuzzer-intermediate-runner":  > [2/2] RUN apt install -y unzip git gdb joe:
Step #5 - "aflplusplus_sanflags_asan-harfbuzz_hb-shape-fuzzer-intermediate-runner": ------
Step #5 - "aflplusplus_sanflags_asan-harfbuzz_hb-shape-fuzzer-intermediate-runner": executor failed running [/bin/sh -c apt install -y unzip git gdb joe]: exit code: 100
Finished Step #5 - "aflplusplus_sanflags_asan-harfbuzz_hb-shape-fuzzer-intermediate-runner"
ERROR
ERROR: build step 5 "gcr.io/cloud-builders/docker" failed: step exited with non-zero status: 1
--------------------------------------------------------------------------------

Also, I found same issue in another experiment(#1872).
In #1872, this issue had solved with adding an RUN apt-get update statement before RUN apt install -y unzip git gdb joe.
So, I modied runner.Dockerfile of the target fuzzers.

I hope this changes fixes the error.

If it doesn't interfere with your work and my modifyings aren't a problem, could you please run this experiment again when you have the time?

Thank you.
Have a nice day!

[DonggeLiu]

Hi @sheepbelldoor, my apologies for the late response: I was on leave in the past month.
Will re-run the experiment below.

[DonggeLiu]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-12-04-sanitizers-1 --fuzzers aflplusplus_afluse_asan aflplusplus_afluse_msan aflplusplus_afluse_ubsan aflplusplus_sanflags_asan

[DonggeLiu]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-12-04-sanitizers-2 --fuzzers aflplusplus_sanflags_msan aflplusplus_sanflags_ubsan aflplusplus_afluse_asan_ubsan aflplusplus_nothing

[DonggeLiu]

Experiment 2023-12-04-sanitizers-1 data and results will be available later at:
The experiment data.
The experiment report.

Experiment 2023-12-04-sanitizers-2 data and results will be available later at:
The experiment data.
The experiment report