Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding muttfuzz and requesting an experiment #1967

Open
wants to merge 10 commits into
base: master
Choose a base branch
from

Conversation

kjain14
Copy link
Contributor

@kjain14 kjain14 commented Apr 13, 2024

Adding MuttFuzz (a binary fuzzing alternative to our "First, Fuzz the Mutants" paper). This will likely integrate better with fuzzbench and thus work on experiments, as we are mutating binaries rather than source code.

@DonggeLiu
Copy link
Contributor

Hi, @kjain14, long time no see!
Thanks for updating the experiment request file.

We have changed our way of running experiments, hope this suits you:

  1. Would you mind making a trivial modification to service/gcbrun_experiment.py?
    This will allow me to launch experiments in this PR before merging.
    Here is an example to add a dummy comment, thanks!

  2. Could you please write your experiment request in this format?
    You can swap the --experiment-name and --fuzzers parameters with your values.
    For example, in your case, the request command this time is

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-04-23-aflpp-muttfuzz --fuzzers aflplusplus_muttfuzz
  1. You no longer have to edit service/experiment-requests.yaml in the future : )

@DonggeLiu
Copy link
Contributor

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-04-23-aflpp-muttfuzz --fuzzers aflplusplus_muttfuzz

@DonggeLiu
Copy link
Contributor

DonggeLiu commented Apr 23, 2024

Experiment 2024-04-23-aflpp-muttfuzz data and results will be available later at:
The experiment data.
The experiment report.

@kjain14
Copy link
Contributor Author

kjain14 commented Apr 23, 2024

We accidentally only ran muttfuzz, we also want to compare against aflplusplus

@kjain14
Copy link
Contributor Author

kjain14 commented Apr 23, 2024 via email

@kjain14
Copy link
Contributor Author

kjain14 commented Apr 23, 2024

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-04-23-full-muttfuzz --fuzzers aflplusplus_muttfuzz aflplusplus aflsmart_muttfuzz aflsmart libfuzzer libfuzzer_muttfuzz libafl libafl_muttfuzz honggfuzz honggfuzz_muttfuzz

@DonggeLiu
Copy link
Contributor

DonggeLiu commented Apr 23, 2024

Thanks for providing us with the experiment request command, @kjain14!

I noticed that the experiment requires 10 fuzzers, which may overwhelm FuzzBench due to its current bottleneck in the measurement stage. Empirically, it's safer to keep the number under 8.
Meanwhile, this config enables FuzzBench to merge fuzzers' results from past experiments into your report when your experiment completes so that we don't have to re-run them in every new experiment.
For example, your previous experiment report should have data of other fuzzers (including afl++) once it completes.
Here is another example command and its report. The report merges core fuzzers' results from past experiments even though they were not explicitly mentioned in the command.

I will revise the new command below, please let me know if they work for you.

@kjain14
Copy link
Contributor Author

kjain14 commented Apr 23, 2024

Thanks! Yes merging them should work!

@DonggeLiu
Copy link
Contributor

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-04-23-full-muttfuzz --fuzzers aflplusplus_muttfuzz aflsmart_muttfuzz libfuzzer_muttfuzz libafl_muttfuzz honggfuzz_muttfuzz

@DonggeLiu
Copy link
Contributor

The request failed because the fuzzer name is libafl_muttfuzz but its directory name is libafll_muttfuzz (with two ls in afll), is this a typo?
Would you like to change it or it's ok to use that name?

@kjain14
Copy link
Contributor Author

kjain14 commented Apr 24, 2024

Apologies for the typo, this should be fixed now

@DonggeLiu
Copy link
Contributor

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-04-24-full-muttfuzz --fuzzers aflplusplus_muttfuzz aflsmart_muttfuzz libfuzzer_muttfuzz libafl_muttfuzz honggfuzz_muttfuzz

@DonggeLiu
Copy link
Contributor

Experiment 2024-04-24-full-muttfuzz data and results will be available later at:
The experiment data.
The experiment report.
The experiment report(experimental).

@kjain14
Copy link
Contributor Author

kjain14 commented May 2, 2024

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-05-02-full-muttfuzz --fuzzers aflplusplus_muttfuzz aflsmart_muttfuzz libfuzzer_muttfuzz libafl_muttfuzz honggfuzz_muttfuzz

@DonggeLiu
Copy link
Contributor

Hi @kjain14, would you mind if I start this experiment next week?

We were trying to investigate this failure and it would be great if we could avoid starting another large experiment at the meantime.

Thanks!

@kjain14
Copy link
Contributor Author

kjain14 commented May 3, 2024

Sure, we can start it next week

@DonggeLiu
Copy link
Contributor

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-05-06-full-muttfuzz --fuzzers aflplusplus_muttfuzz aflsmart_muttfuzz libfuzzer_muttfuzz libafl_muttfuzz honggfuzz_muttfuzz

@DonggeLiu
Copy link
Contributor

Experiment 2024-05-06-full-muttfuzz data and results will be available later at:
The experiment data.
The experiment report.
The experiment report(experimental).

@vanhauser-thc
Copy link
Collaborator

@kjain14 fyi I saw your experiments. if you did not integrate muttfuzz on the same commit as aflplusplus that you fuzzbench is using, you will be comparing apples with oranges because there were significant advances. same if your muttfuzz afl++ commit is newer than the one on fuzzbench. I didnt check your setup. just in case: better add an aflplusplus variant that has the exact same commit id you have based your muttfuzz integration on.
comparing against honggfuzz looks good, so I would expect it should also improve afl++.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants