Unexpected pipeline exit in AFL++ mode after modelling

[CounterCycle]

Hello,

I've been doing some testing with AFL++ after the fixes done in issue #7, thanks for resolving that. I've encountered a couple of additional issues.

Most significantly, some pipeline sessions exit during the middle of fuzzing. The failure seems to occur when a round of modelling is completed. An error line appear stating "Exit code 2 != 0 received from afl-showmap, terminating...". Then an exception in python for no such file or directory at /main/base_inputs. I've pasted the error output below.

I encountered this issue fuzzing the Zephyr SocketCAN binary from the fuzzware-experiments repo. I believe the issue is triggered when an input discovered using the old models triggers a firmware crash when run with the new models.

[!] Exit code 2 != 0 received from afl-showmap, terminating...
[10-25 14:22:25 ERROR] __init__.py - Got exception, shutting down pipeline: [Errno 2] No such file or directory: '/home/vagrant/fuzzware-experiments/02-comparison-with-state-of-the-art/uEmu/Zepyhr_SocketCan/run1/main002/base_inputs'
Traceback (most recent call last):
  File "/home/vagrant/fuzzware/pipeline/fuzzware_pipeline/session.py", line 230, in minimize_inputs
    run_corpus_minimizer(harness_args, self.temp_minimization_dir, self.base_input_dir, silent=silent, use_aflpp=self.parent.use_aflpp)
  File "/home/vagrant/fuzzware/pipeline/fuzzware_pipeline/run_fuzzer.py", line 39, in run_corpus_minimizer
    subprocess.check_call(full_args, env={**os.environ, **{'AFL_SKIP_CRASHES': '1'}})
  File "/usr/lib/python3.8/subprocess.py", line 364, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/home/vagrant/fuzzware/pipeline/fuzzware_pipeline/../../emulator/AFLplusplus/afl-cmin', '-m', 'none', '-U', '-t', '10000', '-i', '/home/vagrant/fuzzware-experiments/02-comparison-with-state-of-the-art/uEmu/Zepyhr_SocketCan/run1/main002/base_inputs_non_minimized', '-o', '/home/vagrant/fuzzware-experiments/02-comparison-with-state-of-the-art/uEmu/Zepyhr_SocketCan/run1/main002/base_inputs', '-e', '--', '/home/vagrant/.virtualenvs/fuzzware/bin/python', '-m', 'fuzzware_harness.harness', '-c', '/home/vagrant/fuzzware-experiments/02-comparison-with-state-of-the-art/uEmu/Zepyhr_SocketCan/run1/main002/config.yml', '@@']' returned non-zero exit status 2.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/vagrant/fuzzware/pipeline/fuzzware_pipeline/__init__.py", line 281, in do_pipeline
    pipeline.start()
  File "/home/vagrant/fuzzware/pipeline/fuzzware_pipeline/pipeline.py", line 751, in start
    self.handle_queue_forever()
  File "/home/vagrant/fuzzware/pipeline/fuzzware_pipeline/pipeline.py", line 704, in handle_queue_forever
    self.add_main_session(pending_prefix_candidate)
  File "/home/vagrant/fuzzware/pipeline/fuzzware_pipeline/pipeline.py", line 541, in add_main_session
    self.curr_main_session.minimize_inputs(prefix_candidate_path=prefix_input_candidate, is_previously_used_prefix=is_previously_used_prefix)
  File "/home/vagrant/fuzzware/pipeline/fuzzware_pipeline/session.py", line 239, in minimize_inputs
    shutil.rmtree(self.base_input_dir)
  File "/usr/lib/python3.8/shutil.py", line 709, in rmtree
    onerror(os.lstat, path, sys.exc_info())
  File "/usr/lib/python3.8/shutil.py", line 707, in rmtree
    orig_st = os.lstat(path)
FileNotFoundError: [Errno 2] No such file or directory: '/home/vagrant/fuzzware-experiments/02-comparison-with-state-of-the-art/uEmu/Zepyhr_SocketCan/run1/main002/base_inputs'
[10-25 14:22:25 INFO] __init__.py - Shutting down pipeline now

Additionally, I've noticed some minor issues with other components when AFL++ is used. Using fuzzware fuzz in AFL++ mode failed to start, even when pipeline works. And the output from fuzzware genstats coverage contains incorrect timestamps for block discoveries, I'm guessing because AFL++ already uses relative times in the plot_data file, while AFL used Unix time.

Thank-you,
CounterCycle

[Scepticz]

Hi CounterCycle,

thank you for reporting the issues. The AFL++ support could use some more love. I think you grasp the issues pretty well!

Are you able to tackle some of these and maybe create a pull request?

Tobi

[CounterCycle]

I made a pull request to the fuzzware-pipeline repo to address the main issue. fuzzware-fuzzer/fuzzware-pipeline#4

It's not a perfect solution, as it still falls back on using the entire corpus, rather than fixing the minimization. I think a proper fix would require patching AFL++.

I don't think I'll have a chance to fix the other issues in the near future, so I'll leave them to someone else if they can take a go at fixing them. They don't prevent fuzzing of any binaries, but would be nice to have.

Thanks,
CounterCycle