feat(fuzz): report symbolic frontier failures#15646
Closed
figtracer wants to merge 2 commits into
Closed
Conversation
Member
Author
|
Closing: I opened this while misreading the retarget request. The intended work is to retarget #15631 instead. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
This makes targeted symbolic frontier solving surface replay-confirmed fuzz failures immediately.
Before this change,
--symbolic-use-fuzz-frontierscould solve a branch frontier, replay the candidate concretely, persist the resulting input into--fuzz-corpus-dir, and then still let the currentforge testinvocation pass. Users had to discover the failure through a laterforge fuzz replayrun.After this change, if the symbolic frontier candidate replays as a fuzz failure, Forge reports it as the fuzz counterexample for the current run and still persists the seed for later corpus replay.
Why
This is the smallest useful closed-loop behavior for the merged branch-frontier pipeline:
That makes the hybrid flow useful without requiring a separate manual replay step.
Benchmark
I used a small hard-interval fuzz target where baseline fuzz misses the bug with a fixed seed, while frontier-guided symbolic execution flips the recorded comparison.
Commands were run with compilation already warm and runtime caches cleared per sample.
--fuzz-runs 10000 --fuzz-seed 0x1234 --threads 1343.8 ms ± 2.9 ms64fuzz runs to capture frontier, then one targeted symbolic frontier solve144.3 ms ± 0.6 mshyperfinereports the hybrid flow as2.38x ± 0.02xfaster for this repro.Notes
This is deliberately much smaller than #15591. It builds on the already-merged stateless frontier capture/consume path and does not depend on the broader stateful invariant symbolic PR.
My read after this benchmark is that #15591 should be kept only if we want the separate stateful-invariant symbolic engine work on its own merits. It should not be the main vehicle for proving the hybrid fuzzer loop: this PR demonstrates the stateless frontier loop with a 3-file behavior change and a focused benchmark, while #15591 remains a broader 22-file symbolic/invariant change.
The second commit is a local clippy hygiene fix for an existing
forge fuzzparser stack-frame warning that reproduces on cleanorigin/masterwith current local nightly. It is kept separate from the behavior change.Validation
cargo +1.96.0 build -p forgecargo +1.96.0 check -p forgecargo +1.96.0 test -p forge --test cli test_cmd::symbolic::symbolic_fuzz_frontier_seeding_persists_branch_flipping_input -- --exact --nocapturecargo +nightly fmt --all -- --checkgit diff --checkcargo +nightly clippy -p forge --test cli -- -D warningsAuthored with AI assistance.