Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add ability to always treat the test case as interesting initially #21

Open
dkasak opened this issue May 30, 2017 · 1 comment
Open

Comments

@dkasak
Copy link
Contributor

dkasak commented May 30, 2017

The ability to specify that lithium should treat the test case as interesting initially (before any chunk removals are done, regardless of whether it is actually interesting or not) would be a nice addition.

It would allow lithium to act as a fuzzer, as explained in the Reducers are Fuzzers blog post.

Some use cases:

  • It often happens that I have a hunch that a certain part of the code is broken, but I'm unable to trigger the faulty behaviour with a test case. What sometimes works is writing a test case that exercises this part of the code but is uninteresting (e.g. doesn't crash) and then letting lithium crashes run while forcing it to treat the case as interesting.
  • A fix for a bug had been deployed but you have a hunch that the fix isn't a proper one and just fixes it for some particular test case. This would allow you to uncover potential related test cases that still trigger the bug.

I'm not sure how to implement this best in lithium, though. Any suggestions? For now, I've just hacked together a non-crashes interestingness test for my personal use. It behaves exactly as crashes, except it forces the test case to be interesting in the initial run. The downside of this approach is that you can't reuse this code with other interestingness tests for which you might want to turn this behaviour on.

@nth10sd
Copy link
Contributor

nth10sd commented May 30, 2017

Having a non-crashes interestingness test would probably be the best way forward.

Note that it is possible to hook in multiple interestingness tests together. e.g. it is possible to use the range interestingness test with the crashes one.

Sample portion of a CLI command:

range 1 20 crashes --timeout=3

will run each iteration 20 times before determining interestingness using the crashes test.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants