-
-
Notifications
You must be signed in to change notification settings - Fork 26
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
1 parent
9b1821a
commit 923b960
Showing
10 changed files
with
115 additions
and
89 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,18 +1,22 @@ | ||
# Getting started | ||
|
||
Just run `cargo mutants` in a Rust source directory, and it will point out | ||
functions that may be inadequately tested: | ||
functions that may be inadequately tested. | ||
|
||
## Example | ||
|
||
```none | ||
; cargo mutants | ||
Found 14 mutants to test | ||
Copy source to scratch directory ... 0 MB in 0.0s | ||
Unmutated baseline ... ok in 1.6s build + 0.3s test | ||
Auto-set test timeout to 20.0s | ||
src/lib.rs:386: replace <impl Error for Error>::source -> Option<&(dyn std::error::Error + 'static)> with Default::default() ... NOT CAUGHT in 0.6s build + 0.3s test | ||
src/lib.rs:485: replace copy_symlink -> Result<()> with Ok(Default::default()) ... NOT CAUGHT in 0.5s build + 0.3s test | ||
src/lib.rs:386: replace <impl Error for Error>::source -> Option<&(dyn std::error::Error + 'static)> | ||
with Default::default() ... NOT CAUGHT in 0.6s build + 0.3s test | ||
src/lib.rs:485: replace copy_symlink -> Result<()> with Ok(Default::default()) ... | ||
NOT CAUGHT in 0.5s build + 0.3s test | ||
14 mutants tested in 0:08: 2 missed, 9 caught, 3 unviable | ||
``` | ||
|
||
In v0.5.1 of the `cp_r` crate, the `copy_symlink` function was reached by a test | ||
but not adequately tested. | ||
but not adequately tested, and the `Error::source` function was not tested at all. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,27 @@ | ||
# How is mutation testing different to coverage measurement? | ||
|
||
Coverage measurements tell you which lines of code (or other units) are reached | ||
while running a test. They don't tell you whether the test really _checks_ | ||
anything about the behavior of the code. | ||
|
||
For example, a function that writes a file and returns a `Result` might be | ||
covered by a test that checks the return value, but not by a test that checks | ||
that the file was actually written. cargo-mutants will try mutating the function | ||
to simply return `Ok(())` and report that this was not caught by any tests. | ||
|
||
Historically, rust coverage measurements have required manual setup of several | ||
OS and toolchain-dependent tools, although this is improving. Because | ||
`cargo-mutants` just runs `cargo` it has no OS-specific or tight toolchain | ||
integrations, and so is simple to install and run on any Rust source tree. | ||
cargo-mutants also needs no special tools to view or interpret the results. | ||
|
||
Coverage tools also in some cases produce output that is hard to interpret, with | ||
lines sometimes shown as covered or not due to toolchain quirks that aren't easy | ||
to map to direct changes to the test suite. cargo-mutants produces a direct list | ||
of changes that are not caught by the test suite, which can be quickly reviewed | ||
and prioritized. | ||
|
||
One drawback of mutation testing is that it runs the whole test suite once per | ||
generated mutant, so it can be slow on large trees with slow test suites. There | ||
are [some techniques to speed up cargo-mutants](performance.md), including | ||
[running multiple tests in parallel](parallel.md). |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,13 @@ | ||
# How is mutation testing different to fuzzing? | ||
|
||
Fuzzing is a technique for finding bugs by feeding pseudo-random inputs to a | ||
program, and is particularly useful on programs that parse complex or untrusted | ||
inputs such as binary file formats or network protocols. | ||
|
||
Mutation testing makes algorithmically-generated changes to a copy of the | ||
program source, and measures whether the test suite catches the change. | ||
|
||
The two techniques are complementary. Although some bugs might be found by | ||
either technique, fuzzing will tend to find bugs that are triggered by complex | ||
or unusual inputs, whereas mutation testing will tend to point out logic that | ||
might be correct but that's not tested. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters