Coverage reporting missing for binaries under-test

[epage]

@mssun

Another issue about testing CLIs is to correctly calculate code coverage. If using std::process::Command to fork and execute a CLI, all existing code coverage tools (tarpaulin, gcov, kcov, etc) will failed to calculate the statistics. There is not a better way to properly calculate the line coverage including unit tests and integration tests.

From https://github.com/rust-lang-nursery/cli-wg/issues/9#issuecomment-385487756

[epage]

@mssun That is a good point but I'm expecting that won't be too much of an issue. End-to-end testing using assert_cli should, ideally, be reserved for the parts of the application that cannot be tested at a lower level. The coverage will be lower and a developer will need to be aware of what is or isn't being tested because of these end-to-end tests.

If there is an enterprising person who has an idea on how to solve that, great! Otherwise, we should call this out as an known issue in assert_clis documentation.

[epage]

@mssun

Ideally, we can use gcov to solve this issue like this https://jbp.io/2017/07/19/measuring-test-coverage-of-rust-programs.html. However, it is not as easy as I thought.

It is impossible to build both binary and tests with -Copt-level=1 -Clink-dead-code -Ccodegen-units=1 -Zno-landing-pads -Cpasses=insert-gcov-profiling -L/Library/Developer/CommandLineTools/usr/lib/clang/9.1.0/lib/darwin/ -lclang_rt.profile_osx flags only.

Because this command cargo rustc --test tests -vv -- -Copt-level=1 -Clink-dead-code -Ccodegen-units=1 -Zno-landing-pads -Cpasses=insert-gcov-profiling -L/Library/Developer/CommandLineTools/usr/lib/clang/9.1.0/lib/darwin/ -lclang_rt.profile_osx will only build test with the flags.

And using the RUSTFLAGS environment variable won't solve this problem. It will add compilation flags to all crates. This will generate gcna and gcno files for other non-related crates in sysroot. This may also introduce some bugs of gcov when merging gcno (still don't know why).

Overall, I think the testing coverage issue is related to many parties such as: cargo, rustc, coverage tools, etc.

[mssun]

Cool, thanks.

[epage]

The responsibility for this is moving to assert_cmd and tis will be addressed in assert-rs/assert_cmd#9

assert_cli is going to morph into a wider-focused tool, including things like assert_fs. See #41