Propose concrete ideas of the merge criteria for Core/Universe

[kenji-miyake]

As written in #3, we'll change the merge criteria for Core/Universe for more efficiency.

In this issue, we'll propose concrete ideas like:

• Required amount and kind of tests
• Required contents and quality of documentation
• Required code metrics

for each of Core and Universe.

Although Autoware.Auto already has the rules, it should be refined for our new process.
Also, Tier IV has previously proposed some criteria in ASWG, but it's yet high-level.

[KeisukeShima]

TL;DR: In my opinion, it is a good idea to measure code coverage for each type of test. This enables a detailed analysis of the tests.

By defining test labels, it is possible to output test coverage for each type of test.
To illustrate this benefit, I forked Autoware.Auto and added test labels.

Here is my repository. https://gitlab.com/keisuke.shima/AutowareAuto/-/commit/c812f150906b63a176c685cf243cb9f8bc5f6876

By specifying the labels and running the tests, we can measure the code coverage for each of the unit tests and the integration tests (including smoke tests and component tests).
This is useful for reviewing pull requests. Knowing which tests are covering the changes helps reviewers to consider whether the changes are properly tested.
It is also possible to define merge criteria for each type of test. For example, the criteria can be set to 80% or more for the unit test, 75% for the integration test.

As an example, here is the coverage of costmap_generator_nodes.

Since there are only launch_test in this code, the gtest coverage and smoke_test coverage are zero. We do not know this from the coverage report calculated from all the tests.

I would like to use this feature for the development of Autoware Core.
ros-metrics-reporter developed by TIER IV is a tool for continuous integration to measure software metrics, including code coverage reports by labels.
Source code and example results can be found below.

ros-metrics-reporter repository
Example for Autoware.Auto report

[KeisukeShima]

Related my previous comment: #2 (comment)

In my previous comment, I showed that detailed code coverage can be measured by testing by label.
(As a related work, I've added a test-by-label workflow here.)

Now that we are ready, I would like to discuss the specific test labels.
As shown here, Autoware.Auto had a testing framework in place: smoke test and interface test.
The benefits of these are explained in the link.

First, I would like to apply the above two tests that were accepted in Autoware.Auto to Core/Universe.
These tests can be added to each Node with only a few changes, and if there is a problem, the CI system will be able to detect the error. And the cost of fixing them is lower than if they are detected by later tests such as scenario tests or real vehicle tests.　(This approach is called Shift-left testing.)
Another advantage is that you do not have to write gtest for the lines that these tests target.

As a result, the following test labels are created in Core/Universe

• gtest
• smoke_test
• interface_test

If you have ideas about adding or removing test labels, please comment. I'm open to ideas here.

[stale]

This pull request has been automatically marked as stale because it has not had recent activity.

[kenji-miyake]

I'll write some documents about this, but it's a low priority.

[stale]

This pull request has been automatically marked as stale because it has not had recent activity.