Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Categorise test cases into test suite(s) #247

Closed
wants to merge 3 commits into from
Closed

Categorise test cases into test suite(s) #247

wants to merge 3 commits into from

Conversation

cskeogh
Copy link
Contributor

@cskeogh cskeogh commented Jun 30, 2024

Meson test cases may belong to test suite(s). If so, in VSCode's "Testing" feature from the Activity Bar, add the test suite(s) as a parent item and add the test cases as children.

The user has the choice to run all tests belonging to a test suite. Will pass the test suite to the meson test command --suite. The option --suite is needed to run test cases that the meson config purposely excludes (for example, test suite "flaky").

Fixes #245

Quote the "test case" in the meson command, so can run test cases with a space in the name. The quotes are needed because cp.ExecFileOptions sets shell=true.

Fixes #256

@cskeogh
Categorise test cases into test suite(s)
bf2e8f9 
Meson test cases may belong to test suite(s). If so, in VSCode's
"Testing" feature from the Activity Bar, add the test suite(s) as a
parent item and add the test cases as children.

The user has the choice to run all tests belonging to a test suite.
Will pass the test suite to the meson test command --suite. The option
--suite is needed to run test cases that the meson config purposely
excludes (for example, test suite "flaky").

Fixes mesonbuild#245

Quote the "test case" in the meson command, so can run test cases with a
space in the name. The quotes are needed because `cp.ExecFileOptions`
sets `shell=true`.

Fixes mesonbuild#256
@cskeogh
Copy link
Contributor Author

cskeogh commented Jun 30, 2024

Using the gtk project, showing the test suite(s):

Screenshot 2024-06-30 124213

@tristan957
Copy link
Contributor

Thanks for the PR. I will try to test and review it this week.

tristan957
tristan957 reviewed Jul 1, 2024
View reviewed changes
src/tests.ts Outdated Show resolved Hide resolved
src/tests.ts Outdated Show resolved Hide resolved
@tristan957
Copy link
Contributor

What do you think of #237? Should I merge that one first?

@cskeogh
Copy link
Contributor Author

cskeogh commented Jul 2, 2024

Updated the PR to fix the review comments.

Note the PR does have some limitations:

  1. Gtk's test case "about state" is in 2 test suites "gtk+:a11y" and "gtk+:flaky". This PR shows "about state" correctly in both test suites, but they are separate instances of TestItem. Ideally, "about state" would be one instance of TestItem, so that the failure icon shows on both (see screenshot yellow highlight). But VSCode doesn't support one TestItem with 2 parents. I tried.
    Screenshot 2024-07-02 215050
    Screenshot 2024-07-02 215210

  2. The user can run the test suite, which will correctly run all test cases belonging to the test suite. But the children TestItem do not show the success / failure icons (see screenshot yellow highlight). Only the test suite shows success / failure. In the below example, I ran test suite "gtk+:gdk" and 7 test cases passed and 1 failed:
    Screenshot 2024-07-02 220438

Ok:                 7   
Expected Fail:      0   
Fail:               1   
Unexpected Pass:    0   
Skipped:            0   
Timeout:            0

@tristan957
Copy link
Contributor

Thanks for explaining the limitations. I see that there is this function. Do we have access to the test runs, so if someone runs a suite, we can properly annotate the test? https://vscode-api.js.org/interfaces/vscode.TestRun.html#passed

@cskeogh
Copy link
Contributor Author

cskeogh commented Aug 9, 2024

I'm working on resolving the limitations. Closing this PR until I resolve.

@cskeogh cskeogh closed this Aug 9, 2024
@tristan957
Copy link
Contributor

Alright. Really appreciate you looking into this more!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@tristan957 tristan957 tristan957 left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Categorise test cases into test suite(s)
2 participants
@cskeogh @tristan957