Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tests: Expand docs. #154

Open
wants to merge 4 commits into
base: main
Choose a base branch
from
Open

Tests: Expand docs. #154

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
bef53c3
Expand Docs.
kkeroo Dec 20, 2024
6243283
Typo fix.
kkeroo Dec 20, 2024
97c273d
Typo.
kkeroo Dec 20, 2024
7cbe858
Integration tests docs expansion.
kkeroo Dec 20, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/e2e_tests.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ on:
required: true
default: 'main'
testbed:
description: 'Testbed to run the tests on. Default: oak4-pro'
description: 'Testbed to run the tests on. Available: oak4-pro, oak4-s'
required: true
default: 'oak4-pro'
depthai-version:
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/integration_tests.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ on:
required: true
default: 'main'
testbed:
description: 'Testbed to run the tests on. Default: oak4-s'
description: 'Testbed to run the tests on. Available: oak4-pro, oak4-s'
required: true
default: 'oak4-s'
depthai-version:
Expand Down
9 changes: 9 additions & 0 deletions tests/end_to_end/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,15 @@ The tests check if the device is able to run a model with neural network node an

You can manually trigger the Github action to run the HIL tests. There is a workflow parameter `additional-parameter` that can be used to specify the desired test. Default is `-all`. The available options are: `-all`, `-p <parser_name>`, `-s <slug_1> <slug_2> ...`, `-nn <path_to_archive_1> <path_to_archive_2> ...`. You can also specify with branch you want to run the tests on. The default is `main`. NOTE: There is a custom parameter `branch` and not the built-in one.

### Required parameters

There are 4 required parameters that need to be set when triggering the Github action:

- `additional-parameter`: The parameter that specifies the desired test. Default is `-all` which runs tests on all public models. The available options are: `-all`, `-p <parser_name>`, `-s <slug_1> <slug_2> ...`.
Copy link
Collaborator

@jkbmrz jkbmrz Dec 20, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

slug should be renamed to model (or HubAI model identifier). I'd also fix the argument call from -s to -m (model)

- `branch`: The branch on which the tests will be run. Default is `main`.
- `testbed`: The testbed on which the tests will be run. Default is `oak4-pro`. Available: `oak4-pro`, `oak4-s`.
- `depthai-version`: The version of the DepthAI that will be used for the tests. Default is `3.0.0a6`.

## Running the tests locally

Currently, you must specify the device IP address in the ENV variables: `RVC2_IP` and `RVC4_IP`. If the ENV variable is empty the script will take the connected device via USB. For sending requests to HubAI you also need to specify `HUBAI_TEAM_ID` and `HUBAI_API_KEY` ENV variables.
Expand Down
11 changes: 10 additions & 1 deletion tests/integration_tests/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,16 @@ with open('nn_datas/ClassificationParser/efficientnet-lite_output.pkl', 'wb') as

In the end, you should have all the files in the parser-specific directory inside `nn_datas` directory. You need to upload the parser directory to the B2 bucket.

## Running the tests
## Running the tests locally

To run the tests, you can use the `main.py` script. You can use `--all` flag to test all parsers or test a specific parser with `-p` flag.
You would need the B2 credentials to download the tests from the bucket and set it in the ENV variables `B2_APPLICATION_KEY_ID` and `B2_APPLICATION_KEY`.

## Running the tests in the CI

The integration tests are triggered in every PR. But you can also trigger them manually. Required parameters are:

- `additional-parameter`: The parameter that specifies the desired test. Default is `-all` which runs tests on all parsers. The available options are: `-all`, `-p <parser_name>`.
- `branch`: The branch on which the tests will be run. Default is `main`.
- `testbed`: The testbed on which the tests will be run. Default is `oak4-s`. Available: `oak4-pro`, `oak4-s`.
- `depthai-version`: The version of the DepthAI that will be used for the tests. Default is `3.0.0a6`.
Loading