Add the examples and the tests in the installed packages. #514
Conversation
|
/te-ci |
nouiz
force-pushed
the
install_example_tests
branch
from
ee90a44 to
0952d76
Compare
|
Note that one issue with this proposed approach is that the Maybe, they should be placed inside the transformer_engine folder? |
denera
force-pushed
the
install_example_tests
branch
from
0952d76 to
e9a584d
Compare
…e don't need to keep the source directory to run all the tests and examples. This is small. Signed-off-by: Frederic Bastien <[email protected]>
…test dir, modified setup.py to install examples and test files Signed-off-by: Alp Dener <[email protected]>
denera
force-pushed
the
install_example_tests
branch
from
b26c483 to
f2e5fa9
Compare
|
@nouiz The folder hierarchy should be correct now with the latest commit, but this involves turning examples into an importable submodule. I also migrated the JAX example-based unit tests out of the examples folder into the tests folder in order to avoid contaminating the new examples submodules with unit test classes. The QA scripts reflect this change. Unit tests are copied over as "package data" so they're not importable. I didn't think it made much sense for them to be submodules. |
Signed-off-by: Alp Dener <[email protected]>
Signed-off-by: Alp Dener <[email protected]>
denera
force-pushed
the
install_example_tests
branch
from
f69336f to
a42b53f
Compare
Signed-off-by: Alp Dener <[email protected]>
|
/te-ci jax |
There was a problem hiding this comment.
Looks good overall.
@mingxu1067 can you also check?
@yhtang @DwarKapex, can you confirm this will fix the original issue?
examples/jax/mnist/__init__.py
Outdated
| # | ||
| # See LICENSE for license information. | ||
|
|
||
| from . import single_gpu |
There was a problem hiding this comment.
For filess that are renamed, grep in TE that they don't appear in the documentation. Maybe you need to update the names there too.
qa/L0_jax_unittest/test.sh
Outdated
| pytest -Wignore -v $TE_PATH/examples/jax/mnist | ||
| pytest -Wignore -v $TE_PATH/tests/jax \ | ||
| --ignore="$TE_PATH/tests/jax/test_encoder.py" \ | ||
| --ignore-glob="$TE_PATH/tests/jax/test_distributed_*.py" |
There was a problem hiding this comment.
If you use a * here, use a * in the other test.sh file. Otherwise, there could be mismatch and some tests won't be run.
| @@ -608,10 +641,11 @@ def main(): | |||
| ext_modules.append(setup_paddle_extension()) | |||
|
|
|||
| # Configure package | |||
| setuptools.setup( | |||
| s = setuptools.setup( | |||
There was a problem hiding this comment.
This is unintentionally leftover from debugging. I will remove it shortly alongside other minor fixes.
examples/jax/__init__.py
Outdated
| # See LICENSE for license information. | ||
|
|
||
| from . import mnist | ||
| from . import encoder |
There was a problem hiding this comment.
nitpik. Many files don't have a new line at the end.
tests/jax/test_mnist.py
Outdated
| @@ -0,0 +1,41 @@ | |||
| # Copyright (c) 2022-2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved. | |||
There was a problem hiding this comment.
Should you add "example" in the new tests filename?
Signed-off-by: Alp Dener <[email protected]>
Signed-off-by: Alp Dener <[email protected]>
|
/te-ci jax |
|
This PR is stale as it was created before the build system overhaul and as a result has several conflicts. After a discussion with @nouiz, we will re-open a fresh PR. |
This way, we don't need to keep the source directory to run all the tests and examples. This is small.