added slow test label and execute full testing suite before merge (#634)

### Description
Adding INCLUDE_SLOW_TESTS label which runs tests of slower execution as
well as enforcing full scope testing in CI pipelines for merge_queue
event

Updated documentation

### Type of changes
<!-- Mark the relevant option with an [x] -->

- [ ]  Bug fix (non-breaking change which fixes an issue)
- [x]  New feature (non-breaking change which adds functionality)
- [ ]  Refactor
- [ ]  Documentation update
- [ ]  Other (please describe):

### CI Pipeline Configuration
Configure CI behavior by applying the relevant labels:

-
[SKIP_CI](https://github.com/NVIDIA/bionemo-framework/blob/main/docs/docs/user-guide/contributing/contributing.md#skip_ci)
- Skip all continuous integration tests
-
[INCLUDE_NOTEBOOKS_TESTS](https://github.com/NVIDIA/bionemo-framework/blob/main/docs/docs/user-guide/contributing/contributing.md#include_notebooks_tests)
- Execute notebook validation tests in pytest

> [!NOTE]
> By default, the notebooks validation tests are skipped unless
explicitly enabled.

### Usage
<!--- How does a user interact with the changed code -->
```python
TODO: Add code snippet
```

### Pre-submit Checklist
<!--- Ensure all items are completed before submitting -->

 - [ ] I have tested these changes locally
 - [ ] I have updated the documentation accordingly
 - [ ] I have added/updated tests as needed
 - [ ] All existing tests pass successfully

---------

Signed-off-by: Peter St. John <[email protected]>
Signed-off-by: Dorota Toczydlowska <[email protected]>
Co-authored-by: Peter St. John <[email protected]>
Signed-off-by: Polina Binder <[email protected]>

.github/pull_request_template.md

@@ -15,6 +15,8 @@ Configure CI behavior by applying the relevant labels:
 
 - [SKIP_CI](https://github.com/NVIDIA/bionemo-framework/blob/main/docs/docs/user-guide/contributing/contributing.md#skip_ci) - Skip all continuous integration tests
 - [INCLUDE_NOTEBOOKS_TESTS](https://github.com/NVIDIA/bionemo-framework/blob/main/docs/docs/user-guide/contributing/contributing.md#include_notebooks_tests) - Execute notebook validation tests in pytest
+- [INCLUDE_SLOW_TESTS](https://github.com/NVIDIA/bionemo-framework/blob/main/docs/docs/user-guide/contributing/contributing.md#include_slow_tests) - Execute tests labelled as slow in pytest for extensive testing
+
 
 > [!NOTE]
 > By default, the notebooks validation tests are skipped unless explicitly enabled.

.github/workflows/unit-tests.yml

@@ -114,12 +114,25 @@ jobs:
           path: ${{ github.run_id }}
 
       - name: Run tests
+        # Tests in this stage generate code coverage metrics for the repository
+        # Coverage data is uploaded to Codecov in subsequent stages
         env:
           BIONEMO_DATA_SOURCE: ngc
         run: ./ci/scripts/run_pytest.sh --no-nbval --skip-slow
 
+      - name: Run slow tests
+        if: |
+          github.event_name == 'merge_group' ||
+          (github.event_name == 'pull_request' && contains(github.event.pull_request.labels.*.name, 'INCLUDE_SLOW_TESTS'))
+        env:
+          BIONEMO_DATA_SOURCE: ngc
+        run: pytest -m "slow" sub-packages/
+
       - name: Run notebook tests
-        if: ${{ contains(github.event.pull_request.labels.*.name, 'INCLUDE_NOTEBOOKS_TESTS') }}
+        if: |
+          github.event_name == 'merge_group' ||
+          (github.event_name == 'pull_request' &&
+            contains(github.event.pull_request.labels.*.name, 'INCLUDE_NOTEBOOKS_TESTS'))
         env:
           BIONEMO_DATA_SOURCE: ngc
         run: pytest --nbval-lax -p no:python docs/ sub-packages/

docs/docs/user-guide/contributing/contributing.md

@@ -126,6 +126,12 @@ Key behaviors:
 - Use when modifying notebooks or notebook-related code
 - Disabled by default
 
+#### **INCLUDE_SLOW_TESTS**
+
+- Enables unit tests labelled as slow ie CLI tests
+- Use when modifying core functionalities and require extensive, end-2-end, testing
+- Disabled by default
+
 ### Developer workflows
 
 You should always carefully test your changes. Run `pytest ...` in your container locally. All tests are done via `pytest`.

sub-packages/bionemo-esm2/tests/bionemo/esm2/model/test_convert.py

@@ -51,4 +51,5 @@ def test_nemo2_conversion_equivalent_650m(tmp_path):
     model_tag = "facebook/esm2_t33_650M_UR50D"
     module = biobert_lightning_module(config=ESM2Config())
     io.import_ckpt(module, f"hf://{model_tag}", tmp_path / "nemo_checkpoint")
-    assert_model_equivalence(tmp_path / "nemo_checkpoint", model_tag)
+    with megatron_parallel_state_utils.distributed_model_parallel_state():
+        assert_model_equivalence(tmp_path / "nemo_checkpoint", model_tag, atol=1e-4, rtol=1e-4)