Skip to content

Commit

Permalink
migrate nose tests to pytest (#998)
Browse files Browse the repository at this point in the history
* CCD-1267 Got pytest working and added test for reformat_date_or_auto

* CCD-1267 Added test for sreprlow

* CCD-1267 Continuing additions of tests for test_bestrefs.py

* CCD-1267 Added PR number to changes

* Added test for class init.

* use `is`

* Added test for class init types and complex init.

* Added more particular tests for complex and auto.

* Added logging tests for warnings/errors and changed order of tests. Corrected some typos.

* Modified logging tests to actually check contents of logs by checking stdout.

* Fixed some spacing issues.

* Added test for test_handle_na_and_not_found

* Added test for test_handle_na_and_not_found

* Screen_bestrefs and added fixture for log clearing.

* Got test working for get_bestrefs.

* Got test working for unkilled_updates.

* Got test working for dataset_to_product_id.

* Got test working for print_affected.

* only run actions/tests on pushes and PRs to master

* Completed pytests and test conversions tests for tests/test_bestrefs.py

* Completed pytests and test conversions tests for tests/test_bestrefs.py

* pytest/heavy-client (#946)

* install pytest deps, comment out nose tests

* migrated heavy_client tests to pytest, added fixtures and cfg to conftest, removed crds/tests from pkg dirs

* remove commented out code relating to nose tests

* move all test data into new test directory, remove old tests from repo (#947)

* added substitutions pytests (#949)

* pytest/bestrefs-fixtures (#948)

* use caplog and fixtures for bestrefs tests, move hst test data into subfolder

* move additional hst test files

* context_to_context

* update rel paths for hst test data mappings

* move canary file

* mark remaining tests

* update rel path links to test data

* uncomment warn tests

* remove unused imports

* Update conftest.py

return broken state config (accidentally removed during merge conflict resolution)

* pytest/core-reftypes (#951)

* initial reftypes pytests

* finalize pytests for reftypes - all tests passing

* added pytests for crds.core.cmdline (#952)

* pytest/test-rmap (#953)

* added pytests for core.rmap module

* remove unused prettyprint import

* Completed pytests and test conversions tests for tests/test_test_list.py

* pytest/test_roman (#959)

* added roman pytests

* add blankline

* Completed pytests and test conversions tests for tests/test_test_list.py

* Completed pytests and test conversions tests for tests/test_test_list.py

* pytest/run-pytests-action (#957)

* github actions for pytest suite, fixes to get all tests passing

* remove duplicate smoke marker

* ensure py3.10 not 3.1

* leave setup_data

* skip setup_data step

* temp - only run ubuntu 3.9 to debug tests

* use test_cache for roman substitutions validation

* fix string in roman rmap test

* catch warnings, fix expected output formatting

* fix expected output

* remove missing outline

* enumerate over list to check tuple

* uncomment py3.9 and 3.11

* Completed pytests and test conversions tests for tests/test_locking.py

* Completed pytests and test conversions tests for tests/test_locking.py

* Attempting fixes for list.py CI checks.

* Attempting fixes for list.py CI checks.

* Attempting fixes for list.py CI checks.

* Attempting fixes for list.py CI checks.

* Attempting fixes for list.py CI checks.

* Test certify (#963)

* unit tests converted to pytest

* remaining certify tests migrated to pytest

* fix warning output

* cleanup state before assertions

* cleanup state before assertions - remaining tests

* Attempting fixes for list.py CI checks.

* Attempting fixes for list.py CI checks.

* Attempting fixes for list.py CI checks.

* Added marks for lock tests.

* Added marks for lock tests.

* CCD-1333 convert checksums tests from nose/doctest to pytest (#964)

Changes:
* initial test module for checksum
* test running as their original doctest version
* complete test_checksum_script_fits_add
* convert test_checksum_script_fits_remove
* convert test_checksum_script_fits_verify_good
* convert test test_checksum_script_fits_verify_bad
* convert test test_checksum_script_rmap_verify_good
* convert test test_checksum_script_rmap_add_bad
* convert test test_checksum_script_rmap_verify_bad
* convert test test_checksum_script_rmap_remove_bad
* convert test test_checksum_script_rmap_verify_missing
* convert test test_checksum_script_unsupported_asdf
* remove test cache state fixture usage
* convert test test_checksum_script_unsupported_json
* convert test test_checksum_script_unsupported_text
* fix source data location after rebase

* Set config functions to not return a value.

* Refactor setup_test_cache to allow for simply updating local cache (#966)

Changes:
* add ability to simply update test cache
* make the argument usage consistent to the docs
* update changelog

* pytest/check-archive (#968)

* added misc marker

* check archive tests added; remove leftover jwst marker from reftypes

* remove extraneous warning string checks, ensure config cleanup before test assertions

* uncommented py39,311 ci testing

* use new hst_shared_cache_state fixture with hst uri

* add observatory kwarg

* return the config...

* Converted bad_files to pytest.

* Converted bad_files to pytest.

* Converted bad_files to pytest.

* Fixed output for test_bad_references_bestrefs_script_error/warning

* pytest/io-tests (#969)

* added factory tests

* added data_file factory tests

* added io.tables tests

* finalized io tests, added markers

* CCD-1347 migrate rowdiff tests (#974)

Changes
* initial commit with the original rowdiff test module
* convert test withtablexts and nodiff
* convert test rowchange
* convert test_rowremoval
* convert test_rowaddition
* convert test_ignorefields
* convert test_ignorefields_specific
* convert test_switchfields_nodiff
* convert test_switchfields_withdiff
* convert test test_nodiff
* convert test_nodiff_diffrows
* convert test_removedmodes
* convert test_duplicatemodes

* refactoring.refactor tests (#972)

* refactoring.refactor tests

* remove duplicate log lines from string check

* check if string in out instead of exact equality check

* use caplog.messages instead of caplog.text

* ignore log pretext

* CCD-1354 Convert table-effects tests (#976)

Changes:
* cleanup of checksum imports and initial table effects commit
* add basic imports and pytest markers
* convert test_table_effects_default_always_reprocess
* convert test_table_effects_reprocess_test
* convert test_table_effects_reprocess_no
* remove some extraneous warning/info checking

* use GitHub Actions cache to cache test cache (#973)

* use GitHub Actions cache to cache test cache

* always cache reference files

* pytest/or-bars-tests (#977)

* added certify orbar tests

* added refactor orbars tests

* CCD-1355 Migrate test_uniqname to pytest-compatible functions (#979)

Changes:
* initial commit for test_uniqname with test test_synphot_uniqname
* convert test test_cdbs_uniqname
* convert test test_has_checksum
* use the hst-specific state

* CCD-1356 convert test_uses module to pytest (#982)

Changes
* initial commit of test module
* convert test_uses_findall_mappings_using_reference
* convert test_uses_rmaps
* convert test_uses_imap
* mark tests as slow
   each one takes ~50 seconds

* Got tests working for newcontext

* Added mark for refactoring module.

* pytest/bestrefs-test-class (#978)

* fix bestrefs class test config

* use Test* for class-based tests, update Bestrefs Test class

* remove comment

* remove smoke marker

* remove test generated files

* remove n infos string checks

* ensure cleanup

* explicitly set observatory in state fixtures

* use test cache, revise fixture implementation

* yield no_cache state fixture, cleanup before assertions

* temporarily only test on py311

* us test cache for mappath single

* remove unused imports

* yield default_test_cache_state, remove cleanup step

* revised state cfg

* yield then cleanup more states, explicilty clear function caches

* clear function caches

* yield all state configs, do cleanup inside fixture

* uncomment py310,311 ci actions

* set client retry to 3

* remove cleanup from new context tests

* use local mode for na pmap tests

* fix context paths, clear function caches

* temporary try except to diagnose server config in test failures

* insert params as arg not kwarg

* set mode to local for roman cache tests

* remove unused imports

* Converted tests for test_sync.py (#980)

* Converted tests for test_sync.py

* Converted tests for test_sync.py

* Added error count to test_sync_organize_instr and full list for test_purge_mappings.

* Got tests working for sync

* revised test class configuration to follow pytest standards, use tmpdir as cache

---------

Co-authored-by: alphasentaurii <[email protected]>

* pytest/synphot (#988)

* added first few synphot tests

* add synphot marker

* converted synphot and synphot-lookup-generator pytests

* certify refs tests must use state fixture where obs is set explicitly to hst

* make tests more resilient to different environments (#989)

Changes:
* remove extraneous whitespace
* make bad to see output in ci
* Revert "make bad to see output in ci"
   This reverts commit 009e8c2.
* remove bunch of stuff and see what occurs
* Revert "remove bunch of stuff and see what occurs"
   This reverts commit 42bcf75.
* fix verbosity so it is consistent locally and in CI
* update docstring
* fix all tests in test_bad_files
* fix test_certify_bad_checksum
* fix test_certify_good_checksum
* fix test_certify_table_comparison_context
* test_certify_table_comparison_reference
* fix test_certify_jwst_valid
* fix test_certify_roman_invalid_asdf_schema
* fix test_certify_roman_invalid_spec_asdf_schema
* fix test_certify_jwst_bad_fits
* fix test_acs_idctab_char_plus_column
* fix certify asdf tests
* removed extraneous logging info
* more truth cleanup
* better handle numpy differences between architectures

* convert test_build6 (#990)

* Convert `diff` to pytest (#971)

* update test_diff.py

* use JWST shared data cache fixture

* add data to test functions

* fix reference checks

* mark tests

* retrieve version of fitsdiff

* add files from master

* update checksum

* add separate workflow to download CRDS test cache (#987)

* add cache workflow

* run on pull request

* use Python 3.11 and trigger on PR label

* print contexts

* add concurrency block

* fix error

* combine install commands

* rename workflow

* only run test cache setup

* use cache in test workflow

* separate contexts into workflow

* remove concurrency lock from context workflow

* install dependencies in environment

* also install datamodels

* cache observatories separately

* use update to not clean directory

* rename steps

* install JWST from source and clean CRDS installation

* update if cache found, otherwise create from clean

* handle case where combined cache might not exist but individual caches still restore

* add Roman files to test cache

* add fitsverify to environment

* use Python 3.11 in cache environment

* fix cache key

* only setup test cache if cache not hit

* assume that cache has been set up

* only install environment if cache not found

* run caching job at midnight

* add default HST server URL

* remove Roman cache

* cache hit boolean

* cache test path as well

* various minor improvements to CI (#985)

* use pytest-xdist to run tests in parallel

* stop jobs when a new commit is added to the branch

* run workflow on demand

* remove parallel test run

* remove xml arg

* use micromamba to install `fitsverify` (#986)

* use micromamba to install fitsverify

* name environment

* test on macos

* install dependencies as part of environment

* actually add `fitsverify`

* use Python 3.11

* pytest/test-special (#991)

* added initial tests

* add special marker

* migrated and refactored special tests

* remove concurrency lock from caching workflow (#993)

* remove concurrency from caching job to resolve deadlock

* test on pytest-migration branch while staging

* fix issue where environment was not using requested python version (#995)

* pytest/test-selectors (#996)

* move tobs tests into test_rmap

* converted tobs tests to pytest, added selectors marker

* delete old tobs tests replaced in test_rmap

* pytest/test-submit (#992)

* converted test_submit to pytest

* disable mock urlopen

* use serverless state

* disable the urlopen and mockup form

* use hst_shared_cache_state instead of serverless

* initialize submission obj inside fixture with patched mock urlopen

* trying alternate way to mock urlopen

* use default server

* change workflow triggers for master (#994)

* update changelog, add pars-chargemigrationstep to jwst filekinds test

---------

Co-authored-by: Hunter Brown <[email protected]>
Co-authored-by: Ru Keïn <[email protected]>
Co-authored-by: alphasentaurii <[email protected]>
Co-authored-by: Hunter Brown <[email protected]>
Co-authored-by: Jonathan Eisenhamer <[email protected]>
  • Loading branch information
6 people committed Oct 19, 2023
1 parent e47e970 commit a006dbd
Show file tree
Hide file tree
Showing 399 changed files with 12,282 additions and 146,428 deletions.
92 changes: 92 additions & 0 deletions .github/workflows/cache.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,92 @@
name: test cache

on:
workflow_call:
outputs:
path:
value: ${{ jobs.cache.outputs.path }}
testing_cache:
value: ${{ jobs.cache.outputs.testing_cache }}
key:
value: ${{ jobs.cache.outputs.key }}
workflow_dispatch:
schedule:
# Weekly Monday midnight
- cron: "0 0 * * 1"

env:
CRDS_TEST_ROOT: /tmp
CRDS_PATH: /tmp/crds-cache-default-test
CRDS_TESTING_CACHE: /tmp/crds-cache-test
CRDS_CLIENT_RETRY_COUNT: 3
CRDS_CLIENT_RETRY_DELAY_SECONDS: 20
LD_LIBRARY_PATH: /usr/local/lib

defaults:
run:
shell: bash -leo pipefail {0} {0}

jobs:
contexts:
uses: ./.github/workflows/contexts.yml
cache:
needs: [ contexts ]
name: download and cache CRDS test files
runs-on: ubuntu-latest
outputs:
path: ${{ env.CRDS_PATH }}
testing_cache: ${{ env.CRDS_TESTING_CACHE }}
key: ${{ steps.key.outputs.key }}
steps:
- uses: actions/checkout@v3
- id: key
run: echo "key=test-cache-${{ needs.contexts.outputs.hst }}-${{ needs.contexts.outputs.jwst }}" >> $GITHUB_OUTPUT
- id: cache
name: restore entire CRDS cache
uses: actions/cache@v3
with:
path: |
${{ env.CRDS_PATH }}
${{ env.CRDS_TESTING_CACHE }}
key: ${{ steps.key.outputs.key }}
- if: steps.cache.outputs.cache-hit != 'true'
id: crds-cache-test
run: |
git clone https://github.com/spacetelescope/crds-cache-test.git
mv crds-cache-test ${{ env.CRDS_TEST_ROOT }}
- if: steps.cache.outputs.cache-hit != 'true'
id: cache_hst
name: restore cached HST files
uses: actions/cache@v3
with:
path: ${{ env.CRDS_PATH }}/**/hst*
key: test-cache-${{ needs.contexts.outputs.hst }}
- if: steps.cache.outputs.cache-hit != 'true'
id: cache_jwst
name: restore cached JWST files
uses: actions/cache@v3
with:
path: ${{ env.CRDS_PATH }}/**/jwst*
key: test-cache-${{ needs.contexts.outputs.jwst }}
- if: steps.cache.outputs.cache-hit != 'true'
uses: mamba-org/setup-micromamba@v1
with:
environment-name: crds-testing
create-args: >-
python=3.11
asdf
astropy
filelock
fitsverify
numpy
parsley
requests
init-shell: bash
cache-environment: true
cache-downloads: true
- if: steps.cache.outputs.cache-hit != 'true'
run: pip install git+https://github.com/spacetelescope/jwst roman-datamodels
- if: steps.cache.outputs.cache-hit != 'true'
run: pip uninstall --yes crds && ./install && pip install .
- if: steps.cache.outputs.cache-hit != 'true'
run: ./setup_test_cache ${{ env.CRDS_TEST_ROOT }} ${{ (steps.cache_hst.outputs.cache-hit == 'true' || steps.cache_jwst.outputs.cache-hit == 'true') && 'u' || 'c' }}
76 changes: 47 additions & 29 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
@@ -1,23 +1,39 @@
name: CI

on:
## uncomment before merging to master once test suite is finalized
push:
branches:
- master
- master
tags:
- '*'
pull_request:
branches:
- master
workflow_dispatch:

concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true

env:
CRDS_TEST_ROOT: /tmp
CRDS_SERVER_URL: https://hst-crds.stsci.edu
CRDS_CLIENT_RETRY_COUNT: 3
CRDS_CLIENT_RETRY_DELAY_SECONDS: 20
LD_LIBRARY_PATH: /usr/local/lib

defaults:
run:
shell: bash -leo pipefail {0} {0}

jobs:
cache:
uses: ./.github/workflows/cache.yml
pytest:
name: ${{ matrix.runs-on }} Python ${{ matrix.python-version }}
needs: [ cache ]
runs-on: ${{ matrix.runs-on }}
env:
CRDS_SERVER_URL: https://hst-crds.stsci.edu
CRDS_TEST_ROOT: /tmp
LD_LIBRARY_PATH: /usr/local/lib
strategy:
fail-fast: false
matrix:
Expand All @@ -26,32 +42,34 @@ jobs:
- macos-latest
python-version:
- 3.9
- '3.10'
- 3.11
steps:
- uses: actions/checkout@v3
with:
fetch-depth: 0
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v4
- uses: mamba-org/setup-micromamba@v1
with:
environment-name: crds-testing
create-args: >-
python=${{ matrix.python-version }}
asdf
astropy
filelock
fitsverify
numpy
parsley
requests
init-shell: bash
cache-environment: true
cache-downloads: true
- run: pip install roman-datamodels git+https://github.com/spacetelescope/jwst
- run: pip uninstall --yes crds && ./install && pip install .[submission,test,docs,synphot]
- id: cache
uses: actions/cache@v3
with:
python-version: ${{ matrix.python-version }}
- name: Install fitsverify
run: |
wget --quiet --output-document - http://heasarc.gsfc.nasa.gov/FTP/software/fitsio/c/cfitsio-3.49.tar.gz | tar -xz -C $HOME
cd $HOME/cfitsio-3.49
./configure --prefix=/usr/local --enable-reentrant --disable-curl
make shared
sudo make install
wget --quiet --output-document - https://heasarc.gsfc.nasa.gov/docs/software/ftools/fitsverify/fitsverify-4.20.tar.gz | tar -xz -C $HOME
cd $HOME/fitsverify-4.20
gcc -o fitsverify ftverify.c fvrf_data.c fvrf_file.c fvrf_head.c fvrf_key.c fvrf_misc.c -DSTANDALONE -lcfitsio
sudo cp fitsverify /usr/local/bin
- name: Install Python dependencies
run: |
pip install roman-datamodels
pip install git+https://github.com/spacetelescope/jwst
pip uninstall --yes crds
./install
pip install .[submission,test,docs,synphot]
./setup_test_cache $CRDS_TEST_ROOT
- name: Run tests
run: ./runtests --cover
path: |
${{ needs.cache.outputs.path }}
${{ needs.cache.outputs.testing_cache }}
key: ${{ needs.cache.outputs.key }}
- run: ./runtests --cover
52 changes: 52 additions & 0 deletions .github/workflows/contexts.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
name: contexts

on:
workflow_call:
outputs:
hst:
value: ${{ jobs.contexts.outputs.hst }}
jwst:
value: ${{ jobs.contexts.outputs.jwst }}
roman:
value: ${{ jobs.contexts.outputs.roman }}
workflow_dispatch:

jobs:
contexts:
name: retrieve latest CRDS contexts
runs-on: ubuntu-latest
outputs:
hst: ${{ steps.hst_crds_context.outputs.pmap }}
jwst: ${{ steps.jwst_crds_context.outputs.pmap }}
roman: ${{ steps.roman_crds_context.outputs.pmap }}
steps:
- id: hst_crds_context
env:
OBSERVATORY: hst
CRDS_SERVER_URL: https://hst-crds.stsci.edu
run: >
echo "pmap=$(
curl -s -X POST -d '{"jsonrpc": "1.0", "method": "get_default_context", "params": ["${{ env.OBSERVATORY }}"], "id": 1}' ${{ env.CRDS_SERVER_URL }}/json/ |
python -c "import sys, json; print(json.load(sys.stdin)['result'])"
)" >> $GITHUB_OUTPUT
- run: echo ${{ steps.hst_crds_context.outputs.pmap }}
- id: jwst_crds_context
env:
OBSERVATORY: jwst
CRDS_SERVER_URL: https://jwst-crds.stsci.edu
run: >
echo "pmap=$(
curl -s -X POST -d '{"jsonrpc": "1.0", "method": "get_default_context", "params": ["${{ env.OBSERVATORY }}"], "id": 1}' ${{ env.CRDS_SERVER_URL }}/json/ |
python -c "import sys, json; print(json.load(sys.stdin)['result'])"
)" >> $GITHUB_OUTPUT
- run: echo ${{ steps.jwst_crds_context.outputs.pmap }}
- id: roman_crds_context
env:
OBSERVATORY: roman
CRDS_SERVER_URL: https://roman-crds.stsci.edu
run: >
echo "pmap=$(
curl -s -X POST -d '{"jsonrpc": "1.0", "method": "get_default_context", "params": ["${{ env.OBSERVATORY }}"], "id": 1}' ${{ env.CRDS_SERVER_URL }}/json/ |
python -c "import sys, json; print(json.load(sys.stdin)['result'])"
)" >> $GITHUB_OUTPUT
- run: echo ${{ steps.roman_crds_context.outputs.pmap }}
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
coverage
.coverage
coverage.*
.profile-stats
.pydevproject
.project
*.pyc
Expand All @@ -16,3 +18,4 @@ documentation/crds_users_guide/*.tar.gz
__pycache__/
crds/_version.py
.vscode/
.DS_Store
9 changes: 9 additions & 0 deletions CHANGES.rst
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,11 @@ General

- Replaced deprecated np.product with np.prod in crds.certify.validators.core [#975]

Testing
-------

- Migrated test suite from nose to pytest, running CI tests for python 3.9, 3.10, 3.11 [#998]


11.17.6 (2023-09-08)
=====================
Expand Down Expand Up @@ -57,9 +62,13 @@ General
-------

- Removed python 3.8 check from ci.yml [#934]

- Removed references to ICD-47 in users guide [#936]

- translate 'ANY' as equal to '*' when selecting match rules in rmap changes. Prevents equal weight special case errors from occurring unnecessarily [#939]

- Refactor setup_test_cache to allow for simply updating local cache [#966]

JWST
----

Expand Down
Loading

0 comments on commit a006dbd

Please sign in to comment.