Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

unify fault reporting across unit and nightly tests for Omex execution #1402

Merged
merged 7 commits into from
Dec 16, 2024

Conversation

jcschaff
Copy link
Member

@jcschaff jcschaff commented Dec 16, 2024

  • OmexTestCase objects hold test cases with expected results - stored in a newline-delimited JSON file (test_cases.ndjson)
  • The BSTSBasedOmexExecTest use the text cases defined in test_cases.ndjson as source of truth for regression testing of OMEX executions.
  • A new CLI command (execute-omex (copied/simplified Nightly tests to use test_cases.ndjson as source of truth for regression testing against all BIOMD OMEX files.
  • OmexTestingDatabase unifies runtime error identification between both execute-omex CLI command and the BSTS based OMEX unit tests.
    • has 'determineFault()' and 'summarize()' methods and a placeholder 'generateReport()' method which will generate a nice error report in the future - via a new omex-report cli command.
  • introduces new 'execute-omex' cli command (derived from 'execute' command, for single omex files with new reporting)
    • each run generates an 'exec_summary.json' file with a single OmexExecSummary object - to be appended to a master 'exec_summary.ndjson' file (one report per line). These OmexExecSummary objects are to be compared with the list of OmexTestCase objects from test_cases.ndjson for final reporting.
    • it also generates another json file per run, a 'tracer.json' file which holds all tracer events ... we can output this in appropriate formats, or pretty print it for human use. the tracer.json file has all of the information from the logs generated by CLIRecordable... but in a structured format (EventLog objects with context) which is machine understandable.

what's next

  1. integrate regression data (from test_cases.ndjson) and executions (from exec_summary.ndjson) into an overall report.
  2. nightly test should "fail" if execution status does not correspond to expected status ... if this happens, we have to update test_cases.ndjson).
  3. note that test_cases.ndjson always holds the single source of truth for the software version.

@jcschaff jcschaff self-assigned this Dec 16, 2024
Copy link
Contributor

@CodeByDrescher CodeByDrescher left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Really like this! SUPER intuitive, and a very useful feature-set!

@jcschaff
Copy link
Member Author

@CodeByDrescher thanks for the review.

@jcschaff jcschaff merged commit 4b1ed10 into master Dec 16, 2024
13 checks passed
@jcschaff jcschaff deleted the omex-error-patterns branch December 16, 2024 23:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: Done
Development

Successfully merging this pull request may close these issues.

2 participants