Mini sbibm

[manuelgloeckler]

What does this implement/fix? Explain your changes

This is a draft for some "benchmarking" capabilities integrated into sbi.

With pytest, we can roughly check that everything works by passing all tests. Some tests will ensure that the overall methodology works "sufficiently" well on simplified Gaussian analytic examples. Certain changes might still pass all tests but, in the end, negatively impact the performance/accuracy.

Specifically, when implementing new methods or, e.g., changing default parameters, it is important to check that what was implemented not just only passes the tests but that it works sufficiently well.

Does this close any currently open issues?

Prototype for #1325

Any relevant code examples, logs, error output, etc?

So it now should work that one simply has to use:

pytest --bm 

Which is a custom tag that will disable testing and instead switch to a "benchmark" mode, which will only run tests that are marked as such and will always pass. Instead, these tests cache a metric on how well an implemented method solved a specific task (currently some examples in "bm_test.py").

Once it finishes, instead of passed/failed, it will return a table with the metric (we still can kinda color some methods that are worse than expected).

Any other comments?

• What tasks to incldue - clearly they must be somewhat "fast" to solve.
• What methods to incldue (i.e. just standard methods with default parameters or more)

[manuelgloeckler]

Alright, on the current examples, the output looks like this:

Runtime linearly increases with a number of train simulations (currently 2k ~ 10 min on my laptop, with 1k, it was like 5 min). It would maybe also be nice to print runtimes on the right.

Overall runtime, of course, also depends on how many different methods should be included. I think some limited control over what is run would be nice i.e

pytest --bm # All base inference classes on defaults (similar to current behavior)
pytest --bm=NPE   # NPE with e.g. different density estimators
pytest --bm=SNPE # SNPE_ABC  2 round test
...

Either way, there needs to be a limit on what is run, and every configuration should finish in a reasonable amount of time.

[manuelgloeckler]

Alright, it is kinda now ready for review. The overall "framework" is done, one from now go into a few directions, i.e., depending on what scope we want this to have. Current interface is

pytest --bm # Runs all major methods on default
pytest --bm --bm-mode=npe # Runs all major npe methods with e.g. different density estimators ...
pytest --bm --bm-mode=nre # Runs all major nre methods with different classfiers ...
pytest --bm --bm-mode=snpe # Runs all sequential NPE methods 2 rounds
pytest --bm --bm-mode=snle/snre # As above
pytest --bm --bm-mode=fmpe # Runs fmpe with different nets
pytest --bm --bm-mode=npse # Runs NPSE with different nets and others.

Not sure how much "configurability" we want this to have.

Tests cannot fail based on performance currently, but you will get a report which shows relative performance of each method on each task i.e. like this:

[janfb]

Overall, this is really great to have - thanks a lot for pushing this!
Love the relative coloring of the results 🎉

Added a couple of comments and questions for clarification.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 78.28%. Comparing base (43ae353) to head (a9978a5).
Report is 1 commits behind head on main.

❗ There is a different number of reports uploaded between BASE (43ae353) and HEAD (a9978a5). Click for more details.

HEAD has 3 uploads less than BASE

Flag	BASE (43ae353)	HEAD (a9978a5)
unittests	4	1

Additional details and impacted files

@@             Coverage Diff             @@
##             main    #1335       +/-   ##
===========================================
- Coverage   89.38%   78.28%   -11.11%     
===========================================
  Files         119      119               
  Lines        8905     8905               
===========================================
- Hits         7960     6971      -989     
- Misses        945     1934      +989     

Flag	Coverage Δ
unittests	78.28% <ø> (-11.11%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

see 35 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[janfb]

Thanks for the edits!

All looks very good. I just have a couple of suggestions for renaming and removing comments.

The samples files are small, so no need to save them via git-lfs?

[manuelgloeckler]

Thanks for the review.

In total, the sample files are 2.78 MB, so small but not super small (we could reduce this by saving only 1000 out of the 10_000 posterior samples (we anyway only use 1k for evaluation currently).

[janfb]

Thanks for the review.

In total, the sample files are 2.78 MB, so small but not super small (we could reduce this by saving only 1000 out of the 10_000 posterior samples (we anyway only use 1k for evaluation currently).

Yes, this will live in tests/ anyways, right? So it will not be distributed during packaging.
We can still reduce the samples to 1000 if we are evaluating it uses 1000 samples anyways.

[janfb]

This looks good now, thanks a ton! 🙏

What we would need is one place with a bit of documentation on how to use this. It would be for developers, so no need to make a tutorial. But maybe add a paragraph about this to contribute.md, e.g., where we write about the tests?

[manuelgloeckler]

Yeah, the contribute.md is a good place to do this. And yeah, should not interfere with packaging just adds a bit to the git.

[manuelgloeckler]

This should now be ready to be merged. The last version, "xdist," support, did unfortunately not print out the results to the console and saved them in pickle files.

The new version improved the xdist support:

• Saves now all results in a human-readable .csv. If running xdist, the results of each worker are collected in this .csv.
• On Linux, the printing to terminal now works reliably with and without xdist.
• Saving results did also simplify the internal coda a bit and does not relay on ad-hoc global variables anymore.

[janfb]

Thanks again for pushing this. This is great!