🚚 Move PertCurator from wetlab here and add CellxGene Curator test

[falexwolf]

Is part of a sequence of PRs that refactors the curators:

• 🎨 Self-contained definition of CellxGene schema #2412
• 🚚 Move CellxGene Curator from cellxgene-lamin here #2403
• ✨ Schema-based curators: DataFrameCurator #2388

Made a "micro-farland2020" dataset to speed up the tests that were running in the wetlab repository: https://lamin.ai/laminlabs/lamindata/transform/JreAW9tkAHrc

• 🚚 Move PertCurator from wetlab here and add CellxGene Curator test #2408

Also added a test for the CxG curator based on the notebook in cellxgene-lamin.

• 🚚 Move Curator to lamindb cellxgene-lamin#120

[falexwolf]

@Zethson, do you have an idea for how to speed the cxg and pertcurator tests up more and turn them more into proper unit tests?

(I hope that with the last commit things are passing; they've been passing locally in isolation all along but these notebook-derived tests seem to create entanglement)

[Zethson]

@Zethson, do you have an idea for how to speed the cxg and pertcurator tests up more and turn them more into proper unit tests?

All of our XCurator tests are currently like pipelines which is easiest to implement and acts like an integration test for the different XCurator functions but it won't ever allow us to parallelize them.

These tests only need super simple datasets with only 1 var index and 2 categoricals each. You already have simple datasets here but maybe we can downsample and reduce them even more.

Honestly, I think we need to sit down and find ways to speed up the fundamental DataFrameCurator. Maybe polars, maybe somehow more bulk transactions, maybe we can somehow cache more? Spending time on that is probably better than optimizing these tests now, no?

I don't have further simple ideas.

[Zethson]

Lines 26 to 41 don't need to be in the test. Just set that as default for the micro dataset.

[falexwolf]

Thanks and understood!

It's a chicken egg problem. I want fast tests to do the refactor / the refactor will start to speed things up.

I will see what can be done more.

[falexwolf]

Pasting our current run times; total is ~5 min:

============================= slowest 50 durations =============================
130.46s call     tests/curators/test_cxg_curator.py::test_cxg_curator
37.81s call     tests/curators/test_pert_curator.py::test_pert_curator
34.19s call     tests/curators/test_cat_curators.py::test_spatialdata_curator
6.59s call     tests/curators/test_cat_curators.py::test_df_curator
6.01s call     tests/curators/test_cat_curators.py::test_soma_curator
5.67s call     tests/curators/test_cat_curators.py::test_soma_curator_genes_columns
5.66s call     tests/curators/test_cat_curators.py::test_anndata_curator[donor]
5.59s call     tests/curators/test_cat_curators.py::test_anndata_curator[all]
3.44s call     tests/curators/test_cat_curators.py::test_unvalidated_data_object

[codecov]

Codecov Report

Attention: Patch coverage is 76.75676% with 43 lines in your changes missing coverage. Please review.

Project coverage is 91.54%. Comparing base (d503387) to head (50552fe).
Report is 30 commits behind head on main.

Files with missing lines	Patch %	Lines
lamindb/curators/__init__.py	76.75%	43 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2408      +/-   ##
==========================================
- Coverage   91.71%   91.54%   -0.18%     
==========================================
  Files          62       62              
  Lines        9138     9646     +508     
==========================================
+ Hits         8381     8830     +449     
- Misses        757      816      +59     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[github-actions]

🚀 Deployed on https://679db8fb9705fcd0ecd4200b--lamindb-qnwk.netlify.app

[falexwolf]

The commits before already sped up things to ~3 min total, bringing down the CxG test from 130 sec to 50 sec:

50.40s call     tests/curators/test_cxg_curator.py::test_cxg_curator
29.27s call     tests/curators/test_cat_curators.py::test_spatialdata_curator
28.43s call     tests/curators/test_pert_curator.py::test_pert_curator
5.35s call     tests/curators/test_cat_curators.py::test_soma_curator_genes_columns
4.90s call     tests/curators/test_cat_curators.py::test_soma_curator

[falexwolf]

More to come in the next PR with the first refacor.

So far, everything has been merely copy & pasted.

[falexwolf]

In the next PR, the CxG test runtime goes further down to 20sec by removing the coupling to laminlabs/cellxgene: #2412 (comment)