✨ Schema-based curators: AnnDataCurator

[falexwolf]

Is part of a sequence of PRs that refactors the curators:

• 🎨 Let Curator.validate() throw an error #2417
• ♻️ Re-purpose BaseCurator as Curator and introduce CatCurator #2416
• ♻️ Refactor organism handling in curators #2415
• 🚚 Bulk-rename old-style curators to CatCurator #2413
• 🎨 Self-contained definition of CellxGene schema #2412
• 🚚 Move PertCurator from wetlab here and add CellxGene Curator test #2408
• 🚚 Move CellxGene Curator from cellxgene-lamin here #2403
• ✨ Schema-based curators: DataFrameCurator #2388

---

This PR introduces the counterpart to the schema-based DataFrameCurator introduced in:

• ✨ Schema-based curators: DataFrameCurator #2388

The AnnData schema and Curator

Docs preview.

Compare with `DataFrameCurator`

Docs preview.

Notes

Inferred/annotating vs. validating schema

Originally the plan was to populate artifact.schema with the inferred schema that links/annotates by all all features of a dataset in the same way artifact._schemas_m2m already does to make the artifact queryable by all features.

Because we decided to make Schema.composite a self-referential foreign key rather than a ManyToMany, every component can only be used by a single composite schema. This prohibits to use artifact.schema for an inferred composite schema of which we'd have a high number due to variations in the detailed high-dimensional feature sets. For each combination, we'd need to create a copy of a low-cardinality feature set like the obs schema in the quickstart example.

With this PR we now capture both the information about the curation constraints (the "validating schema") and about the inferred/annotating schemas (what we've always done under the name "feature sets"). There might be a more parsimonious solution if we dropped _schemas_m2m and replace it with a good way to create and populate artifact.schema with an inferred schema. But we postpone this consideration to lamindb v2.

This also means that Schema.validated_by is not going to be used and should be hidden from the docs: artifact.schema indicates the validating schema (and not an inferred schema).

Materials

Internal Notion page.

[github-actions]

🚀 Deployed on https://67a0e5224e5a57007d734ed4--lamindb-qnwk.netlify.app

[codecov]

Codecov Report

Attention: Patch coverage is 91.17647% with 21 lines in your changes missing coverage. Please review.

Project coverage is 91.49%. Comparing base (d503387) to head (b3a2f1d).
Report is 37 commits behind head on main.

Files with missing lines	Patch %	Lines
lamindb/curators/__init__.py	91.30%	8 Missing ⚠️
lamindb/models.py	72.41%	8 Missing ⚠️
lamindb/_schema.py	91.48%	4 Missing ⚠️
lamindb/core/_feature_manager.py	88.88%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2418      +/-   ##
==========================================
- Coverage   91.71%   91.49%   -0.23%     
==========================================
  Files          62       63       +1     
  Lines        9138     9697     +559     
==========================================
+ Hits         8381     8872     +491     
- Misses        757      825      +68     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[falexwolf]

This should have read Schema.artifacts instead of Schema.feature_sets to enable schema.artifacts pointing to the M2M.

But rather than this hacky solution, I want to properly redo the feature sets backward compatibility

[falexwolf]

@Koncopd Here is an example for how to use it.