[python] tiledbsoma.io.add_obs_layer helper function

[maarten-devries]

Issue and/or context:

Changes:

Notes for Reviewer:

[johnkerl]

This is very interesting -- thanks for sharing, @maarten-devries !!

Let me think about this a bit 🙏

[johnkerl]

The add_X_layer function can be used to write a new component in the X collection -- e.g. if X["raw"] exists, you can create X["norm"] -- as X is a Collection of SparseNDArray.

Here, there's just one obs -- since obs is a DataFrame, not a Collection of DataFrame.

This is really a variant of update_obs.

What this does that tiledbsoma.io.update_obs doesn't:

• It accepts a pyarrow.Table (which is awesome!)

What tiledbsoma.io.update_obs does that this doesn't:

• Check old and new schema, and evolve the schema if necessary
  • That's fine here as long as it's clear, but it might benefit us to assert that the old and new schema match -- rather than deferring to DataFrame.write to raise an exception.
• Check that the old and new row-count are the same -- this allows row-count changes
  • If the row-count is increasing here, that's probably a good thing (although it needs a warning label that this may require a referential-integrity update on X, obsm, and obsp matrices -- caveat emptor)
  • If the row-count is decreasing here -- say from 100 rows to 98 -- then those last two rows from the previous write will still be visible. We'd need to write through the TileDB Core deletion API (which is solid and well-tested, simply not yet wired into tiledbsoma) to mark them as deleted. And in this case, too, a warning label about referential integrity.

[codecov]

Codecov Report

Attention: Patch coverage is 11.11111% with 8 lines in your changes missing coverage. Please review.

Project coverage is 86.12%. Comparing base (fe8e64d) to head (248c657).
Report is 27 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3625      +/-   ##
==========================================
- Coverage   86.23%   86.12%   -0.11%     
==========================================
  Files          55       55              
  Lines        6378     6387       +9     
==========================================
+ Hits         5500     5501       +1     
- Misses        878      886       +8     

Flag	Coverage Δ
python	86.12% <11.11%> (-0.11%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
python_api	86.12% <11.11%> (-0.11%)	⬇️
libtiledbsoma	∅ <ø> (∅)