20221024 developer call notes

[hammer]

Pre-notes

• Notes from previous meeting

PRs

Issues

• sgkit#931 import sgkit takes ~1.5s
• sgkit#936 Optimise VCF import
• sgkit#938 Add asv benchmark for compression ratio of some example VCFs

Notes

Attendees

1. Jeff Hammerbacher
2. Tom White
3. Jerome Kelleher
4. Ben Jeffery

Discussion

• JH: why plink export? Isn’t plink less popular now?
• JK: legacy pipeline for Alistair
• TW: understand a bit more about what Ben’s working on
  • Get familiar with sgkit, hence hit import time quickly
  • Xarray doing some upstream work to make imports faster
  • Working w/ Tuberculosis data: VCF per sample, 13k samples
  • Had a scikit-allel workflow that was much slower in sgkit
  • VCF → sgkit → tsinfer tree sequence → sgkit → VCF
  • TW: we’re not good at merging VCF per sample
• JK: thoughts about merging datasets in general?
  • TW: hit this issue recently, not sure how to do it yet
  • TW: maybe use xarray’s merge
  • BJ: Zarr’s virtual blocks could be useful, bit buggy though
• sgkit#938 arose from Ben Elsworth’s work at Our Future Health sgkit#925
  • Mostly just reduce precision for floats
  • JK: can we make the benchmarks more visible?
  • JK: if we’re going to do benchmarking, do it in a paper-able way w/ figures.
  • JH: we should spelunk performance issues in Hail that sent us to sgkit
• sgkit#931
  • TW: Numba compilation time a UX issue
  • JK: can we do compilation at install time?
  • BJ: sgkit has ability to disable Numba cache, but default is cache on
  • TW: will add comment to issue on distance and pandas
  • BJ: first import is 17s, second import is 2s. Numba
  • JH: Compiling code ahead of time in Numba docs
  • TW: Can we just call “import sgkit” as part of the installation process?
  • JK: will probably have to do separately from conda and pip

[timothymillar]

FWIW I ran into pydata/xarray#6483 when using xarray's merge a while back.