20200807 developer call notes

[hammer]

Pre-notes

Process and tools

• sgkit#82: contributing.md merged!
  • sgkit#26: document our git merge strategy?
• sgkit#7: would be great to get a code of conduct ASAP
• sgkit#30: does anyone want to be involved in the logo process?
• PyData Global submission?
• sgkit#51: Sphinx documentation merged!
  • sgkit#78: Docs (Minimal Numpy, load from MalariaGEN Zarr, load from VCF)
  • sgkit#87 data structures and design philosophy
  • sgkit#88 user tutorials
  • Thoughts from Eric
  • What’s next for documentation? 13 open issues!

Open PRs

• sgkit#40: Add read_vcfzarr
  • Tom to respond to Alistair’s review comments
  • sgkit#56: VCF Zarr improvements has follow up work
• sgkit#57: CLI
  • Eric can share experiences w/ UKB: may be too complex for CLI
• sgkit#66: REGENIE
  • Eric waiting for Tom/Rafal +1
• sgkit#76: HWE
  • Waiting on review. Tom?

Open Issues

• Added a good first issue label
• sgkit#21: genotype call array to dosage; how to get resolution?
• sgkit#29: variant/sample summary statistics
  • Opportunity to collaborate?
• sgkit#43: Domain-specific classes and/or methods
  • Rafal experimenting with generic class-based API in api_dataset branch
  • Related to Eric’s comments on documentation
  • sgkit#91 rename api.py also related
• sgkit#56: VCF Zarr improvements
  • Are these a priority for the Oxford team?
  • It would be nice to do real work with Ag1000G data ASAP
• sgkit#65: monorepo or multiple repos?
  • Increasingly believe we should have a single repo.
• sgkit#67: Requirements for UKB GWAS
  • Top level issue to track RS focus
• sgkit#68: Add benchmark suite
  • Increasing urgency for us as we decide how to best implement core operations
• sgkit#94: popgen stats
  • Great discussion sketching approach before PR!
• sgkit#95: PCA user story
  • Sketch approach similar to sgkit#94?

Notes

Attendees

• Jeff Hammerbacher
• Eric Czech
• Rafal Wojdyla
• Tom White
• Alistair Miles
• Jerome Kelleher
• Eric Kelly
• Jillian Rowe
• Dale Tovar

Discussion

Process and tools

• Git merge strategy
  • Rafal: Use mergify at some point. Until then, rebase.
• Code of conduct
  • Jerome: Use contributor code of conduct.
  • Tom: Outstanding issue is finding an email address.
  • Jeff: Will register email asap.
• Logo
  • Jeff: was going to reach out to person who did numpy and jupyterlab.
• Pydata global
  • Eric nominated to present.
  • Eric: Started on submission this morning. Passed general deadline, smaller chance of getting talk, but will put something together.
• sgkit#51: Sphinx documentation merged!
  • Discussion on what documentation to add next?
  • Jeff: data structures and design philosophy, needs to be documented soon. Provides core mental model for contributors and users. But we're still figuring that out ourselves. Issue Document data structures and design philosophy #87, please add thoughts.
  • Eric: I assumed, we don't have data structures, we just use xarray. Hard to express all constraints on a dataset, even genotype calls, because e.g. some methods might require specific ploidy, or biallelic variants. So we use xarray, and use documentation on functions that consume or produce data. Hail does something similar.
  • Jerome: Short version of that would be a great start. How we imagine data flowing through API. Makes sense.
  • Eric: VCF example forces a lot of that. Plink and BGen less so. Worked example with VCF would be good.
  • Jeff: How we use xarray, put that documentation in the readers, e.g., bgen reader or zarrvcf. Start with documentation there, and documentation in functions that consume like count_alleles. Trying to write conceptual documentation is a little hard in detail.
  • Eric: Need for global namespace. E.g., "variant_allele_count". Make that part of the documentation. Registered namespace of known quantities.
  • Tom: Like a data dictionary or glossary.
  • Alistair: Sounds good, like reserved names for variables, plus some constraints.
  • Jeff: Like the constraints-based approach. Also don't want to get in the way of pydata containers.

Open PRs

• sgkit#40: Add read_vcfzarr
  • Tom: Will resolve issue around object dtype. Will then follow up to add support for cloud stores and other features.
  • Jeff: We've been aided in work on bgen reader by trying to do work on UK biobank. Would be good to do something similar for popgen and Ag1000G dataset. Requirements for UKB GWAS #67, high-level issue on GWAS on UK BioBank. Would be good to have a similar issue for population genomics.
  • [discussion of datasets in the cloud]
  • Initial docs on public datasets #9 - includes the Ag1000G phase 2 and the human 1000 genomes phase 3 variation datasets
• sgkit#57: CLI
  • Eric: lots of variation encountered with loading plink datasets. Idiosyncrasies.
  • Jerome: UK biobank definitely. But can be a high bar. If someone has a modest plink dataset, want to turn it into a format they can run sgkit on. Asking them to write a script, might turn people away. Ultimately CLIs we will want, lots of users are CLI-centric. VCF files, often a default conversion will be good enough, even if could be tweaked to improve it.
  • ...initial CLI should be super simple, good user experience in getting data into sgkit. Nice interface, progress bars, uses all your CPUs.
  • ...question of how to split things up, related to how to organise multiple modules?
  • Jeff: Issue Monorepo vs. multi-repo #65. We're converging on single repo for developer experience. Htslib is the big outstanding issue. Won't get resolution until tackle VCF. Also had call with Karl Cady and Danilo Horta, authors of bgen and bed file readers, have good communication. Could build wheels as part of CI, take away concerns about users having to compile stuff. A fair few open questions still. My approach would be to tackle VCF, then multirepo, then CLI.
  • Rafal: Can we build wheels for htslib?
  • ...
  • Jerome: If we're vendoring, are there licensing issues? htslib is OK, but wondered about the bed and bgen file formats, C libraries. Can get complicated with htslib with plugins maybe we don't need.

Open issues

• sgkit#21: genotype call array to dosage; how to get resolution
• sgkit#29: variant/sample summary statistics
  • Jeff: Eric has what he needs. But if additional needs maybe opportunity to collaborate.
  • Tom: Starting to compile a PR. I'll post that to anchor discussion, then tag people in.
• sgkit#43: Domain-specific classes and/or methods
  • Jeff: Rafal has a sketch.
  • Rafal: Can discuss later. Aligned with philosophy here, not a rigid class systems.
  • Jeff: Useful to explore these questions. ... Will need to feel our way through by implementing some workflows.
• sgkit#68: Add benchmark suite
  • Jeff: Becoming more urgent. E.g., allele count, different ways it could be done. Need benchmarking to help make decisions about which way to go. Good opportunity for collaboration. E.g., ASV. Invest in alongside building out core operations.
• sgkit#94: popgen stats
  • Dale: working on getting tskit and msprime installed, then will put up a PR