20200820 developer call notes

[hammer]

Pre-notes

Process and tools

• sgkit#107: Mergify installed and configured
• sgkit#125: Improve contributing docs
• Name committers? Cf. Jerome’s comments

Open PRs

Core operations

• sgkit PR#100 (Dale): sgkit#94 popgen stats
• sgkit PR#114 (Tim): sgkit#85 count_allele_calls
  • sgkit#49 reopened to address Numba issues
• sgkit PR#123 (Jillian): sgkit#95 PCA

API

• sgkit PR#124 (Rafal): sgkit#43 Dataset with schema
  • sgkit#103: Append output variables from functions to input dataset

IO

• sgkit-bgen PR#12 (Eric): sgkit-bgen#9 read_bgen performance issues
• sgkit-plink PR#28 (Eric): sgkit-plink#27 remove slash from reader doc variables
• tomwhite/sgkit-vcf (Tom): sgkit#104 read data from VCF files

Open Issues

• sgkit#23: simulate genotypes
  • Colocate benchmarking, validation, and public dataset documentation/scripts in a separate repo?
• sgkit#113: Common interface to parallel file IO

Notes

Attendees

1. Jeff Hammerbacher
2. Eric Czech
3. Rafal Wojdyla
4. Tom White
5. Jerome Kelleher
6. Alistair Miles
7. Jillian Rowe
8. Dale Tovar

Discussion

• sgkit#107: Mergify installed and configured
  • TL;DR mergify not working. Looks like it's not installed in the repo.
  • Rafal: reset +1 after changes are submitted to a PR
  • Jerome: let’s just add that now
• sgkit#125: Improve contributing docs
  • Docstring style?
  • Eric: Types for arrays, sometimes use dask convention, sometimes not. No opinion.
  • Tom: No opinion yet.
• Name committers? Cf. Jerome’s comments
  • Jeff: Need to define a committer role? Committer role mainly about who merges (adds the automerge tag to a PR). Came up on Document mergify and issue before PR #126.
  • Tom: When to add automerge tag, versus if someone else needs to look at it?
  • Rafal: Can define different groups, link to mergify.
  • Jeff: I'll create a user group. Name? Committer? Maintainer?
  • Alistair: do we need a steering committee?
  • Jerome: who do we copy
• sgkit PR#100 (Dale): sgkit#94 popgen stats
  • One more stat to add, need to rebase, close to ready!
• sgkit PR#114 (Tim): sgkit#85 count_allele_calls
  • Tim started making PRs on scikit-allel this year
  • Alistair hasn’t actually met him; seems to be working on plants?
  • No Dask or Xarray built-ins to use, so now reach for Numba
  • Also brings up the issue of named functions in the API that just do simple operations over containers; for now these named functions seem fine.
• sgkit PR#123 (Jillian): sgkit#95 PCA
  • Alistair notes there’s a weird edge case: variant where all samples have a het genotype, you get an error from the SVD
  • Also raises the issue of where to put validation code and data.
    • Cf. discussion on sgkit#23: simulate genotypes
    • Jerome: how to distinguish between unit tests and validation tests.
    • Eric: when tests rely on tools we don’t want to depend on.
    • Eric: mostly isolate the production of data that unit tests depend upon.
    • So it’s a separate question to run on large data.
    • Rafal: mark some tests as slow to run not w/ PRs but nightly
    • Rafal: can we use hypothesis to generate data?
    • Alistair: for testing sgkit’s PCA, let’s compare to scikit-allel
    • Where to put the scripts and the fixture data they generate?
    • Rafal: case-by-case decision; if it’s easy, just do it
• sgkit PR#124 (Rafal): sgkit#43 Dataset with schema
  • Tom notes that the latest PR is not a wrapper, it’s just schema checking
  • Tom and Eric still need to review
• sgkit-bgen PR#12 (Eric): sgkit-bgen#9 read_bgen performance issues
  • Our wrapper was introducing performance issues
  • Eric benchmarking it on UKB and finding bottlenecks
  • Danilo made a thin Python wrapper for C library, but he’s on vacation
• tomwhite/sgkit-vcf (Tom): sgkit#104 read data from VCF files
  • Testing on 1000G chr22
  • Next step is to get it into the build
  • Tom: gnomAD VCF doesn’t have samples!
  • Jerome notes all we should care about is conversion to Zarr
• sgkit#113: Common interface to parallel file IO
  • Situation: reader that doesn’t work over the network, so we need to pull data off cloud storage and into a worker to do the reading.
  • Maybe just document how to do it with a third party workflow manager

[hammer]

(Posted by @timothymillar)

seems to be working on plants?

Yes Tim works on Kiwifruit (Actinidia chinensis) which is mixed ploidy species complex, commonly 2x, 4x or 6x