Code

LightningDataModule for webdataset#100

Code

Merged

DejunL merged 70 commits intov2-mainNVIDIA/bionemo-framework:v2-mainfrom dejunl/diffdock-datamoduleNVIDIA/bionemo-framework:dejunl/diffdock-datamoduleCopy head branch name to clipboardAug 30, 2024

+1,794-2

Commits

Commits on Aug 29, 2024

• BugFix: parser of docker version string

  Show description for 0d71fa0
  DejunL

  committed
  0d71fa0

• Enhancement: migrate diffdock utils modules from v1

  Show description for 1d30c43
  DejunL

  committed
  1d30c43

• Enhancement: explicitly pass tr/rot/tor_sigma's to t_to_sigma

  DejunL

  committed
  9a7f82e

• Enhancement: add diffdock score model data module and tests

  DejunL

  committed
  756a2e3

• BugFix: control wds shardshuffle randomness

  Show description for 0b8411a
  DejunL

  committed
  0b8411a

• Test: refactor ScoreModelWDS fixture to allow comparing different instances of ScoreModelWDS

  DejunL

  committed
  76ec1c7

• Test: consistent ligand position after NoiseTransform

  DejunL

  committed
  ca0cc65

• BugFix: reduce total_size factor to ensure enough number of shards

  DejunL

  committed
  7d95a67

• Test: parametrize by split type

  DejunL

  committed
  b82ee03

• Enhancement: move NoiseTransform to GenerateNoise in the diffusion module

  Show description for 8420700
  DejunL

  committed
  8420700

• BugFix: always GenerateNoise in score model data

  DejunL

  committed
  662ce27

• Regress: with_length -> with_epoch

  Show description for 99d8e6e
  DejunL

  committed
  99d8e6e

• BugFix: define length on SizeAwareBatching dataset

  DejunL

  committed
  e604a49

• Test: setup data loader

  DejunL

  committed
  8e236d6

• Enhancement: factor out GenerateNoise outside of data module

  Show description for 0e4c7bd
  DejunL

  committed
  0e4c7bd

• Enhancement: use a dict: Split -> List[str] as the input subset complex names

  Show description for 88035d6
  DejunL

  committed
  88035d6

• BugFix: check _xform_gen_wds's avail before composing

  DejunL

  committed
  377f276

• Enhancement: move the batching from wds to webloader

  Show description for 7c5add1
  DejunL

  committed
  7c5add1

• Test: assert sample and batch type

  DejunL

  committed
  69620fb

• Enhancement: factor out the batching method as input webloader composable

  DejunL

  committed
  6ee03c0

• Enhancement: remove local_batch_size

  Show description for 598a4ae
  DejunL

  committed
  598a4ae

• BugFix: remove unused random.seed() call

  DejunL

  committed
  213095b

• Enhancement: rename ScoreModelWDS -> PickledDataWDS

  DejunL

  committed
  1d8f162

• Enhancement: internalize the wds archive formatter

  Show description for add7870
  DejunL

  committed
  add7870

• Test: rename score model test data directory

  DejunL

  committed
  7bbc651

• Test: confidence model data module

  DejunL

  committed
  78c1f80

• Test: parametrize tests with model type

  Show description for e13b325
  DejunL

  committed
  e13b325

• Enhancement: factor out the webdataset-specific method into a base data module class

  Show description for c8a128b
  DejunL

  committed
  c8a128b

• Enhancement: reduce args redundancy in PickledDataWDS by passing *args and **kwargs to the parent class

  DejunL

  committed
  80149e2

• Enhancement: add integration tests in lightning trainer workflows

  DejunL

  committed
  cc911b0

• Enhancement: implement predict stage

  DejunL

  committed
  6541bc2

• BugFix: remove select for sample size > 1

  Show description for bc00979
  DejunL

  committed
  bc00979

• Enhancement: remove legacy dataloader config

  Show description for 975c91e
  DejunL

  committed
  975c91e

• Enhancement: allow input an iterable of generators passed as pipelines

  Show description for a3b07e2
  DejunL

  committed
  a3b07e2

• Enhancement: move the webdataset data module into bionemo-core

  Show description for 2c8a2c8
  DejunL

  committed
  2c8a2c8

• Enhancement: rename WDSModule -> WebDataModule

  DejunL

  committed
  b622a2c

• Enhancement: actually support Iterable[str] as suffix_keys_wds

  DejunL

  committed
  1ffde95

• Test: WebDataModule with mock-up dataset and model

  DejunL

  committed
  5948230

• Enhancement: remove hard-coded shuffle from workflow

  DejunL

  committed
  78ed189

• Enhancement: rename kwargs_dl -> kwargs_wld

  DejunL

  committed
  01bb63e

• Enhancement: replace hard-coded webdatast kwargs with user input

  DejunL

  committed
  c34dd0f

• Test: move data into bionemo2_root/test_data

  DejunL

  committed
  40044fc

• Doc: docstring and readme.md

  Show description for 6f0d4b2
  DejunL

  committed
  6f0d4b2

• Test: PickledDataWDS init and setup()

  DejunL

  committed
  19c3dcd

• Doc: update class description

  DejunL

  committed
  2e6e8db

• Test: re-org test data location

  DejunL

  committed
  9693aba

• Test: generate the test data for webdatamodule

  Show description for 71391a8
  DejunL

  committed
  71391a8

• Enhancement: remove unused functions

  DejunL

  committed
  eff5093

• BugFix: remove license

  DejunL

  committed
  c8c396b

• Enhancement: use list comprehension

  DejunL

  committed
  458611d

• License: clean up old licenses

  DejunL

  committed
  e6f9c3f

• BugFix: add __init__.py

  DejunL

  committed
  c470b21

• BugFix: my email

  DejunL

  committed
  e1e5551

• Package: move webdatamodule under bionemo-webdatamodule

  DejunL

  committed
  0f46ed8

• Package: remove bionemo-diffdock

  DejunL

  committed
  b8f2bf5

• BugFix: change license header

  DejunL

  committed
  57b6c3a

• BugFix: ignore E731 and fix format

  DejunL

  committed
  009fa23

• Regress: use PEP604

  DejunL

  committed
  5d01a19

• Enhancement: pickle within fh context

  DejunL

  committed
  6bcccb6

• Bugfix: default to python 3.10 for consistency

  DejunL

  committed
  168bc9b

• BugFix: f-string typo

  DejunL

  committed
  4c611ba

• Enhancement: use sample counts instead of size to enforce number of tar created

  Show description for 86f97e3
  DejunL

  committed
  86f97e3

• Test: sample overlap tests via md5 hash of pickles

  Show description for 553258d
  DejunL

  committed
  553258d

• Test: assert throw from webdataset when num_workers > num_shards

  DejunL

  committed
  d5284e6

• Enhancement: reuse the webdatamodule dirs_tars_wds arg in PickledDataWDS

  DejunL

  committed
  6e51a7e

• Enhancement: allow multiple data objects for each sample

  DejunL

  committed
  5b6f309

• Regress: remove default python version

  DejunL

  committed
  8522273

• Test: rename test to be consistent with module

  DejunL

  committed
  26a59d6

• Test: factor pickle generation into a new fixture

  DejunL

  committed
  0d5c144

• Test: multiple data objects per sample

  DejunL

  committed
  a35765f