Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Changes to add unit test and Internal Testing #114

Merged
merged 56 commits into from
Jul 30, 2024
Merged
Changes from 3 commits
Commits
Show all changes
56 commits
Select commit Hold shift + click to select a range
413fc25
changes to add unit test
hariharan-devarajan Feb 29, 2024
caf44f8
changes to support DLIO with DYAD
hariharan-devarajan Mar 4, 2024
21a39f8
Adds HDF5 API to pydyad
ilumsden Mar 4, 2024
2fdc2ce
Changes pydyad.hdf.File to pydyad.hdf.DyadFile to avoid name conflicts
ilumsden Mar 4, 2024
dc796ec
changes to support DLIO with DYAD
hariharan-devarajan Mar 5, 2024
549d49f
Merge remote-tracking branch 'origin/feature/unittests' into feature/…
hariharan-devarajan Mar 5, 2024
f30904c
changes to support DLIO with MuMMI and DYAD
hariharan-devarajan Mar 5, 2024
9c6bb66
changes to support DLIO with MuMMI and DYAD
hariharan-devarajan Mar 6, 2024
e5f5119
Does initial changes for DataSpaces baseline plus reworks the data pl…
ilumsden Mar 18, 2024
6e588c0
Updates DataSpaces data plane tests to account for better understandi…
ilumsden Mar 19, 2024
780f95e
Updates CMake to build DataSpaces test
ilumsden Mar 19, 2024
0b4e978
Update unit_test.cpp
hariharan-devarajan Mar 19, 2024
9483d21
add test cases for reproducability
hariharan-devarajan Mar 20, 2024
b309e3d
fix compile issue
hariharan-devarajan Mar 20, 2024
1d5cccb
Adds freopen to DataSpaces tests so that we can get the internal brea…
ilumsden Mar 20, 2024
9745f7f
fixed tests
hariharan-devarajan Mar 20, 2024
214bd7c
Merge branch 'feature/unittests' of github.com:flux-framework/dyad in…
hariharan-devarajan Mar 20, 2024
c8117ef
add test cases for reproducability
hariharan-devarajan Mar 20, 2024
b7fba32
add test cases for reproducability
hariharan-devarajan Mar 20, 2024
2188a0a
fix merge
hariharan-devarajan Mar 20, 2024
c02892a
add test cases for reproducability
hariharan-devarajan Mar 20, 2024
922504b
Merge remote-tracking branch 'origin/feature/unittests' into feature/…
hariharan-devarajan Mar 20, 2024
3e092bd
add test cases for reproducability
hariharan-devarajan Mar 20, 2024
0038eaf
add test cases for reproducability
hariharan-devarajan Mar 20, 2024
5c2e17f
fine tune parameters for reproducability
hariharan-devarajan Mar 20, 2024
77bf89b
add test cases for reproducability
hariharan-devarajan Mar 20, 2024
0035170
fixed tests
hariharan-devarajan Mar 20, 2024
c59ff79
add test cases for reproducability
hariharan-devarajan Mar 20, 2024
339797d
Merge remote-tracking branch 'origin/feature/unittests' into feature/…
hariharan-devarajan Mar 20, 2024
29a8ba1
add test cases for reproducability
hariharan-devarajan Mar 20, 2024
212b56a
fixed mdm
hariharan-devarajan Mar 20, 2024
12ae958
Changes to make Catch MPI friendly
hariharan-devarajan Mar 21, 2024
e8b78e5
Updates DataSpaces test
ilumsden Mar 21, 2024
0508d24
Adds scripts for running the DataSpaces tests for the SC paper
ilumsden Mar 21, 2024
4628e8b
Added scripts for debugging mpi.
hariharan-devarajan Mar 23, 2024
e17b467
Merge branch 'feature/unittests' of github.com:flux-framework/dyad in…
hariharan-devarajan Mar 23, 2024
fb7bf62
add compound test
hariharan-devarajan Mar 23, 2024
362464a
fixed tests
hariharan-devarajan Mar 24, 2024
12352d2
fixed printing
hariharan-devarajan Mar 24, 2024
aae77fd
fixed writing to log
hariharan-devarajan Mar 24, 2024
9a485ff
Fix Clang Formatting
hariharan-devarajan Mar 25, 2024
77bc3eb
Current state of DataSpaces test
ilumsden Mar 25, 2024
885209b
Adds data preloader and PyTorch data loader for DataSpaces
ilumsden Mar 27, 2024
7edc5a6
Updates the setup-env and run scripts for DLIO and DataSpaces
ilumsden Mar 27, 2024
5faf27c
Adds a utility Python application to make the Python bindings of Data…
ilumsden Mar 27, 2024
6d9a3e5
Adds cleanup of dataspaces.conf and conf.ds
ilumsden Mar 27, 2024
93d6516
Adds a script to run DLIO w/ DataSpaces through flux batch
ilumsden Mar 27, 2024
7524c08
Bug fix through Unit Test
hariharan-devarajan Mar 28, 2024
2f163ce
added unit tests for dyad_core gen_path_key
hariharan-devarajan Mar 28, 2024
3c9d69a
Merge branch 'feature/unittests' of github.com:flux-framework/dyad in…
hariharan-devarajan Mar 28, 2024
1dae96e
Updates scripts for DataSpaces testing
ilumsden Apr 3, 2024
4cef77b
Final version of DataSpaces baseline code
ilumsden Apr 3, 2024
91ff208
Final version of DataSpaces stuff for DLIO
ilumsden Apr 3, 2024
fdf571e
Merge branch 'feature/unittests' of github.com:flux-framework/dyad in…
ilumsden Apr 3, 2024
3780bed
remove error file.
hariharan-devarajan Jul 30, 2024
f83e792
Merge branch 'main' into feature/unittests
hariharan-devarajan Jul 30, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
41 changes: 41 additions & 0 deletions pydyad/pydyad/hdf.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
from pydyad.bindings import Dyad

from pathlib import Path

import h5py


class DyadFile(h5py.File):

def __init__(self, *args, **kwargs, dyad_ctx=None, metadata_wrapper=None):
# According to H5PY, the first positional argument to File.__init__ is fname
self.fname = args[0]
if not isinstance(self.fname, Path):
self.fname = Path(self.fname)
self.fname = self.fname.expanduser().resolve()
self.mode = None
if "mode" in kwargs:
self.mode = kwargs["mode"]
elif len(args) > 1:
self.mode = args[1]
else:
raise NameError("'mode' argument not provided to pydyad.hdf.File constructor")
if dyad_ctx is None:
raise NameError("'dyad_ctx' argument not provided to pydyad.hdf.File constructor")
self.dyad_ctx = dyad_ctx
if self.mode in ("r", "rb", "rt"):
if (self.dyad_ctx.cons_path is not None and
self.dyad_ctx.cons_path in self.fname.parents):
if metadata_wrapper:
self.dyad_ctx.consume_w_metadata(str(self.fname), meatadata_wrapper)
else:
dyad_ctx.consume(str(self.fname))
super().__init__(*args, **kwargs)


def close(self):
super().close()
if self.mode in ("w", "wb", "wt"):
if (self.dyad_ctx.prod_path is not None and
self.dyad_ctx.prod_path in self.fname.parents):
self.dyad_ctx.produce(str(self.fname))
1 change: 1 addition & 0 deletions pydyad/setup.cfg
Original file line number Diff line number Diff line change
@@ -8,4 +8,5 @@ classifier =
python_requires = >=3.7
install_requires =
numpy
h5py
dlio_profiler_py @ git+https://github.com/hariharan-devarajan/dlio-profiler.git