Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HDF5: Remove Support for Old Versions #1514

Open
ax3l opened this issue Aug 18, 2023 · 5 comments
Open

HDF5: Remove Support for Old Versions #1514

ax3l opened this issue Aug 18, 2023 · 5 comments
Assignees
Milestone

Comments

@ax3l
Copy link
Member

ax3l commented Aug 18, 2023

If we bump up our support to newer HDF5 versions, then we can optimize meta-data features & performance:

The call H5Pset_libver_bounds will set the guaranteed compatibility with older readers. The less old versions are supported, the more performant we can be.

Cool features in Quincy Koziol's ATPESC23 or HUG23 slides (to be linked):

  • size of metadata structures H5Pset_istore_k, H5Pset_sym_k, etc.
  • metadata aggregation in larger blocks H5Pset_meta_block_size
  • Align objects in the file H5Pset_alignment
  • Metadata cache size
  • Page allocation and page buffering (see: File Space management)

We got to check, but can probably dump everything before 1.12 because our users & readers use recent pip/conda packages that mostly ship HDF5 1.14 already.

cc @jeanbez already increased quite a few of these buffers, but I was unaware of H5Pset_libver_bounds performance implications so far.

@ax3l ax3l added this to the 0.16.0 milestone Aug 18, 2023
@ax3l ax3l self-assigned this Aug 18, 2023
@jeanbez
Copy link
Member

jeanbez commented Aug 23, 2023

@ax3l let me know the minimum version you plan to support so I can open a PR with this. I can check the other optimizations since most have already been applied. I'm curious about their possible impact on openPMD.

@ax3l
Copy link
Member Author

ax3l commented Jun 11, 2024

@jeanbez I think we want to drop at least everything older than HDF5 1.10.

I would like to bump to 1.12+ in the comings years, too, but some distros like Ubuntu is still a bit behind: https://repology.org/project/hdf5/versions

@jeanbez
Copy link
Member

jeanbez commented Jun 12, 2024

Got it, I believe that's a good way to go! I'll take a look into that and at the changes to include the H5Pset_libver_bounds

@ax3l
Copy link
Member Author

ax3l commented Aug 5, 2024

Awesome! Yes we would cut the 0.16 release later this month. @jeanbez do you have time to post this small PR?

@jeanbez
Copy link
Member

jeanbez commented Aug 13, 2024

@ax3l, what is the cut-off date for the release?

@ax3l ax3l modified the milestones: 0.16.0, 0.17.0 Oct 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants