[5.0.0] question about the organisation of the project

[picca]

Hello,

when packaging for Debian, I found it difficult to know which plugin is a copy from another upstream and which plugin is a self made plugin which should be build and distribut with python3-hdf5plugin.

another issue is the pytables plugin state, it provides c-blosc and c-blosc2 plugins whcih are in fact also sort of fork of the cblosc-hdf5 projects etc...

I would like to package using as much as possible the real upstream of these plugin.
the current situation is that you are forking all the plugins (which is great if the current upstream is sort of dead), but not that great if upstream is still alive.

I know the hdf5 plugin ecosystem is a real mess. Maybe it would be great to start by adding the upstream url of each plugin in the setup.py file. This way I would be easier to find the 'real' upstream.

Do you also have a idea of the dead/no dead status of these upstreams ?

thanks for considering

Fred.

[vasole]

I would like to package using as much as possible the real upstream of these plugin.
the current situation is that you are forking all the plugins (which is great if the current upstream is sort of dead), but not that great if upstream is still alive.

We DO use the upstream package. No modification of it is made at our side, therefore I would not speak about a fork. We regularly update the package with the upstream code (git subtree)

[picca]

We **DO** use the upstream package. No modification of it is made at our side, therefore I would not speak about a fork. We regularly update the package with the upstream code (git subtree)

Where can I find the upstream links of all these subtree ? I am not that familiar with subtree . Fred

[vasole]

Nothing prevents a user/packager from building the complete hdf5plugin module as supplied.

I think you are making your life more difficult trying to synchronize everything. That is something we try to do with new releases, but it is not needed.

You can build and install the individual plugins as upstream foresees. Nothing will stop working. This module is designed in such a way that if a plugin is already available, it will not be overwritten by this module. This module is made exactly to simplify end users life... You can build and install the upstream plugins if you want (and manage to do so) and have them installed in parallel.

The information about the different sources:

http://www.silx.org/doc/hdf5plugin/latest/information.html#license

Alternatively, you can search squashed in the commit history.

Example, commit c204480

Squashed 'src/fcidecomp/' content from commit e88f83c0

git-subtree-dir: src/fcidecomp
git-subtree-split: e88f83c03bafcd0769c167dca14aa7aabf728e1b

[picca]

Nothing prevents a user/packager from building the complete hdf5plugin module as supplied.

yes :))

I think you are making your life more difficult trying to synchronize everything. That is something we try to do with new releases, but it is not needed.

I do not try to synchronize things, I just try to find the person in charge of each plugins. This way our users could look at the Debian tracker info and find the real person in charge of these plugins. I agree with you this is not something that easy and enjoyable :). Maybe we should discuss with these upstreams in order to convince them to contribute directly into hdf5plugins.

You can build the individual plugins as upstream foresees. Nothing will stop working. This module is designed in such a way that if a plugin is already available, it will not be overwritten by this module. This module is made exactly to simplify end users life... You can build the upstream plugins if you want (and manage to so) and having installed in parallel.

yes :)

The information about the different sources: http://www.silx.org/doc/hdf5plugin/latest/information.html#license

thanks

Alternatively, you can look about `squashed` in the commit history.

I will see

[vasole]

Maybe we should discuss with these upstreams in order to convince them to contribute directly into hdf5plugins.

I'm afraid that will never work nor we want to take the responsibility of maintaining upstream code.

The plugins are meant to have their live:

• The upstream building mechanism relies on cmake or whatever the upstream uses.
• The fact of having a plugin is sometimes an afterthought because what upstream originally wants is a compressor...

The only way forward would be the HDF5 Group taking care of the work we are doing. Only they could be considered legitimate to host upstream code and binaries. At that point, this project will not have any reason of being.

[picca]

I'm afraid that will never work nor we want to take the responsibility of maintaining upstream code.

ok

- The upstream building mechanism relies on cmake or whatever the upstream uses. - The fact of having a plugin is sometimes an afterthought because what upstream originally wants is a compressor...

yes an dI need to buils these plugins with the hdf5 flavour available in Debian :)

The only way forward would be the HDF5 Group taking care of the work we are doing. Only they could be considered legitimate to host upstream code and binaries. At that point, this project will not have any reason of being.

Yes. In real life, the only plugin I need is bitshuffle for the MX images of their eiger detector... which one are you using for real at the ESRF ?

[vasole]

which one are you using for real at the ESRF ?

I think bitshuffle did not change since our hdf5plugin version 4.1.0 @t20100 could confirm.

To me commit 53d3808

git-subtree-dir: src/bitshuffle
git-subtree-split: b9a1546133959298c56eee686932dbb18ff80f7a

[vasole]

It seems to be the case:

https://github.com/silx-kit/hdf5plugin/releases/tag/v4.1.0

You can follow the whole history https://github.com/silx-kit/hdf5plugin/releases

[picca]

----- Le 5 Mar 25, à 17:40, V. Armando Solé ***@***.*** a écrit :

vasole left a comment (silx-kit/hdf5plugin#329) > which one are you using for real at the ESRF ? I think bitshuffle did not change since our hdf5plugin version 4.1.0 @t20100 could confirm.

I take care of this one directly from the bitshuffle 0.5.2 repository. here https://tracker.debian.org/pkg/bitshuffle But I would like to know which compression is really in use or planned to be at the ESRF.

[picca]

https://github.com/silx-kit/hdf5plugin/releases/tag/v4.1.0

I do not discuss the fact that you are up to date :-)

[t20100]

Do you also have a idea of the dead/no dead status of these upstreams ?

You'll have to check the upstreams.
For the official list of plugins and link to upstreams, you can look at: https://github.com/HDFGroup/hdf5_plugins/blob/master/docs/RegisteredFilterPlugins.md#list-of-filters-registered-with-the-hdf-group
or the link @vasole provided you in hdf5plugin documentation.

In real life, the only plugin I need is bitshuffle for the MX images of their eiger detector...
which one are you using for real at the ESRF ?

Currently the same as you: bitshuffle (and lz4 for older files).

Maybe we should discuss with these upstreams in order to convince them to contribute directly into hdf5plugins.

This project is not meant to be the place to store plugins: we embed upstream code for convenience so a user can install all plugins at once from source without problem, and it's only for Python.
The central repo for plugins (or at least a curated list of plugins) should be https://github.com/HDFGroup/hdf5_plugins . We discussed this with the HDFGroup some time ago, but the situation is not clear.

[picca]

You'll have to check the upstreams. For the official list of plugins and link to upstreams, you can look at: https://github.com/HDFGroup/hdf5_plugins/blob/master/docs/RegisteredFilterPlugins.md#list-of-filters-registered-with-the-hdf-group or the link @vasole provided you in `hdf5plugin` documentation.

thanks

> In real life, the only plugin I need is bitshuffle for the MX images of their > eiger detector... > which one are you using for real at the ESRF ? Currently the same as you: bitshuffle (and lz4 for older files).

ok, I am now confident that we have something usable out of the box on trixie.

> Maybe we should discuss with these upstreams in order to convince them to > contribute directly into hdf5plugins. This project is not meant to be the place to store plugins: we embed upstream code for convenience so a user can install all plugins at once from source without problem, and it's only for Python. The central repo for plugins (or at least a curated list of plugins) should be https://github.com/HDFGroup/hdf5_plugins . We discussed this with the HDFGroup some time ago, but the situation is not clear.

thanks for the clarification. We will continue this work in order to use the upstream repo for all our plugins. thanks a lot Fred