H5Z_FLAG_OTPTIONAL compliant behavior when lz4 compressoin enabled #111
Conversation
|
Hi @hernot it would be great to get this merged at some point soon, thanks for all the effort. It kind of arrived right about the time we were merging in the zstd support, and a large number of other changes so it's ended up with a bunch of merge conflicts. Would you be able to resolve those and then I'll take a close look at the changes? I guess this also won't have the compliant behaviour for the zstd pipeline, but we can look into adding that later on based on your changes. Thanks! |
|
Sure. I already expected. Just need to find some time to rebase to master/head and check the differences. I guess that zstd support is not so much different from lz4 so i can have a look and include necessary adjustments or at leas come up with some ideas how to also apply to zstd. |
When lz4 compression is enabled and the H5Z_FLAG_OPTIONAL flag is set compression is abborted and an error returned when the size of output stream exceeds the size of the input stream in bytes. This is the behaviour for compression filters recommended by the libhdf5 documentation for the HZ5_FLAG_OPTIONAL. Added compiletime macros allowing fast abort in single threaded mode and when compiled with omp support. In the latter case it is ensured that no futher chunk is shuffled and compressed after abort has been signalled by any of the threads. The actual abort occurs as soon as all workers have safely cleared their queue. Missing calls to ioc_destroy in case of error by bshuf_blocked_wrap_fun ======================================================================== ioc_destroy function was only called on return when bshuf_bloced_wrap_fun finished successfully. In any error case just the error coded was returned but the ioc_pipipine was not released. In single threaded mode this does not have any effect as ioc_destroy is a noop which is very likely removed during call site optimization step. But when compiling with omp enabled than it is responsible that all synchonization barriers and locks are properly released again. Which was not the case when bshuf_blocked_wrap_fun aborted returning an error code. Added Compile time macros for tearing down ioc before returning error code. Bitte geben Sie eine Commit-Beschreibung für Ihre Änderungen ein. Zeilen, die mit '#' beginnen, werden ignoriert, und eine leere Beschreibung bricht den Commit ab.
not able to run tests cause of some errors need help figuring and fix
|
Ok rebase done but have some issues with running tests running python 3.8.10 and h5py 3.6.0 compiled with
and set In other words i need some hint what could be missing or wrong. git difftool -t vimdiff master is not really helpful for figuring. |
|
Ok figured at least for building and text_ext.py and test_h5filter.py libzstd latest version seems to include some assembly language files but these have postfix Should i squash my commits into one single commit or keep them? Question Just see that pullrequest #78 is still open since 3 years. What is the reason it was not considered? If it is just cause sumiter had no time to resolve conflicts? In this case I could offer to reflect it in this pull-request. As a side-mark: Windows can be a bitch with its tons of versions of c runtime environment and their neither compatible nor linked memory allocators. |
Makes bitshuffle with lz4 compression enabled abort when output size exceeds input size as recommended by libhdf5 documentation when H5Z_FLAG_OPTIONAL is set on compression/compressing filters as h5py high level interface does for all registered compression filters. Plausibility check whether bithsuffling in combination with lz4 can reduce size of input more than the 16 bytes required to store raw input size, overall output size and the size of the first compressed block plus a minimum of 2bytes payload resulting from lz4 compression following bitshuffling. If it input is too short for this minimum gain bithsuffling with lz4 compression is not attempted at all. In case lz4 compression is not enabled all data is shuffled disregarding whether H5Z_FLAG_OPTIONAL is set or not.
Further noticed that when compiled with omp enabled than in case of error the locks an mutices used to protect ioc_chain are not torn down properly in case of error. Just when bithsullfling with and without compression enabled completes successfully. Details see commit message.
This PR is related to and resolves issue #110
Awaiting your your comments and review.