Support of parallel decompression for fixed rate

[doiko]

Is there any plan to support parallel decompression for fixed rate compressed arrays?

[lindstro]

Since zfp already supports parallel CUDA decompression, I assume you're referring to the OpenMP implementation of zfp_decompress (and not the compressed-array C++ classes).

We have plans to add support for both fixed and variable rate decompression over the coming months. It is likely that this feature will be available sometime early next year, even though our primary focus will be on CUDA and HIP.

[doiko]

Indeed @lindstro we are looking for decompressing in CPUs. Do you have any reference for expecting performance for parallel decompressing on multicore CPUs? Throughput vs Number of cores.

[doiko]

Also will the fixed rate parallel cpu is planned to be supported by the python zfpy package?

[lindstro]

Without having implemented it, it's difficult to guess, but we do have performance results for OpenMP parallel compression. See page 11 of this presentation. You can probably extrapolate from the ratio between serial compression and decompression throughput.

[lindstro]

Our immediate plans are to support parallel decompression through the high-level C API. This alone requires re-engineering zfp to keep track of block offsets (for the variable-rate modes). For decompression, such offsets also have to be embedded in the compressed stream, which is not easy to incorporate without breaking backwards compatibility.

To also support this via the zfpy interface, we'd have to make additional changes to the compressed format and the Python API. I think this is doable but it would require rethinking the API before we're ready to support parallel decompression. We'll get to it eventually, but probably not right away.

[doiko]

Hi @lindstro indeed decompression with variable rate sounds a complicate task but is probably simplified for fix rate compression and the python community will get a great tool when such is implemented. Already @zarr and @numcodecs supports zfp through zfpy but the decompressing speed performance might not be satisfactory for many applications I guess.
Thanks for the feedback and the great work.

[lindstro]

We should be able to add OpenMP fixed-rate decompression without much effort and support it through zfpy. For a seamless integration of variable-rate decompression with zfpy, we really need to update the compressed format to allow additional metadata to be stored, which likely won't happen for at least another year. There are several other changes we'd like to make to the next version of the zfp codec that will take time to implement, and we want to make such major changes only infrequently.