Performance issue ...

[kif]

Hi,
I am looking for a fast and portable CRC implementation for Python ... but I think I found a performance bug:

import numpy,zlib,fastcrc
a = numpy.random.random((2048,2048))
%timeit zlib.crc32(a)
26.4 ms ± 29 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
%timeit fastcrc.crc32.aixm(a.tobytes())
83.9 ms ± 8.78 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
%timeit pyFAI.ext.fastcrc.crc32(a)
7.59 ms ± 2.07 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

The implementation from pyFAI relies on the SSE4 instruction, thus not portable.

[akhilles]

FYI, crc-rs has a new (unreleased) implementation that's ~6x faster in micro-benchmarks: https://github.com/mrhooray/crc-rs#throughput-gibs. However, a portable implementation just won't be able to match the performance of a SIMD implementation.