Bug description
HilbertIndexer.hilbertCurvePosBytes pads the N-dimensional 63-bit hilbert index to a fixed 63 bytes. 63 bytes only holds 8 dimensions worth of index bits: with 9 or more order columns (accepted by the hilbert sorter — Flink/core only validate size() > 1 — and the Spark Hilbert UDF), paddingToNByte drops entire low-order bytes of the big-endian index, so distinct points collapse into the same sort key and the space-filling order degrades. For 16 dimensions nearly a full byte per point is lost.
Expected behavior
The byte width should cover the full index: 63*N/8 + 1 bytes for N > 8 (the +1 covers BigInteger's sign byte when the top bit is set). Up to 8 dimensions keep the legacy 63-byte width so existing keys stay byte-stable — and even there the low byte is lost for the half of the space whose index has the top bit set (e.g. any NULL order column, which maps to Long.MAX_VALUE); that quirk is preserved for compatibility since hilbert keys are transient sort keys in all current consumers.
Bug description
HilbertIndexer.hilbertCurvePosBytespads the N-dimensional 63-bit hilbert index to a fixed 63 bytes. 63 bytes only holds 8 dimensions worth of index bits: with 9 or more order columns (accepted by the hilbert sorter — Flink/core only validatesize() > 1— and the Spark Hilbert UDF),paddingToNBytedrops entire low-order bytes of the big-endian index, so distinct points collapse into the same sort key and the space-filling order degrades. For 16 dimensions nearly a full byte per point is lost.Expected behavior
The byte width should cover the full index:
63*N/8 + 1bytes for N > 8 (the +1 covers BigInteger's sign byte when the top bit is set). Up to 8 dimensions keep the legacy 63-byte width so existing keys stay byte-stable — and even there the low byte is lost for the half of the space whose index has the top bit set (e.g. any NULL order column, which maps toLong.MAX_VALUE); that quirk is preserved for compatibility since hilbert keys are transient sort keys in all current consumers.