Apply ZSTD_c_deterministicRefPrefix to ZSTD_createCDict_byReference , ensure deterministic output.

[cdcxd]

Solves #4738

TL;DR

The PR ensures deterministic compression output by making the opt-in param ZSTD_c_deterministicRefPrefix apply to ZSTD_createCDict_byReference . It also added a unit test.

Context

My use case was one RAII wrapper loading the dict content from disk, then multiple threads create the cdict by ref. I noticed this non deterministic output may happen once per 3-5 millions of compression calls. I constructed a UT that can reproduce this discrepancy.

By default the output would be slightly different if input data is adjacent to the dict buffer. There's no param to control it. With this change, we can set ZSTD_c_deterministicRefPrefix and opt-in the deterministic mode.

It is a no-op on the attach copy cdict path. Its window never refers to the caller's dictionary buffer.

Why reuse the existing flag

1. I saw the comment

   zstd/lib/zstd.h

   Line 2275 in 82d322c

   * If you really care about determinism when using a dictionary or prefix,

   saying If you really care about determinism when using a dictionary or prefix, like when doing delta compression, you should select this option. So the param is reused here.

2. The behavior at

   zstd/lib/compress/zstd_compress.c

   Line 5280 in 82d322c

   && ( pledgedSrcSize < ZSTD_USE_CDICT_PARAMS_SRCSIZE_CUTOFF

   is inconsistent. If it meets the condition

pledgedSrcSize < ZSTD_USE_CDICT_PARAMS_SRCSIZE_CUTOFF
        || pledgedSrcSize < cdict->dictContentSize * ZSTD_USE_CDICT_PARAMS_DICTSIZE_MULTIPLIER
        || pledgedSrcSize == ZSTD_CONTENTSIZE_UNKNOWN
        || cdict->compressionLevel == 0

It calls ZSTD_resetCCtx_usingCDict , that doesn't honor the flag ZSTD_c_deterministicRefPrefix. Otherwise it would honor the flag. The PR makes both paths honor the flag.

Tests

All unit tests passed.

Benchmark result

Corpus is zstd's own regression data (tests/regression/data.c): github.tar
(757,760 B) compressed against the github.dict dictionary (112,640 B)
, both fetched from the regression-data release the CI was using. x86-64, gcc 13, pinned
16 core.

Measuring the overhead with adjacent input + flag on , otherwise it's a no-op.

input size	level	dev adjacent	dev separate	branch + flag adjacent	branch + flag separate	speed change vs dev-adjacent	compressed size dev-adj → branch-flag
		MB/s	MB/s	MB/s	MB/s	%	bytes
64 KB	1	1,540	1,416	1,334	1,337	-13.4%	3,568 → 3,568 (+0.0%)
64 KB	3	1,008	934	941	940	-6.7%	3,550 → 3,531 (-0.5%)
256 KB	3	1,225	1,035	1,024	1,032	-16.4%	13,754 → 13,592 (-1.2%)
512 KB	3	1,228	1,026	1,012	1,008	-17.6%	26,974 → 26,779 (-0.7%)
640 KB	19	4.9	4.8	4.7	4.7	-4.1%	28,538 → 28,539 (+0.0%)
704 KB (above size gate)	3	1,146	1,008	1,002	999	-12.6%	36,581 → 36,250 (-0.9%)

The adjacent input possibility is really low, so in the benchmark adjacent input was created deliberately. The performance loss with happened-to-be-adjacent-input is actually higher than the comment said (2.5%, link:

zstd/lib/zstd.h

Line 2277 in 82d322c

* at a speed penalty of about ~2.5% if the dictionary and data happened to be

).

I tried the original refPrefix + ZSTD_c_deterministicRefPrefix (not the cdict one) with both HEAD@dev and the PR commit 172b4b6 in 2021, with many types of data, all combinations have 5%-20% speed loss (adjacent, flag on vs off). I wasn't able to reproduce the pre-existing 2.5% speed loss result. It might make sense to update the comment too (not included).