Zstandard v1.4.9

This is an incremental release which includes various improvements and bug-fixes.

>2x Faster Long Distance Mode

Long Distance Mode (LDM) --long just got a whole lot faster thanks to optimizations by @mpu in #2483! These optimizations preserve the compression ratio but drastically speed up compression. It is especially noticeable in multithreaded mode, because the long distance match finder is not parallelized. Benchmarking with zstd -T0 -1 --long=31 on an Intel I9-9900K at 3.2 GHz we see:

File	v1.4.8 MB/s	v1.4.9 MB/s	Improvement
silesia.tar	308	692	125%
linux-versions*	312	667	114%
enwik9	294	747	154%

* linux-versions is a concatenation of the linux 4.0, 5.0, and 5.10 git archives.

New Experimental Decompression Feature: ZSTD_d_refMultipleDDicts

If the advanced parameter ZSTD_d_refMultipleDDicts is enabled, then multiple calls to ZSTD_refDDict() will be honored in the corresponding DCtx. Example usage:

ZSTD_DCtx* dctx = ZSTD_createDCtx();
ZSTD_DCtx_setParameter(dctx, ZSTD_d_refMultipleDDicts, ZSTD_rmd_refMultipleDDicts);
ZSTD_DCtx_refDDict(dctx, ddict1);
ZSTD_DCtx_refDDict(dctx, ddict2);
ZSTD_DCtx_refDDict(dctx, ddict3);
...
ZSTD_decompress...

Decompression of multiple frames, each with their own dictID, is now possible with a single ZSTD_decompress call. As long as the dictID from each frame header references one of the dictIDs within the DCtx, then the corresponding dictionary will be used to decompress that particular frame. Note that this feature is disabled with a statically-allocated DCtx.

Changelog

• bug: Use umask() to Constrain Created File Permissions (#2495, @felixhandte)
• bug: Make Simple Single-Pass Functions Ignore Advanced Parameters (#2498, @terrelln)
• api: Add (De)Compression Tracing Functionality (#2482, @terrelln)
• api: Support References to Multiple DDicts (#2446, @senhuang42)
• api: Add Function to Generate Skippable Frame (#2439, @senhuang42)
• perf: New Algorithms for the Long Distance Matcher (#2483, @mpu)
• perf: Performance Improvements for Long Distance Matcher (#2464, @mpu)
• perf: Don't Shrink Window Log when Streaming with a Dictionary (#2451, @terrelln)
• cli: Fix --output-dir-mirror's Rejection of ..-Containing Paths (#2512, @felixhandte)
• cli: Allow Input From Console When -f/--force is Passed (#2466, @felixhandte)
• cli: Improve Help Message (#2500, @senhuang42)
• tests: Avoid Using stat -c on NetBSD (#2513, @felixhandte)
• tests: Correctly Invoke md5 Utility on NetBSD (#2492, @Niacat)
• tests: Remove Flaky Tests (#2455, #2486, #2445, @Cyan4973)
• build: Zstd CLI Can Now be Linked to Dynamic libzstd (#2457, #2454 @Cyan4973)
• build: Avoid Using Static-Only Symbols (#2504, @skitt)
• build: Fix Fuzzer Compiler Detection & Update UBSAN Flags (#2503, @terrelln)
• build: Explicitly Hide Static Symbols (#2501, @skitt)
• build: CMake: Enable Only C for lib/ and programs/ Projects (#2498, @concatime)
• build: CMake: Use configure_file() to Create the .pc File (#2462, @lazka)
• build: Add Guards for _LARGEFILE_SOURCE and _LARGEFILE64_SOURCE (#2444, @indygreg)
• build: Improve zlibwrapper Makefile (#2437, @Cyan4973)
• contrib: Add recover_directory Program (#2473, @terrelln)
• doc: Change License Year to 2021 (#2452 & #2465, @terrelln & @senhuang42)
• doc: Fix Typos (#2459, @ThomasWaldmann)