create: --map and --reuse-from for efficient block device snapshot backups

[ThomasWaldmann]

Implements the idea from #4363: efficiently back up snapshots of (thin provisioned) block devices by not reading the parts whose contents we already know.

borg create --map MAPFILE

Gives borg an externally generated map of the input's content ranges, so it does not need to read all of the input. The map file has one range per line, START LENGTH STATE (decimal or 0x hex), # starts a comment:

• data: read and back up that range.
• zero: the range reads as all-zero bytes - borg stores a hole of that size without reading it.

The map must be sorted, contiguous, start at offset 0 and cover the exact input size (a mismatch is an error - that catches maps belonging to a different input). --map requires exactly one input path, a regular file or (with --read-special) a block device.

This alone already enables cheap full backups of thin LVs: only the allocated ranges are read.

The chunkers already accepted an fmap (and FileFMAPReader already handled partial maps) - this just exposes it.

borg create --reuse-from ARCHIVE [--reuse-path PATH]

Adds a third map state, same: the range is identical to that range of the input as backed up in the reference archive (usually the backup of the previous snapshot). For such ranges, borg reuses the reference item's chunks instead of reading the input.

build_reuse_plan() walks the reference item's chunk list with cumulative offsets: a reference chunk is reused if its whole extent lies inside same ranges and the chunk still exists in the repo (otherwise it is read again, like the files cache does when it lost a chunk). Reference chunks that only partially overlap same ranges are re-read completely, so the byte stream is preserved and the result is correct with any chunker; with a fixed block size chunker, read windows align with the reference chunk grid and nothing is re-read at the edges. Each read segment gets its own chunkify() call, so chunks never span a gap left by reused parts. Growing and shrinking inputs are handled (tail beyond the reference chunks is read, reference chunks beyond the new size are dropped).

The result is an ordinary, complete item - no repo/archive format changes, extract/check/mount/diff work as usual.

scripts/lvm-thin-map.py

Converts thin-provisioning-tools XML into the map format:

• full mode: thin_dump --dev-id output -> allocated ranges as data, everything else zero.
• delta mode: thin_delta output -> same/different/right_only/left_only mapped to same/data/data/zero.

The script's docstring documents the complete workflow (snapshot creation, reserve_metadata_snap / release_metadata_snap, which snapshot to keep as the reference for the next delta). docs/usage/notes.rst points from the existing --read-special LVM example to this approach.

Typical usage:

# initial backup
thin_dump -m --dev-id $(lvs --noheadings -o thin_id vg/snap1) /dev/mapper/vg-pool_tmeta \
    | scripts/lvm-thin-map.py full --device /dev/vg/snap1 > snap1.map
borg create --read-special --chunker-params fixed,4194304 --map snap1.map lv-backup /dev/vg/snap1

# incremental backup
thin_delta -m --snap1 <id1> --snap2 <id2> /dev/mapper/vg-pool_tmeta \
    | scripts/lvm-thin-map.py delta --device /dev/vg/snap2 > snap2.map
borg create --read-special --chunker-params fixed,4194304 \
    --map snap2.map --reuse-from lv-backup lv-backup /dev/vg/snap2

Trust

The map is trusted: if it is wrong, the archive does not match the input and borg cannot detect that. This is documented in the --help epilog and the docs, together with the advice to do periodic full read backups. Note that thin pool metadata is the same information the kernel uses to serve reads, so it is authoritative rather than heuristic - unlike e.g. mtime based change detection.

Tests

• unit tests for the map parser and for build_reuse_plan (reuse/read/partial overlap/merging, zero ranges, grown/shrunk input, missing chunk).
• integration tests for --map and --reuse-from, incl. a test that deliberately puts different data in a range declared same to prove the reference chunks are really reused (and one for zero), a CDC test with an unaligned change, resize tests and the error cases.

[codecov]

Codecov Report

❌ Patch coverage is 92.51701% with 11 lines in your changes missing coverage. Please review.
✅ Project coverage is 86.19%. Comparing base (def9383) to head (b35d627).
⚠️ Report is 17 commits behind head on master.
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
src/borg/archiver/create_cmd.py	77.50%	6 Missing and 3 partials ⚠️
src/borg/archive.py	97.26%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #10137      +/-   ##
==========================================
- Coverage   87.05%   86.19%   -0.86%     
==========================================
  Files         101      101              
  Lines       17848    17999     +151     
  Branches     2705     2752      +47     
==========================================
- Hits        15538    15515      -23     
- Misses       1609     1781     +172     
- Partials      701      703       +2     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.