pack-aggregate: cheaply consolidate loose objects and small packs

.gitignore

@@ -119,6 +119,7 @@
 /git-name-rev
 /git-notes
 /git-p4
+/git-pack-aggregate
 /git-pack-redundant
 /git-pack-objects
 /git-pack-refs

Documentation/config/pack.adoc

@@ -39,11 +39,26 @@ is set to "multi", reuse parts of just the bitmapped packfile. This
 can reduce memory and CPU usage to serve fetches, but might result in
 sending a slightly larger pack. Defaults to true.
 
+pack.aggregateMaxObjects::
+	Default for linkgit:git-pack-aggregate[1]'s `--max-objects`
+	option.  Defaults to `100000`; see that option for details.
+
+pack.aggregateMaxInputPackSize::
+	Default for linkgit:git-pack-aggregate[1]'s `--max-input-pack-size`
+	option.  Defaults to `0` (automatic); see that option for details.
+
+pack.aggregateMaxLooseObjects::
+	Default for linkgit:git-pack-aggregate[1]'s `--max-loose-objects`
+	option.  Defaults to `100000`; see that option for details.
+
+pack.aggregateMaxPacks::
+	Default for linkgit:git-pack-aggregate[1]'s `--max-packs`
+	option.  Defaults to `10000`; see that option for details.
+
 pack.island::
 	An extended regular expression configuring a set of delta
 	islands. See "DELTA ISLANDS" in linkgit:git-pack-objects[1]
 	for details.
-
 pack.islandCore::
 	Specify an island name which gets to have its objects be
 	packed first. This creates a kind of pseudo-pack at the front

Documentation/config/repack.adoc

@@ -64,3 +64,21 @@ repack.midxNewLayerThreshold::
 When the tip layer has fewer packs than this threshold, those packs are
 excluded from the geometric repack entirely, and are thus left
 unmodified. Must be at least 1. Defaults to 8.
+
+repack.aggregateOnce::
+	If set to true, linkgit:git-repack[1] will run
+	linkgit:git-pack-aggregate[1] once before inspecting the packs
+	and loose objects to repack.  This can quickly reduce a large
+	number of files before the more thorough repack begins.
+	Defaults to false.  Can be overridden on the command line with
+	`--aggregate-once` or `--no-aggregate-once`.
+
+repack.aggregateLoop::
+	If set to true, linkgit:git-repack[1] will spawn a background
+	linkgit:git-pack-aggregate[1] for the duration of its main
+	pack-objects run.  The aggregator rolls up small packs and
+	loose objects that arrive after pack-objects has enumerated
+	its inputs, preventing them from accumulating in
+	`objects/pack/` and slowing other Git operations on busy
+	servers.  Defaults to false.  Can be overridden on the
+	command line with `--aggregate-loop` or `--no-aggregate-loop`.

Documentation/git-pack-aggregate.adoc

@@ -0,0 +1,202 @@
+git-pack-aggregate(1)
+=====================
+
+NAME
+----
+git-pack-aggregate - Quickly roll up loose objects and small packs
+without doing any delta compression or search
+
+SYNOPSIS
+--------
+[verse]
+'git pack-aggregate' (--once | --loop) [--interval=<seconds>]
+                   [--min-loose=<n>] [--min-packs=<n>]
+                   [--max-loose-objects=<n>]
+                   [--max-objects=<n>] [--max-packs=<n>]
+                   [--max-input-pack-size=<bytes>]
+                   [--keep-pack=<pack-name>]
+                   [--exclude-pack-file=<path>]
+                   [--exclude-loose-file=<path>]
+                   [--parent-pipe-fd=<n>]
+
+DESCRIPTION
+-----------
+
+`git pack-aggregate` rolls accumulated loose objects and small
+packfiles into new packs.  It can run one cycle as a quick recovery
+step, or run cycles periodically to keep new material under control
+while longer-running maintenance proceeds.  Each cycle has two steps:
+
+1. Bundle local loose objects (minus any listed in
+   `--exclude-loose-file`) into new packs and unlink the loose copies.
+2. Aggregate small local packs (minus any excluded by `--keep-pack` or
+   `--exclude-pack-file`, or those referenced by the multi-pack-index,
+   or those carrying a `.keep`, `.promisor`, `.mtimes`, or `.bitmap`
+   sidecar) into new packs and unlink the source packs.  If step 1
+   writes a single pack, it is naturally a candidate here and will
+   normally be folded into the step-2 output.  Multiple loose-rollup
+   packs are left for a later cycle.
+
+No delta search is performed and no bitmaps or commit-graphs are updated.
+Existing packed representations are reused where possible.  When an
+object has both delta and full-object representations in the input
+packs, aggregation prefers the existing delta.  All output packs are
+written with `pack-objects --window=0 --mark-bad-deltas`, so each one
+ships with a `.baddeltas` sidecar (see linkgit:gitformat-pack[5]).  A
+subsequent thorough repack (linkgit:git-repack[1]) honors that marker
+and reconsiders intra-pack deltas at that time.
+
+The `pack.packSizeLimit` setting can split either step's output into
+multiple packs.  All outputs from a batch are installed before its
+input packs or loose objects are removed.  Size-based splitting may
+require expanding deltas whose bases are in another output pack.
+By default, step 2 skips packs larger than half that limit to avoid
+repeatedly copying nearly full packs without reducing the pack count.
+See `--max-input-pack-size` for a smaller input-size threshold.
+
+This command serves two related purposes.  With `--once`, it provides a
+quick recovery step for a repository that has accumulated many loose
+objects or small packs.  With `--loop`, it keeps new material under
+control while a long-running repack is in flight.  Both reduce the
+number of files that unrelated Git operations must scan, while leaving
+a later thorough linkgit:git-repack[1] to optimize deltas and pack
+layout.  The `repack.aggregateOnce` and `repack.aggregateLoop`
+configuration variables (see linkgit:git-repack[1]) let `git repack`
+request either behavior independently.
+
+Like linkgit:git-repack[1], `git pack-aggregate` takes no locks of its
+own.  Callers that need serialization must arrange it themselves (for
+example by running under `git gc`).
+Callers can declare "do not touch these inputs" with `--keep-pack`,
+`--exclude-pack-file`, and `--exclude-loose-file`.
+When spawned by `git repack`, protection across the parent's MIDX update
+is layered: each aggregation cycle skips packs already named by the
+current MIDX and packs with protective sidecars; `--emit-input-packs`
+protects every local pack visible when the parent's `pack-objects`
+prepares its inputs; and temporary `.keep` files protect the parent's new
+packs before their indexes become visible and remain until after its MIDX
+write.  The parent's explicit MIDX include list is drawn from those
+protected existing and newly-written packs, so the aggregator cannot
+retire a pack that the parent is about to reference.
+
+OPTIONS
+-------
+
+--once::
+	Run a single cycle and exit.  Exactly one of `--once` or
+	`--loop` is required.
+
+--loop::
+	Run cycles forever, sleeping `--interval` seconds between
+	cycles.  Stops on `SIGTERM`, `SIGHUP`, or `SIGINT`.
+
+--interval=<seconds>::
+	Number of seconds to sleep between the end of one cycle and the
+	start of the next.  Defaults to 60.  Only meaningful with
+	`--loop`.
+
+--min-loose=<n>::
+	Skip aggregation of loose objects if fewer than `<n>` remain
+	after applying `--exclude-loose-file`.  Defaults to 5.
+
+--min-packs=<n>::
+	Skip aggregation of small packfiles if fewer than `<n>`
+	aggregatable packs remain after applying all exclusions.  Defaults
+	to 5.
+
+--keep-pack=<pack-name>::
+	Exclude the given pack from aggregation. `<pack-name>` is the pack
+	file name without a leading directory (e.g. `pack-123.pack`).
+	This option can be repeated to keep multiple packs.
+
+--exclude-pack-file=<path>::
+	Read a list of pack basenames (one per line) from `<path>` and
+	never touch any of those packs.  Lines may name a basename with
+	or without a `.pack` or `.idx` suffix; the suffix is stripped.
+	Blank lines and lines beginning with `#` are ignored.  When
+	`git pack-aggregate` is spawned by a long-running `git repack`,
+	this file is populated by that `pack-objects` itself via
+	`--emit-input-packs`, and may contain a conservative superset
+	of the packs it will actually consume (such as in `--geometric`
+	mode).
+
+--exclude-loose-file=<path>::
+	Read a list of loose object IDs (one per line) from `<path>`
+	and never pack or unlink any of those loose objects.  Blank
+	lines and lines beginning with `#` are ignored.  When `git
+	pack-aggregate` is spawned by a long-running `git repack`, this
+	file is populated by that `pack-objects` itself via
+	`--emit-input-loose`.
+
+--parent-pipe-fd=<n>::
+	An inherited pipe file descriptor whose write end the parent
+	process holds open.  When the parent exits the pipe is closed
+	and `git pack-aggregate` exits at the next cycle boundary,
+	without waiting out the remainder of `--interval`.  This is an
+	internal plumbing option used by `git repack` to keep its
+	companion aggregator from outliving it.
+
+--max-loose-objects=<n>::
+	Process at most `<n>` loose objects per tranche.  All output packs
+	are installed and the loose copies are removed before the next
+	tranche begins, so repository performance can improve incrementally
+	during recovery from an extreme backlog.  Loose objects created
+	after the cycle starts are left for the next cycle.
+	If multiple packs are produced, they are left for a later aggregation
+	cycle instead of being copied again immediately.
+	Defaults to the `pack.aggregateMaxLooseObjects` configuration value,
+	or 100000 if that is unset.  A value of `0` disables the limit.
+
+--max-objects=<n>::
+	Skip any pack that contains more than `<n>` objects, leaving it
+	untouched by step 2.  The object count is estimated cheaply from
+	the size of the pack's `.idx` file rather than by opening it.
+	This keeps `git pack-aggregate` focused on rolling up the small
+	packs it is meant for, instead of repacking a large base pack
+	(which would amount to a near-full repack).  Defaults to the
+	`pack.aggregateMaxObjects` configuration value, or 100000 if that
+	is unset.  A value of `0` disables the limit.  Note that packs
+	already referenced by the multi-pack-index are excluded
+	regardless of this setting, so on a normally-maintained
+	repository the large base pack is skipped anyway.
+
+--max-input-pack-size=<bytes>::
+	Skip input packs whose `.pack` file is larger than `<bytes>`.
+	This byte-size gate applies in addition to `--max-objects`;
+	it does not limit loose-object rollup or individual output packs.
+	The suffixes `k`, `m`, and `g` are supported.
++
+Defaults to `pack.aggregateMaxInputPackSize`, or `0` (automatic).
+Automatic selection uses half of `pack.packSizeLimit`, after applying
+the same 1-MiB minimum for nonzero output limits as `pack-objects`.
+An explicit positive input limit above that ceiling is an error;
+a lower limit is allowed.  If `pack.packSizeLimit` is unset or zero,
+automatic selection imposes no byte-size gate, and any explicit
+positive input limit is allowed.  An explicit `0` overrides the
+configuration and restores automatic selection.
+
+--max-packs=<n>::
+	Process at most `<n>` input packs per batch.  When more than
+	`<n>` aggregatable packs are present, they are split into several
+	evenly-sized batches, each producing one or more output packs.
+	This bounds how many packs any single `pack-objects` run (and any
+	later reader) must keep open at once, which matters on repositories
+	with such an extreme number of packs that per-process limits on
+	memory mappings or open files would otherwise be exceeded.  Note
+	that a pack in use costs roughly three such resources -- its
+	`.idx`, `.pack`, and `.rev` files -- so a batch of `<n>` packs
+	holds on the order of `3 * <n>` at peak; leave headroom below the
+	relevant limit when choosing `<n>`.  Batching happens on whole-pack
+	boundaries, keeping each delta and its base in the same input batch.
+	Defaults to the `pack.aggregateMaxPacks` configuration value, or
+	`10000` if unset.  A value of `0` disables the input-pack limit and
+	processes all input packs in one batch.
+
+SEE ALSO
+--------
+linkgit:git-repack[1], linkgit:git-pack-objects[1],
+linkgit:gitformat-pack[5]
+
+GIT
+---
+Part of the linkgit:git[1] suite

Documentation/git-pack-objects.adoc

@@ -143,6 +143,14 @@ options which imply `--revs`.
 	have an mtime older than `<approxidate>`. If unspecified (and
 	given `--cruft`), then no objects are eliminated.
 
+--mark-bad-deltas::
+	Write a `.baddeltas` marker file alongside each output pack.  The
+	marker signals that objects within the pack have not been fully
+	delta-searched against other objects within the same pack and
+	that future repacking should consider them.  Any deltas that do
+	exist within this pack can still be reused, however.
+	Incompatible with `--stdout`.
+
 --window=<n>::
 --depth=<n>::
 	These two options affect how the objects contained in
@@ -246,6 +254,13 @@ depth is 4095.
 	wholesale enforcement of a different compression level on the
 	packed data is desired.
 
+--prefer-reused-deltas::
+	Only meaningful with `--stdin-packs`.  When the same object is
+	present as a base in one included pack and a delta in another,
+	record the delta copy rather than the base.  This costs a
+	cheap object-header read per duplicate, and has no effect
+	under `--no-reuse-delta`.
+
 --compression=<n>::
 	Specifies compression level for newly-compressed data in the
 	generated pack.  If not specified,  pack compression level is

Documentation/git-repack.adoc

@@ -13,6 +13,7 @@ SYNOPSIS
 	[--window=<n>] [--depth=<n>] [--threads=<n>] [--keep-pack=<pack-name>]
 	[--write-midx[=<mode>]] [--name-hash-version=<n>] [--path-walk]
 	[--filter=<filter-spec>] [--drop-filtered [--dry-run]]
+	[--[no-]aggregate-once] [--[no-]aggregate-loop]
 
 DESCRIPTION
 -----------
@@ -274,6 +275,9 @@ picks the smallest set of packfiles such that as many of the larger
 packfiles (by count of objects contained in that pack) may be left
 intact.
 +
+Selection also accounts for packs forced into the roll-up by `.baddeltas`
+markers, including another pack if needed to preserve the progression.
++
 Unlike other repack modes, the set of objects to pack is determined
 uniquely by the set of packs being "rolled-up"; in other words, the
 packs determined to need to be combined in order to restore a geometric
@@ -337,6 +341,26 @@ created for any new pack(s) without disturbing the existing chain.
 	Pass the `--path-walk` option to the underlying `git pack-objects`
 	process. See linkgit:git-pack-objects[1] for full details.
 
+--aggregate-once::
+--no-aggregate-once::
+	Run linkgit:git-pack-aggregate[1] once before inspecting the
+	packs and loose objects to repack.  This can quickly reduce a
+	large number of files before the more thorough repack begins.
+	Packs named by `--keep-pack` are excluded from this preliminary pass.
+	Overrides the `repack.aggregateOnce` configuration variable.
+	Off by default.
+
+--aggregate-loop::
+--no-aggregate-loop::
+	Spawn a background linkgit:git-pack-aggregate[1] for the
+	duration of the main pack-objects run.  The aggregator rolls
+	up small packs and loose objects that arrive after
+	pack-objects has enumerated its inputs, preventing them from
+	accumulating in `objects/pack/` and slowing other Git
+	operations on busy servers.  Overrides the
+	`repack.aggregateLoop` configuration variable.  Off by
+	default.
+
 CONFIGURATION
 -------------
 
@@ -361,6 +385,7 @@ SEE ALSO
 --------
 linkgit:git-pack-objects[1]
 linkgit:git-prune-packed[1]
+linkgit:git-pack-aggregate[1]
 
 GIT
 ---

Documentation/gitformat-pack.adoc

@@ -12,6 +12,7 @@ SYNOPSIS
 $GIT_DIR/objects/pack/pack-*.{pack,idx}
 $GIT_DIR/objects/pack/pack-*.rev
 $GIT_DIR/objects/pack/pack-*.mtimes
+$GIT_DIR/objects/pack/pack-*.baddeltas
 $GIT_DIR/objects/pack/multi-pack-index
 
 DESCRIPTION
@@ -357,6 +358,35 @@ All 4-byte numbers are in network byte order.
     and a checksum of all of the above (each having length according
     to the specified hash function).
 
+== pack-*.baddeltas files
+
+The optional `.baddeltas` file is an empty marker sitting alongside a
+`pack-*.pack` (and its `.idx`).  It signals to `git pack-objects` that
+the delta layout of the pack should not be trusted: even when two
+objects appear together in the same pack and neither is stored as a
+delta, the next packing run should still call out to its delta search
+routine for the pair instead of assuming a prior pack-objects already
+considered (and rejected) the pair.
+
+This is intended for producers that intentionally skip delta search
+when writing a pack (for example, processes that bulk-import objects
+or aggregate multiple existing packs without recomputing deltas).
+Without this marker, the same-pack delta skip in `git pack-objects`
+would silently inherit those producers' lack of delta search into
+future repacks.
+
+The contents of the file are currently ignored.  Producers should
+write an empty file; consumers must tolerate (and ignore) any
+content.
+
+When `git repack` replaces a marked pack with an unmarked one, it
+removes the old marker even if the pack's name is unchanged.
+
+The marker only affects whether `git pack-objects` will attempt to
+compute new deltas for object pairs that share the marked pack.  It
+does not disable reuse of existing on-disk deltas, nor does it affect
+multi-pack-index, bitmap, or pack reuse for transfer.
+
 == multi-pack-index (MIDX) files have the following format:
 
 The multi-pack-index files refer to multiple pack-files and loose objects.

Documentation/meson.build

@@ -99,6 +99,7 @@ manpages = {
   'git-name-rev.adoc' : 1,
   'git-notes.adoc' : 1,
   'git-p4.adoc' : 1,
+  'git-pack-aggregate.adoc' : 1,
   'git-pack-objects.adoc' : 1,
   'git-pack-refs.adoc' : 1,
   'git-patch-id.adoc' : 1,