Repository navigation
Conversation
Git checks out files one at a time unless `checkout.workers` says otherwise. On Windows, replacing a file in the worktree (unlink, create, write, close) costs about 250 microseconds even on a Dev Drive (ReFS), and several times that on NTFS with real-time antivirus scanning. That cost overlaps well across processes, so parallel checkout makes a 2,000-file checkout 28-38% faster (see below). Since 373d334 (entry: flush fscache after creating directories and writing files) and d7b8fab (parallel-checkout: limit worker count to what poll() can wait on), parallel checkout works with the FSCache and with any worker count. Starting a worker is not free on Windows either, so the upstream threshold of 100 files is too low. With four workers and that threshold, switching between branches that differ in 100 files took 33ms longer than the 138ms of a sequential checkout. Default to four workers on Windows, and use them only for checkouts of at least 500 files. Both settings keep working as before; this only changes what Git does when they are not set. Switching between branches of a 40,000-file repository that differ in N files, Windows 11 on a 32-thread desktop with an NVMe drive. Medians; time saved is the median paired difference with its 95% bootstrap interval, over 12-15 interleaved runs: Dev Drive (ReFS) NTFS, Defender real-time N files before saved before saved 250 145ms (sequential) 529ms (sequential) 500 187ms 0ms [-36, 7] 671ms 123ms [95, 148] 1000 287ms 40ms [27, 66] 1024ms 282ms [219, 308] 2000 461ms 129ms [61, 180] 1668ms 633ms [603, 645] `git worktree add` of all 40,000 files went from 7.07s to 4.51s (2.56s saved [2.00, 3.13]) on the Dev Drive. Below the threshold the files are still written sequentially, and those checkouts were not slower (250 files: 10ms faster [4, 16] on the Dev Drive, 24ms faster [5, 63] on NTFS). Eight workers were no better than four at 2,000 files, and `checkout.workers=0` (one per logical CPU, 32 here) was no faster than sequential checkout. The defaults are compile-time macros, overridable per platform, in the same way as FSYNC_METHOD_DEFAULT and PROTECT_NTFS_DEFAULT, so other platforms are unchanged. A compiled default covers every Git for Windows distribution in one place. Shipping it as configuration, the way core.fscache is shipped, would mean adding it separately to the installer, the MSI, PortableGit and MinGit, and to any system gitconfig that a tool bundling Git writes for itself. Add a test that checks the default worker count on both sides of the threshold. Signed-off-by: Alexandre Zollinger Chohfi <alzollin@microsoft.com>
|
To be honest, I'm a little bit concerned about this patch because of the unintended consequences. Yes, it accelerates the checkout, but also it increases the memory requirements. For context, when I check out the Git for Windows SDK in CI, I don't use four workers, even if the runner agents have two CPU cores available. No. I use 56 workers. Yes. Fifty-six. In my tests this struck the best balance between performance and robustness. The checkout was even faster when I increased that number to, say, 64 workers. The problem, however, was that frequently these checkouts would run out of memory. And that is the exact problem here. Unfortunately, due to its attribute handling, it is not possible to make Git's checkout logic multi-threaded. Ask me how I know. Therefore, the only chance we have is to make it multi-process. That, however, multiplies the memory requirements. It's somewhat eased by the fact that at least the packfiles are memory-mapped, so those memory pages can be shared between processes. In theory. I did not actually verify that they are shared, but in theory they could be. However, the packfiles are packfiles, but unpacked objects are unpacked objects, and they occupy a lot of memory. And I mean a lot. Therefore, if you increase the default number of workers, you immediately also put a potentially enormous strain on the memory requirements of checkouts, in particular of large repositories, the very scenario with which this PR is supposed to help. |
|
Actually this goal of increasing performance is related with the following pull request, maybe it could solve your problem. I was even testing parallel cloning on ReFS with multiple workers and it should not be that memory demanding because those workers mainly wait for system call of cloning to be finished and not holding entire buffer for copying. Buffer is allocated only for last unallocated block of file. So I will retest how the implementation in this branch works with parallel worktree add in NTFS/RFS old and new implementation. |
This branch has not been deployed
3 participants
Git checks out files one at a time unless
checkout.workerssays otherwise. On Windows,replacing a file in the worktree (unlink, create, write, close) costs about 250 microseconds
even on a Dev Drive (ReFS), and several times that on NTFS with real-time antivirus scanning.
That cost overlaps well across processes, so parallel checkout makes a 2,000-file checkout
28-38% faster (see below). Since 373d334 (entry: flush fscache after creating directories and
writing files, #6250) and d7b8fab (parallel-checkout: limit worker count to what poll() can
wait on, #6395), parallel checkout works with the FSCache and with any worker count.
Starting a worker is not free on Windows either, so the upstream threshold of 100 files is too
low. With four workers and that threshold, switching between branches that differ in 100 files
took 33ms longer than the 138ms of a sequential checkout.
Default to four workers on Windows, and use them only for checkouts of at least 500 files. Both
settings keep working as before; this only changes what Git does when they are not set.
Switching between branches of a 40,000-file repository that differ in N files, Windows 11 on a
32-thread desktop with an NVMe drive. Medians; time saved is the median paired difference with
its 95% bootstrap interval, over 12-15 interleaved runs:
git worktree addof all 40,000 files went from 7.07s to 4.51s (2.56s saved [2.00, 3.13]) onthe Dev Drive.
On a real repository, nodejs/node at v26.10.0 (51,849 files), with the same two values passed with
-cto 2.55.0.windows.5 (medians of 5 interleaved passes; saved, with paired 95% intervals):git worktree addgit cloneBelow the threshold the files are still written sequentially, and those checkouts were not
slower (250 files: 10ms faster [4, 16] on the Dev Drive, 24ms faster [5, 63] on NTFS). Eight
workers were no better than four at 2,000 files, and
checkout.workers=0(one per logical CPU,32 here) was no faster than sequential checkout.
The defaults are compile-time macros, overridable per platform, in the same way as
FSYNC_METHOD_DEFAULTandPROTECT_NTFS_DEFAULT, so other platforms are unchanged. A compileddefault covers every Git for Windows distribution in one place. Shipping it as configuration,
the way
core.fscacheis shipped, would mean adding it separately to the installer, the MSI,PortableGit and MinGit, and to any system gitconfig that a tool bundling Git writes for itself.
Add a test that checks the default worker count on both sides of the threshold.
Notes for reviewers
-c. Thisbranch is on
main.not affect four workers.
git.exespawns four workers for a 600-file switchand none for a 200-file one.
make DEVELOPER=1using the Git for Windows SDK.t20*,t24*,t1091-sparse-checkout-builtin,t5601-cloneandt7102-resetpass (49 scripts, 1,058 tests). The new test is included.
checkout.workerscauses checkouts of multiple files to fail #4331 (failures withcheckout.workersand the FSCache) looks like the bug entry: flush fscache after creating directories and writing files #6250fixed; it was not retested here. worktree: use ReFS block cloning for unchanged files #6424 (ReFS block cloning for
worktree add) iscomplementary: cloned files skip the write, and the rest still benefit from the workers.
checkout.workersdocumentation alreadywarns about the latter);
online_cpus()? With 4 workers on a 2-core machine,the overlap is mostly I/O wait, so it may still help, but that is untested.