Multithreading and performance optimization

[oxalica]

It seems that we are only using a single thread when formatting tons of files. But this can be trivially parallelized. This should also ease the adoption of format-checking in git commit hook as mentioned in NixOS/nixpkgs#322537 (comment)

Eg: fd -e nix --exec-batch nixfmt (--exec-batch passes as much files to nixfmt as possible in a single exec)
It takes takes 1:49.27 (~110s) on my machine with only a single thread (~99% CPU) being occupied.

[infinisil]

We previously decided to deprecate the directory mode of nixfmt and instead encourage treefmt, for which I recently implemented parallel formatting.

So I don't think there's anything to do here, let's keep the complexity of this codebase at a minimum :)

[piegamesde]

I think that there is quite a bit of code performance which Nixfmt currently leaves on the table. Someone with knowledge of profiling Haskell applications would likely find quite a few low-hanging fruits.

Off the top of my head I can already think of https://github.com/NixOS/nixfmt/blob/master/src/Nixfmt/Predoc.hs#L178-L192 which has overall quadratic complexity, but by delaying computations into a single pass at the end this could be reduced to linear. (I didn't do it when I wrote the code because it requires a minor data types refactoring and I wanted to get stuff to work first.) Of course I can't tell how much this actually affects overall performance, but there are probably quite a few like this which add up.

[toastal]

You can also run it with GNU parallel: parallel "nixfmt {}" ::: *.nix

[infinisil]

I think we can close this actually, because we now deprecated recursive mode, and are recommending treefmt instead, which does run in parallel.