Lockfiles

src/blog/2024/lockfile-history-and-workflows-in-npm/README-Old.md

@@ -0,0 +1,200 @@
+---
+layout: article
+title: "Lockfile History and Workflows with npm"
+serif: true
+publishDate: "2024-01-26T23:43:11.233Z"
+---
+
+Lockfiles.
+Love them or hate them, they are the commonly accepted solution to the reproducible dependency set problem in Node.js.
+They work, but introduce trade-offs, are a source of contention, have underspecified workflows and life-cycles and are often misunderstood and misused.
+
+// TODO: a huge lockfile diff number
+
+The following post goes over some of the history of Lockfiles in Node.js, how they are/were intended to be used, some of the common misconceptions around how they work, and a few lesser known and discussed workflows to make more effective use of them, while also minimizing some of their drawbacks. I also offer workable alternatives that skip them completely for your consideration.
+
+[[toc]]
+
+## Node.JS Before Lockfiles
+
+[Node.js](https://nodejs.org/en) and [npm](https://docs.npmjs.com/cli/v10/commands/npm) didn't used to have a lockfile.
+You would define dependencies and their acceptable [semver](https://semver.npmjs.com) ranges in your `package.json`. Your dependencies would define their dependencies with their desired version ranges, so on and so forth and everything was 🌈groovy🌈.
+
+When I say 🌈groovy🌈, I mean non-deterministic, optimistically up to date, and somehow still working very reliably. And yes, there were issues with this arrangement, but they hadn't come to a head so to speak.
+
+When someone in your dependency tree accidentally published a breaking change in a non-breaking version range, everyone in the dependent network who was actively working would get a notification in the form of a runtime error the next time they installed, or ran tests in CI.
+Usually someone would quickly open an issue, and often include a patch! Because tooling overhead was still minimal at this point, and the use of [bespoke monorepos](https://github.com/lerna/lerna) (that make small contributions much more difficult and time consuming) hadn't proliferated far and wide yet, generating these patches was usually a minutes long affair. The patches would land, and the maintainer would publish a patch in a minute/day or two. Then everyone would get the patch without lifting a finger.
+
+It was really cool. It was really fun and this worked surprisingly well.
+
+Remember how I said npm defaults to optimistic updates within semver ranges?
+Well often, when this breaking change situation arose, you wouldn't even notice it, because someone got to it first.
+The next time you installed, you would just skip over the trouble and not even have to think about it.
+This ecosystem aimed to minimized disruption for most users and for a moment, totally realized that goal.
+It was really great!
+
+## When Node.js got Lockfiles
+
+Here is my lazy history of lockfiles getting adopted in Node.js as far as I can remember.
+
+Something something, Yehuda Katz, npm was now a hot new startup but was facing some scaling issues with the perceived performance of their CLI as people's project dependency trees began growing in size like metastasizing tumors. Some if the things happening at the time:
+
+- React was hot.
+- (LOTS of) Webpack plugins were hot.
+- ~~6to5~~ ~~pretend esm~~ Babel was hot.
+- Monorepos were heating up.
+- [`left-pad`](https://archive.is/b7MDy), the most over-hyped semi-malicious registry bomb that was [solved immediately afterwards with a simple policy change](https://blog.npmjs.org/post/141577284765/kik-left-pad-and-npm.html) [archive.ph/8fkpf](https://archive.ph/8fkpf), was still on peoples minds serving as the most basic reality check that they were, indeed, running other peoples code primarily in their own projects.
+
+Baby, we needed to populate a LOT of files into node_modules many times a a day.
+
+An emerging general “vibe” in wider JS ecosystem at the time was that that development momentum on the `npm` CLI was not moving as fast as it could and that npm had become much more insular and closed to ecosystem feedback since their incorporation.
+
+In October 2016, the [`yarn` package manager is released](https://classic.yarnpkg.com/blog/2016/10/11/introducing-yarn/) ([archive.ph/xRZ8H](https://archive.ph/xRZ8H)) coming out of the Facebook/React sphere of the ecosystem.
+Yarn ran faster, had a more polished DX and included cute spinners when you installed — Catnip in the dev tool ecosystem at the time.
+`yarn` also generated a new file in your projects called `yarn.lock`.
+`yarn.lock` was modeled after the ruby `bundler` lockfile [`Gemfile.lock`](https://bundler.io/guides/rationale.html).
+This was deemed a “Good Idea™” at the time because it seemed like it solved a problem for people feeling growing pains of large dependency trees constantly optimistically updating their semver ranges, and then breaking without any built-in tools in npm to fix the problem.
+
+It also arrived with the following directive: one [MUST commit `yarn.lock` to your git repo](https://classic.yarnpkg.com/blog/2016/11/24/lockfiles-for-all/) ([archive.ph/PTYoQ](https://archive.ph/PTYoQ)), apps and libraries inclusive, because this offered a solution to the “Works On My Machine” problem AKA the "In-range Breaking Change" problem so long as **EVERYONE** commits their lockfiles. (It failed to solve this problem generally, because the install step could still fail in a million different ways even with a `yarn.lock` present, and the solution often required, ironically, a full regeneration of the lockfile).
+
+npm responded initially with some blog posts, and finally with `npm@5` on May 25th, 2017 that introduced `package-lock.json`, an alternative lock file format to `yarn.lock` that was also recommended to be committed to your git repo.
+
+- [npm Blog Archive: Hello, Yarn!](https://blog.npmjs.org/post/151660845210/hello-yarn.html) ([archive.ph/6htXJ](https://archive.ph/6htXJ))
+- [npm@5, specifications, and our RFC process](https://blog.npmjs.org/post/154473364440/npm5-specifications-and-our-rfc-process.html) ([archive.ph/29uVE](https://archive.ph/29uVE))
+- [v5.0.0](https://blog.npmjs.org/post/161081169345/v500.html) ([archive.ph/pEQiV](https://archive.ph/pEQiV))
+
+npm played performance catch-up with yarn, introduced an RFC process that was sort of followed to some success over the years, introduced a "standardized" `package-lock.json` that would fail to gain uptake by any of the other alternative package managers, and generally the major performance and capability gap was closed between yarn and npm in less than a year.
+
+(I have purposely ignored `pnpm` during this discussion, because it wasn't widely used at the time, and frankly, not really involved in this feud, though it has certainly taken off lately and introduces its own `pnpm-lock.yaml`. Maybe someone can introduce `yet-another-lock.toml` next!).
+
+## The Lockfile “commit to git” Happy Path
+
+The accepted workflow with lockfiles varies from every team I've been on, and include many subtleties and problems that get glossed over in the lockfile discussions I've witnessed.
+
+The most commonly understood workflow patterns I've observed generally follows these rules:
+
+- A node_module based project, in a `git` repo with a `package.json`, generates a `package-lock.json`/`yarn.lock`/`pnpm-lock.yaml` on someone's machine.
+- This lockfile is committed to the repo.
+    - (Bonus points if you commit more than one lock file format with skewed dependency sets.)
+- Packages are added with `npm`/`yarn`/`pnpm` (package manager), and the lockfile and `package.json` are updated accordingly to the date, time and network speed conditions. (Did you know network speeds can determine how node_module dependencies resolve? 💫)
+- These machine generated lockfile changes come along for the ride in a PR. They aren't typically reviewed closely by a human.
+- Everyone else with conflicting lockfile changes rebase and let the package manager resolve the conflict somehow.
+- Flapping lock file changes are discarded from commits or continue to flap between developer machines and PRs.
+- Tools like Dependabot or Renovate send automated patches to top level dependencies and resultant modifications to the lockfile. This generates lockfiles in a way that collects non-commutative state that's nearly impossible to recreate.
+- Eventually, some transitive dependency breaks, and you start getting so many version and depreciation warnings on install, some brave soul on the team declares lockfile bankruptcy and re-generates the lockfile by deleting it, and regenerating it from scratch. This usually works, but often results in a landmine field of bugs in your codebase that haven't revealed themselves before due to building around bugs in your dependency tree.
+- In CI and deployment, `npm ci` or equivalent is used, resulting in a deterministic rebuild of your dependency tree prior to tests and deployment.
+- During development, some developers choose to use `npm ci` or equivalent, however, if any sort of host/arch specific dependencies sneak into the lockfile and no longer work on a wider set machines, the developer will often have to drop to the `npm i` non-deterministic install that tends to fix these problems. Fingers crossed the updated lockfile after this step won't break the host that generated the lockfile last.
+
+## Why the typical “commit to git” lockfile workflow falls over
+
+The "commit to git" workflow is applied in both "project" based repos, and "library/module" repos but has many drawbacks and points of non-determinism:
+
+- Mutating lockfiles that are patched over time introduce ["dependency divergence"](https://socket.dev/blog/dependency-divergence). There isn't a great solution for this other than full lockfile regeneration over time. In a team context, who and when is doing this? I've seen no clear consensus on this ever, and it doesn't fit well into this workflow.
+- Mutating lockfile generation isn't deterministic.
+- Regenerating lockfiles isn't event deterministic. Package manager version drift and hardware differences between developers, date, time and network conditions introduces subtle differences in the resulting lockfiles.
+- There is an under-defined process for re-generating lockfiles. Either way, its disruptive and creates overhead with tools with git. This is a reality that wastes time and creates needless make-work. These are real concerns in a business context.
+- Its difficult to force consistent use of the `npm ci` (or variant of this command) between developer machines, and could be argued that is not event the correct approach in general.
+It certainly **wasn't** part of the original workflow that npm designed around.
+
+When the "commit to git" workflow is applied to module or library code, the situation gets even worse. Remember, lockfiles never get published to npm or used by downstream dependents, so when used in the context of a module, all you are doing is creating environment skew between your development environments and your users. Don't just take it from me, take it from [sindresorhus](https://github.com/sindresorhus/ama/issues/479#issuecomment-310661514), one of the top publishers of popular modules on npm.
+
+## What are Lockfiles good for then?
+
+- Lockfiles are good for reproducing one dependency set generated on machine A, at a later date, on machine B, when used with the appropriate install command (`npm ci` or similar, typically not the default install command).
+- The process of lockfile generation is non-deterministic.
+- They can increase install speed which is important for obscenely large dependency trees, usually found in monorepo variants.
+
+## When are Lockfiles bad?
+
+- When mutated over time as the lockfile is exposed to different developer environments and package manager update commands and dependency bot patches, you accumulate non-deterministic, non-commutative state in your lockfile and the value and reliability assumptions of the lockfile artifact drops considerably.
+- When used in a module or library development context, the only effect it has is to introduce environment skew between the developer and the users of the library. All assumed benefits of the lockfile become much more superficial and hazardous to the reliability of automated test results, especially as time marches on.
+- When used with very large dependency sets, they can help hide the intrinsic nature of the instability and unreliability of of that set, but doesn't fundamentally address the underlying issue causing that unreliability by artificially forcing stability.
+
+## Solution 1: `.gitignore` lockfiles
+
+A common alternative to the default "commit to git" lockfile workflow is the "no lockfile" approach.
+This approach is commonly found in module code, because developing module code **with** a lockfile unhelpfully introduces the problem of running all future CI test results against the last date at which the lockfile was regenerated and committed.
+This is a problem because locking your tests to a specific date **can and will** hide bugs that your users will encounter when installing.
+
+To take this approach you simple add lockfiles to your `.gitignore`:
+
+```gitignore
+yarn.lock
+package-lock.json
+pnpm-lock.yaml
+```
+
+You can optionally disable lockfile generation for the repo by adding an `.npmrc` file to the repo with the following setting:
+
+```
+package-lock=false
+```
+
+In this workflow, I recommend the `.gitignore` approach, and omitting `.npmrc`.
+That way developers can choose locally if they want to generate lockfiles during development or not.
+This approach effectively returns to a pre-lockfile workflow and is the most realistic arrangement for module development.
+It also encourages fresh, disposable lockfile generation over mutation, which more closely matches the reality your module code will operate in.
+
+### What about lockfiles in "projects" or "apps"?
+
+There is a stronger argument to be made with respect to the "lockfiles committed to git" approach when working on "apps" or projects that don't get consumed as a module.
+Lockfiles in repos can provide useful reproducibility between dev machines during development in this context.
+Additionally, they offer a pragmatic guard against having to deal with dependency issues too frequently or at inopportune times.
+
+But they also come with the trade-offs already discussed: date-pining to when they were generated and non-deterministic state if they are mutated.
+
+Here's a few thoughts on this:
+
+- A good goal should be to engineer your project and its dependencies such that it would work most of the year in a dev environment *as if there was no lockfile committed*. If this isn't possible, your dependency foundation will be a major source of instability, lockfile or none.
+- You should feel comfortable and frequently fully regenerate your lockfiles.
+- You should avoid mutating your lockfile over too long of a period.
+- You should use your lockfiles strategically for reproducibility on tests and deploys.
+- Consider that fixing one isolated dependency issue a week will take less time than dealing with 4 simultaneous dependency issues once a month and calibrate your regeneration schedule around this.
+
+All that said, it is still possible to
+
+## Solution 2: Strategic Lockfiles Snapshots
+
+Lockfiles are useful artifacts for generating a deterministic reproduction of fresh installations performed in the past.
+If you generate fresh lockfiles at strategic moments, such as CI test runs, or deploy pipelines, it can be helpful to archive them in case you ever need to go back and reproduce for debugging purposes.
+
+GitHub actions makes this very simple.
+To strategically archive lockfiles from test runs, you can add a step similar this to any Action run after a `npm i` or equivalent is run.
+
+```yaml
+    - run: npm i
+    - name: Archive lockfile
+      uses: actions/upload-artifact@v4
+      with:
+        name: pacakge-lock.json
+        path: package-lock.json
+```
+
+If in the event you need the lockfile to recreate the dependency set of a failing test or working or broken deploy, you simply download the lockfile and use it, temporarily commit it etc.
+
+## Solution 3: Use the Date Luke!
+
+`npm` now has a `--before` config flag that can be used to perform a dependency tree resolution as if it were run on the provided date.
+
+```bash
+npm i --before "2021-01-27T19:04:22.125Z"
+```
+
+The primary "purpose" of a lockfile is to deterministically re-assemble a dependency tree from a previous install.
+
+Remember, though, that npm is a (mostly) immutable package registry. This means that re-resolving your dependency tree at a given date will *also be (mostly) deterministic* (save for oddball undefined behaviors like network speeds, host conditions and various optional and peer dependency conditions).
+
+Save the implementation details and the folktales surrounding them, lockfiles are most useful at reconstructing a working dependency tree from the past, when a transitive dependency that you don't control is breaking something. The `--before` flag also provides this functionality.
+
+### Pros of `--before`
+
+- Re-creates the cleanest tree resolution at a given date time (avoids state, gives you what you actually want)
+- Nearly deterministic
+- Less diff noise
+- Simple to understand and audit and update
+
+### Cons of `--before`
+
+- Nobody knows about it.
+- No `npm ci` style speedup (if you aren't committing lockfiles)
+- Still subject to non-deterministic and undefined behavior of the resolution step of the used package manager.