Yarn NPM Caching Gotchas

The cfgov-refresh app caches its NPM dependencies inside of the repo using a custom Yarn config. During deploys and CI runs, it prevents Yarn from requesting packages from the network and makes it load dependencies exclusively from the local cache folder. This saves a lot of network bandwidth and allows them to run builds more frequently and have a portable package cache that goes wherever the repo is cloned, with no additional steps to set up.

It can potentially make things a bit tricky when adding or updating dependencies, however. Here are a few common scenarios you'll encounter and how to deal with them:

I want to add a new NPM dependency to my app

The .yarnrc file in the project root contains 2 lines that can make installing new dependencies fail:

--install.pure-lockfile true # This prevents Yarn from saving a new yarn.lock file with updated dependency info when "yarn add" runs
--install.offline true # This prevents Yarn from loading dependencies from the network

The easiest way to get around these issues is to just temporarily comment out the above lines and run your yarn add command. Edit the .yarnrc to look like this, and save it:

yarn-offline-mirror "./npm-packages-offline-cache"
yarn-offline-mirror-pruning true
#--install.pure-lockfile true
#--install.offline true

Run yarn add my-dependency. Make sure you run this command in the appropriate folder. If you just need the dependency in the app you're working on, then make sure you run it in cfgov/unprocessed/apps/MY_APP as opposed to the project root.

When Yarn is finished, remove the comments you placed in .yarnrc and save it again. You should now have an updated package.json and yarn.lock that you can add, commit, and push. Yarn will have also updated the package cache folder, called npm-packages-offline-cache. You'll want to add that folder's changes and commit them as well.

Yarn said it couldn't find a particular package and it caused my Jenkins build to fail, but everything works fine locally!

You might have a package cached in the place where Yarn usually stores things, somewhere in a hidden folder in your home directory. The best way to make sure your npm-packages-offline-cache folder has everything it needs in order to build the app is to completely rebuild the cache.

First, you'll need to edit the .yarnrc file in the project root so that it's allowed to download packages over the network. See the previous section for which lines to comment out. Once that's done, it's time to reinstall everything:

# First, clear your yarn cache:
yarn cache clean

# Next, wipe ALL of the node_modules folders in the project:
rm -rf node_modules cfgov/unprocessed/apps/*/node_modules

# Delete the yarn.lock file in the project root, as well as the ones in the apps you've been working on:
rm yarn.lock cfgov/unprocessed/apps/my-money-calendar/yarn.lock # that last one is just an example

# Reinstall all the things:
yarn

# Once Yarn is done reinstalling packages, add and commit all the changes:
git add -A npm-packages-offline-cache yarn.lock cfgov/unprocessed/apps/*/yarn.lock
git commit -m "Rebuild offline yarn cache"

# Push back up to GitHub
git push

You will also need to push your branch to upstream in order for CFPB Jenkins to be able to pull the changes. Don't forget to uncomment the last 2 lines in .yarnrc when you're done, and make sure you don't commit that file with those lines commented out.

Re-run the build in Jenkins and cross your fingers.