I've been working on improving our deployment processes as part of the High Performance Django Infrastructure project we're building out. One consistent pain point is our front-end build system, and particularly, npm installs. For a number of reasons, instead of updating the existing environment in place, we are creating clean builds on every deploy. Unfortunately, installing all the packages from npm on every build is prohibitively slow (despite npm's local download cache). One of npm's strengths -- lots of tiny composable modules -- means huge dependency chains and tons of HTTP requests to recreate node_modules from scratch.

When discussing this issue with Brian, he mentioned that the build times on Heroku were much better than what we were seeing on our own servers. I dug around in their Node.js buildpack and found that they cache the node_modules directory across builds. I implemented a similar technique with great results. Note: All the tests were run with an up-to-date ~/.npm cache.

Test #1: Production Builds

The large requirements for our production CSS are node-sass and autoprefixer. The JavaScript is made up of react and a bunch of complementary packages. It gets built with browserify and compressed with uglify.

Clean Production Build

A clean build on my laptop takes just over 30 seconds, downloading and installing all the dependencies from npm:

time NODE_ENV=production npm install

real    0m38.922s
user    0m32.528s
sys     0m11.246s

Cached Production Build

Here's the same build using (and recreating) a cache of the node_modules directory. It takes just over 10 seconds:

export NODE_ENV=production
time (tar xf ../nm_cache.tar && npm prune && npm install && tar cf ../nm_cache.tar node_modules)

real    0m11.082s
user    0m5.454s
sys     0m3.560s

That's a 3.5x improvement, shaving almost 30 seconds off our build time.

Test #2: Development Builds

Our development builds add a bunch of other packages including gulp, browsersync, and imagemin. A number of the packages contain compiled modules, slowing down the build time significantly.

Clean Development Build

A full clean build on my laptop takes about 3 minutes:

time npm install

real    2m59.663s
user    2m54.274s
sys     0m46.725s

Cached Development Build

Executing the same build from cache (and again, rebuilding the cache) takes less than a minute:

time (tar xf ../nm_cache.tar && npm prune && npm install && tar cf ../nm_cache.tar node_modules)

real    0m52.106s
user    0m24.376s
sys     0m14.601s

Again, we see almost a 3.5x improvement in the total time. This time, shaving a full 2 minutes off the build.

Fast builds are important to lower the barrier for testing/deployment and shorten the feedback loop for developers. We considered the alternative of "vendoring" the node_modules directory into the repository, but chose against it. For now, we prefer building them (from a cache) instead. I think there's still room for improvement here, however. If you have other techniques you use to make npm installs fast, we'd love to hear about them in the comments.