Primary image for Speeding Up NPM Installs

Speeding Up NPM Installs

I’ve been working on improving our deployment processes as part of the High Performance Django Infrastructure project we’re building out. One consistent pain point is our front-end build system, and particularly, npm installs. For a number of reasons, instead of updating the existing environment in place, we are creating clean builds on every deploy. Unfortunately, installing all the packages from npm on every build is prohibitively slow (despite npm’s local download cache). One of npm’s strengths – lots of tiny composable modules – means huge dependency chains and tons of HTTP requests to recreate node_modules from scratch.

When discussing this issue with Brian, he mentioned that the build times on Heroku were much better than what we were seeing on our own servers. I dug around in their Node.js buildpack and found that they cache the node_modules directory across builds. I implemented a similar technique with great results. Note: All the tests were run with an up-to-date ~/.npm cache.

Test #1: Production Builds

The large requirements for our production CSS are node-sass and autoprefixer. The JavaScript is made up of react and a bunch of complementary packages. It gets built with browserify and compressed with uglify.

Clean Production Build

A clean build on my laptop takes just over 30 seconds, downloading and installing all the dependencies from npm:

time NODE_ENV=production npm install

real	0m38.922s
user	0m32.528s
sys		0m11.246s

Cached Production Build

Here’s the same build using (and recreating) a cache of the node_modules directory. It takes just over 10 seconds:

export NODE_ENV=production
time (tar xf ../nm_cache.tar && npm prune && npm install && tar cf ../nm_cache.tar node_modules)

real	0m11.082s
user	0m5.454s
sys		0m3.560s

That’s a 3.5x improvement, shaving almost 30 seconds off our build time.

Test #2: Development Builds

Our development builds add a bunch of other packages including gulp, browsersync, and imagemin. A number of the packages contain compiled modules, slowing down the build time significantly.

Clean Development Build

A full clean build on my laptop takes about 3 minutes:

time npm install

real	2m59.663s
user	2m54.274s
sys		0m46.725s

Cached Development Build

Executing the same build from cache (and again, rebuilding the cache) takes less than a minute:

time (tar xf ../nm_cache.tar && npm prune && npm install && tar cf ../nm_cache.tar node_modules)

real	0m52.106s
user	0m24.376s
sys		0m14.601s

Again, we see almost a 3.5x improvement in the total time. This time, shaving a full 2 minutes off the build.

Fast builds are important to lower the barrier for testing/deployment and shorten the feedback loop for developers. We considered the alternative of “vendoring” the node_modules directory into the repository, but chose against it. For now, we prefer building them (from a cache) instead. I think there’s still room for improvement here, however. If you have other techniques you use to make npm installs fast, we’d love to hear about them in the comments.

Peter Baumgartner

About the author

Peter Baumgartner

Peter is the founder of Lincoln Loop, having built it up from a small freelance operation in 2007 to what it is today. He is constantly learning and is well-versed in many technical disciplines including devops, …