I’ve been working on improving our deployment processes as part of the High Performance Django Infrastructure project we’re building out. One consistent pain point is our front-end build system, and particularly, npm installs. For a number of reasons, instead of updating the existing environment in place, we are creating clean builds on every deploy. Unfortunately, installing all the packages from npm on every build is prohibitively slow (despite npm’s local download cache). One of npm’s strengths – lots of tiny composable modules – means huge dependency chains and tons of HTTP requests to recreate node_modules from scratch.
When discussing this issue with Brian, he mentioned that the build times on Heroku were much better than what we were seeing on our own servers. I dug around in their Node.js buildpack and found that they cache the node_modules directory across builds. I implemented a similar technique with great results. Note: All the tests were run with an up-to-date ~/.npm cache.
Test #1: Production Builds
The large requirements for our production CSS are node-sass and autoprefixer. The JavaScript is made up of react and a bunch of complementary packages. It gets built with browserify and compressed with uglify.
Clean Production Build
A clean build on my laptop takes just over 30 seconds, downloading and installing all the dependencies from npm:
time NODE_ENV=production npm install
...
real 0m38.922s
user 0m32.528s
sys 0m11.246s
Cached Production Build
Here’s the same build using (and recreating) a cache of the node_modules directory. It takes just over 10 seconds:
export NODE_ENV=production
time (tar xf ../nm_cache.tar && npm prune && npm install && tar cf ../nm_cache.tar node_modules)
real 0m11.082s
user 0m5.454s
sys 0m3.560s
That’s a 3.5x improvement, shaving almost 30 seconds off our build time.
Test #2: Development Builds
Our development builds add a bunch of other packages including gulp, browsersync, and imagemin. A number of the packages contain compiled modules, slowing down the build time significantly.
Clean Development Build
A full clean build on my laptop takes about 3 minutes:
time npm install
...
real 2m59.663s
user 2m54.274s
sys 0m46.725s
Cached Development Build
Executing the same build from cache (and again, rebuilding the cache) takes less than a minute:
time (tar xf ../nm_cache.tar && npm prune && npm install && tar cf ../nm_cache.tar node_modules)
real 0m52.106s
user 0m24.376s
sys 0m14.601s
Again, we see almost a 3.5x improvement in the total time. This time, shaving a full 2 minutes off the build.
Fast builds are important to lower the barrier for testing/deployment and shorten the feedback loop for developers. We considered the alternative of “vendoring” the node_modules directory into the repository, but chose against it. For now, we prefer building them (from a cache) instead. I think there’s still room for improvement here, however. If you have other techniques you use to make npm installs fast, we’d love to hear about them in the comments.
