I've been working on improving our deployment processes as part of the High Performance Django Infrastructure project we're building out. One consistent pain point is our front-end build system, and particularly,
npm installs. For a number of reasons, instead of updating the existing environment in place, we are creating clean builds on every deploy. Unfortunately, installing all the packages from npm on every build is prohibitively slow (despite npm's local download cache). One of npm's strengths -- lots of tiny composable modules -- means huge dependency chains and tons of HTTP requests to recreate
node_modules from scratch.
When discussing this issue with Brian, he mentioned that the build times on Heroku were much better than what we were seeing on our own servers. I dug around in their Node.js buildpack and found that they cache the
node_modules directory across builds. I implemented a similar technique with great results. Note: All the tests were run with an up-to-date
Test #1: Production Builds
The large requirements for our production CSS are
react and a bunch of complementary packages. It gets built with
browserify and compressed with
Clean Production Build
A clean build on my laptop takes just over 30 seconds, downloading and installing all the dependencies from npm:
time NODE_ENV=production npm install ... real 0m38.922s user 0m32.528s sys 0m11.246s
Cached Production Build
Here's the same build using (and recreating) a cache of the
node_modules directory. It takes just over 10 seconds:
export NODE_ENV=production time (tar xf ../nm_cache.tar && npm prune && npm install && tar cf ../nm_cache.tar node_modules) real 0m11.082s user 0m5.454s sys 0m3.560s
That's a 3.5x improvement, shaving almost 30 seconds off our build time.
Test #2: Development Builds
Our development builds add a bunch of other packages including
imagemin. A number of the packages contain compiled modules, slowing down the build time significantly.
Clean Development Build
A full clean build on my laptop takes about 3 minutes:
time npm install ... real 2m59.663s user 2m54.274s sys 0m46.725s
Cached Development Build
Executing the same build from cache (and again, rebuilding the cache) takes less than a minute:
time (tar xf ../nm_cache.tar && npm prune && npm install && tar cf ../nm_cache.tar node_modules) real 0m52.106s user 0m24.376s sys 0m14.601s
Again, we see almost a 3.5x improvement in the total time. This time, shaving a full 2 minutes off the build.
Fast builds are important to lower the barrier for testing/deployment and shorten the feedback loop for developers. We considered the alternative of "vendoring" the
node_modules directory into the repository, but chose against it. For now, we prefer building them (from a cache) instead. I think there's still room for improvement here, however. If you have other techniques you use to make
npm installs fast, we'd love to hear about them in the comments.