Lincoln Loop

Simple & Easy Deployment with Fabric and Virtualenv

In the process of prepping Gondola CMS for public consumption, we’ve grown from having a developer (me), to having a development team. One pain point that quickly arose was the amount of time it took for new developers to setup a local development environment. In addition to our source code, the project depends on nearly a dozen other Django apps and Python packages. Initially, we simply tracked the requirements in a text file, but it required a lot of manual work to get them downloaded and installed.

Necessity is the Mother of Invention

Of course, this problem has been tackled many, many times before¹. As I looked into what was out there, my first thought was that most of the existing options were far too complex for our needs. If I didn’t grok a system after clicking around the docs for 10 minutes or so, I moved on. The tool I eventually decided on was Fabric.

I was attracted to Fabric, primarily because I found it simple and easy to understand. Unfortunately, I didn’t find a lot of information on using it for local buildouts, so I forged ahead building my own.

fab bootstrap

Our needs were simple. I wanted a script that would:

1. Setup our standard directory structure
2. Download source code from a number of disparate systems, properly handling branches and version requirements
3. Load the source into a virtualenv

The results of a few hours of hacking are available as lincoln-loop-deploy. Our local buildout is now a simple process:

1. Get requirements: easy_install Fabric virtualenv
2. Checkout our buildout repository (only contains 2 files fabfile.py and fabreqs.py)
3. Build project: fab bootstrap
4. Activate the virtualenv: source ./ve/bin/activate

My goal is to keep the fabfile.py totally generic so it can be used across all our projects. While it serves my purposes for now, we’ll likely be extending it in the future to allow for remote deployment, source updates, and additional version control systems. Yann and I also discussed an option to work without the virtualenv requirement for those who haven’t seen the light yet don’t want to use it.

Do you have a better way of doing things? We’d love to hear what you have to say. Patches are always welcome too.

¹ Ian Bicking’s summary of build systems

Comments

• December 8, 2008 at 1:28 p.m. #
  Rock Howard commented:

  The fab bootstrap worked fine for me.

  I have used fabric for various uploading tasks, but only recently started working on some automation for client deployments. Some of this stuff will be quite helpful!

• December 9, 2008 at 12:47 a.m. #
  David Pombar added:

  Hello. I’m learning about python and django and i’m very interested in help you with Gondola CMS.
  If you want an spanish translation for the app, write me.

  Congratulations on you work

• December 9, 2008 at 3:20 a.m. #
  Dan Hilton chimed in with:

  I’d also have a look at Dan Fair’s article on using django recipe for zc.buildout that does very similar things:
  http://www.stereoplex.com/two-voices/a-django-development-environment-with-zc-buildout

  HTH
  Dan

• December 9, 2008 at 3:08 p.m. #
  Arthur chimed in with:

  It would be interesting to see if pip (http://pypi.python.org/pypi/pip) could be used for this, since it supports creating and installing from “Requirements Files”.

• December 10, 2008 at 9:59 a.m. #
  Alex piped up:

  Nice timing – I just started to think about building something that whould’ve looked a bit like your solution ;)
  It would be nice to be able to update an existing deployment just by changing the requirements file and running fab update or something like that…

• December 11, 2008 at 2:06 p.m. #
  Jökull commented:

  I think this is an awesome solution. We’ve taken your fabfile.py and made some improvements. We’ll share them as soon as we’re confident they are actually improvements.

  Alex brings up an interesting point. Could we handle new requirements in a nice way? Maybe “fab update”.

• December 11, 2008 at 2:15 p.m. #
  Peter Baumgartner mentioned:

  Yep, `fab update` needs to be in there. If any of you feel like tackling it before I do, it should reparse fabreqs.py, updating any packages that aren’t tied to specific version number and installing new packages that have been added.

• December 13, 2008 at 9:07 a.m. #
  Marlun commented:

  I got curious about “Yann and I also discussed an option to work without the virtualenv requirement for those who haven’t seen the light yet don’t want to use it.”

  Is there something wrong about virtualenv? I’m new to python and just read a little about it and it seemed like a good idea?

• March 8, 2009 at 2:21 p.m. #
  Andrew Ingram commented:

  I modified my version of the fabfile to detect the version of python in use rather than using a hardcoded python2.5.

  I also played around with a fresh build of python (2.6) to work out the minimum baseline Python install for the virtualenv to inherit:

  - PIL
  – PyCrypto (from pypi)
  – Virtualenv
  – fabric
  – setuptools
  – mercurial
  – readline (mac)
  – mysqldb (if you need it)

  These things are either need to run virtualenv and fabric or can’t be installed easily under a virtualenv.

  After doing this, I now have a requirements file that allows me to build a complete working environment for my sites (I put my site code in the requirements file as well) with just one command. Now I just need to get it playing nice with pydev and mod_wsgi (shouldn’t be too challenging).

• February 24, 2010 at 9:15 a.m. #
  klaus mentioned:

  great stuff .. next to learn for me is this from

  http://github.com/jacobian/django-deployment-workshop

  a lot off…

• March 23, 2010 at 10:01 a.m. #
  Matt Wilson added:

  This is my release process:

  1. fabric connects to the production box and uses supervisord to shut down our python webapp and offline scripts.

  2. fabric activates a virtualenv, changes into a particular directory and then runs a git pull to grab the most recent code.

  3. fabric looks for a xxx.sql file with a particular name for xxx, and if it exists, then fabric runs that script. I use xxx.sql to alter the database schema.

  4. fabric looks for a xxx.py file, and if it exists, it runs that. I use xxx.py to handle any one-time data transformations that are required for this release.

  5. fabric tells supervisord to start up the webapp and the scripts.

  Recently we set up a new production server from scratch, and it was a big hassle installing and configuring postgres, apache, postfix, etc, and then creating all the users and groups I want.

  After that, I spent the weekend studying puppet, but it turned me off. I am considering using fabric to “bootstrap” a new server from the very beginning.

Got something to say?

Name
Email
Website
Comment