Lincoln Loop

Easy Fabric Deployment, Part 1: Git/Mercurial and SSH

We’re firm believers in the practices described by the Continuous Integration method of software engineering. Among those are:

• Maintain a code repository
• Automate the build
• Automate deployment

We use git for our code repositories and Fabric to automate our build/deployment process. The tiny bit of overhead it take to write out a Fabric script pays off very quickly against the tedium and error-prone practice of manually building/deploying. In building our “fabfile”, we encountered a couple of issues that took a little head-scratching to work out.

Git and SSH Keys

Git, like Mercurial and others, confirms your credentials via an private/public key pair when used over SSH. We use gitosis to manage our private repositories, so managing everyone’s keys isn’t much of an issue. The problem comes when developers need to start pulling the repository on different machines. Our developers all push to a central “origin” and our development server runs a clone of that repository.

We want every push to the master branch to automate the build and deployment. The first thought is to manage this with post-commit hooks, but that raises some security concerns. I want the git user that owns all the repositories to have as few permissions as possible. It’s job is to act as a gatekeeper for our repositories, not handle deployment across any number of servers.

The simple workaround is a Fabric script that allows our developers to use one command to push, build and deploy. Our script does the following:

1. Push local git repository to origin
2. Pull development server git repository from origin
3. Install any 3rd party libraries as specified in the pip requirements file
4. Run any necessary migrations
5. Run syncdb
6. Restart the WSGI daemon

For the most part, those are trivial commands. The one that gets us into trouble is pulling to our development server(s). Our developer’s local SSH keys aren’t available server-side. Having to manage a whole new set of SSH keys or require them to copy their private keys to the server was not desirable.

The good news is that, ssh provides a way to forward your authentication agent connection. From the ssh man page:

     -A      Enables forwarding of the authentication agent connection.  This
             can also be specified on a per-host basis in a configuration
             file.

Agent forwarding should be enabled with caution.  Users with the

             ability to bypass file permissions on the remote host (for the

             agent’s Unix-domain socket) can access the local agent through

             the forwarded connection.  An attacker cannot obtain key material

             from the agent, however they can perform operations on the keys

             that enable them to authenticate using the identities loaded into

             the agent.

The bad news is that neither Fabric, nor paramiko (the guts of Fabric’s SSH implementation) expose this switch. The fix is simply to use your local shell to run any commands over SSH. We created a Fabric function that looks like this:

env.hosts = ['our.development.server']
# ...
def sshagent_run(cmd):
    """
    Helper function.
    Runs a command with SSH agent forwarding enabled.
    
    Note:: Fabric (and paramiko) can't forward your SSH agent. 
    This helper uses your system's ssh to do so.
    """

    for h in env.hosts:
        try:
            # catch the port number to pass to ssh
            host, port = h.split(':')
            local('ssh -p %s -A %s "%s"' % (port, host, cmd))
        except ValueError:
            local('ssh -A %s "%s"' % (h, cmd))

Now anywhere you would have called run() in Fabric, you can call sshagent_run() and it works as expected.

The Authentication Agent

There is one more step you may need to take on your local computer for this technique to work. You’ll need to ensure that your identity has been added to the authentication agent. As far as I can tell, current versions of Ubuntu do this automatically. For other operating systems (OS X and other Linux variants), you may need to do this manually. You can check by running ssh-add -l from the command line. If you get a line back containing the path to your private key, you can skip this step. If you don’t, you’ll want to run ssh-add to add your key to the list. Since this isn’t a persistent setting, you’ll probably want to add it to your ~/.bashrc or someplace else it can run on login.

With your Fabric script and authentication agent setup, you can run automated build and deployments across any number of servers without ever worrying about which machine you are on and where your SSH key lives.

Have you solved this issue in a different way? Leave us a comment, we’d love to hear your thoughts.

Comments

• September 22, 2009 at 4:50 p.m. #
  vinilios added:

  Thanks for the tip,

  deployment using fabric and pip/venv looks like hot topic at the moment on djangonauts blogs. In our company we deploy our apps using pip and fabric but we usually write custom fabric script for each project since deployment procedure differs each time.

  A point that might block the deployment procedure you refer is the requirement of pip for installed vcs libraries/packages/binaries when external packages (-e) are inside pip requirement files. Pip provide an alternative way (package bundles) to this which we currently use to deploy applications in strict environments (e.g. shared hosting). Bundles cover our needs in most cases but they are not so flexible (frequently package upgrades/installations require much time to create the bundle and deploy it).

• October 17, 2009 at 6:06 a.m. #
  Polprav mentioned:

  Hello from Russia!
  Can I quote a post in your blog with the link to you?

• October 18, 2009 at 4:40 p.m. #
  Peter Baumgartner added:

  Polprav, you may not “re-post” or copy the full post verbatim, but feel free to quote a few sentences with a link back to the original source article.

  Thanks for reading!

• May 10, 2010 at 7:41 a.m. #
  WK responded:

  Thanks! saved my day!

  I also had to add env.user to it because my deployment user is different that my local user.

  for h in env.hosts:
  try:
  # catch the port number to pass to ssh
  host, port = h.split(’:’)
  local(‘ssh -p %s -A %s@%s ”%s”’ % (port, env.user, host, cmd))
  except ValueError:
  local(‘ssh -A %s@%s ”%s”’ % (env.user, h, cmd))

Got something to say?