Easy Fabric Deployment, Part 1: Git/Mercurial and SSH

We’re firm believers in the practices described by the Continuous Integration method of software engineering. Among those are:

• Maintain a code repository
• Automate the build
• Automate deployment

We use git for our code repositories and Fabric to automate our build/deployment process. The tiny bit of overhead it take to write out a Fabric script pays off very quickly against the tedium and error-prone practice of manually building/deploying. In building our “fabfile”, we encountered a couple of issues that took a little head-scratching to work out.

Git and SSH Keys

Git, like Mercurial and others, confirms your credentials via an private/public key pair when used over SSH. We use gitosis to manage our private repositories, so managing everyone’s keys isn’t much of an issue. The problem comes when developers need to start pulling the repository on different machines. Our developers all push to a central “origin” and our development server runs a clone of that repository.

We want every push to the master branch to automate the build and deployment. The first thought is to manage this with post-commit hooks, but that raises some security concerns. I want the git user that owns all the repositories to have as few permissions as possible. It’s job is to act as a gatekeeper for our repositories, not handle deployment across any number of servers.

The simple workaround is a Fabric script that allows our developers to use one command to push, build and deploy. Our script does the following:

1. Push local git repository to origin
2. Pull development server git repository from origin
3. Install any 3rd party libraries as specified in the pip requirements file
4. Run any necessary migrations
5. Run syncdb
6. Restart the WSGI daemon

For the most part, those are trivial commands. The one that gets us into trouble is pulling to our development server(s). Our developer’s local SSH keys aren’t available server-side. Having to manage a whole new set of SSH keys or require them to copy their private keys to the server was not desirable.

The good news is that, ssh provides a way to forward your authentication agent connection. From the ssh man page:

     -A      Enables forwarding of the authentication agent connection.  This
             can also be specified on a per-host basis in a configuration
             file.

Agent forwarding should be enabled with caution. Users with the
ability to bypass file permissions on the remote host (for the
agent’s Unix-domain socket) can access the local agent through
the forwarded connection. An attacker cannot obtain key material
from the agent, however they can perform operations on the keys
that enable them to authenticate using the identities loaded into
the agent.

The bad news is that neither Fabric, nor paramiko (the guts of Fabric’s SSH implementation) expose this switch. The fix is simply to use your local shell to run any commands over SSH. We created a Fabric function that looks like this:

env.hosts = ['our.development.server']
# ...
def sshagent_run(cmd):
    """
    Helper function.
    Runs a command with SSH agent forwarding enabled.
    
    Note:: Fabric (and paramiko) can't forward your SSH agent. 
    This helper uses your system's ssh to do so.
    """

    for h in env.hosts:
        try:
            # catch the port number to pass to ssh
            host, port = h.split(':')
            local('ssh -p %s -A %s "%s"' % (port, host, cmd))
        except ValueError:
            local('ssh -A %s "%s"' % (h, cmd))

The Authentication Agent

There is one more step you may need to take on your local computer for this technique to work. You’ll need to ensure that your identity has been added to the authentication agent. As far as I can tell, current versions of Ubuntu do this automatically. For other operating systems (OS X and other Linux variants), you may need to do this manually. You can check by running ssh-add -l from the command line. If you get a line back containing the path to your private key, you can skip this step. If you don’t, you’ll want to run ssh-add to add your key to the list. Since this isn’t a persistent setting, you’ll probably want to add it to your ~/.bashrc or someplace else it can run on login.

With your Fabric script and authentication agent setup, you can run automated build and deployments across any number of servers without ever worrying about which machine you are on and where your SSH key lives.

Have you solved this issue in a different way? Leave us a comment, we’d love to hear your thoughts.