Lincoln Loop

Serving Django Projects (Revisited)

After reading the comments on my last post on the subject, I realized there was definitely some room for improvement in my strategy. This is take two of the original post.

Problems

My original strategy had a couple of downfalls:

• Poisoning the Python Path
  I was adding directories to the Python Path that weren’t Python. This raised the chance of namespace collision and just wasn’t a very clean way to do things.
• No project source repository
  I was still storing all my source files in one big folder. Tracking which project is using what was more difficult than it needed to be.

Solutions

The first step to recovery was forgetting about sharing libraries between projects altogether. In theory it sounds great, in practice it was cumbersome to manage. My directory structure now looks like this:

|-domain1.com
|--apache
|---django.wsgi
|--src
|---domain1_project_src
|---third_party_repo1
|---third_party_repo2

This is much cleaner and allows for simple control over the versioning on third party repositories. For source with stable APIs, like Django 1.0, I still place it in /usr/local/src, allowing me to update all sites using the library in one go without worrying about breakage.

Virtualenv is Your New Best Friend

As recommended in the comments on the last post, if you’re hosting multiple projects on one box, virtualenv should be a requirement. Install it on your development machine too. You’ll kick yourself for not using it sooner. Grab virtualenvwrapper while you’re at it too.

After installing and getting a feel for how it works, you can plug it into mod_wsgi. The trouble is it works slightly differently with virtualenv than your interactive sessions. It imports the global site-packages first and then your virtualenv. The other option is a directive to load an empty virtualenv on startup. This hung me up a bit. I need libraries like psycopg2 and PIL and didn’t want to have to build them for each project, but the global site-packages had all sorts of cruft living in it that I didn’t want. I decided to strip it down to the bare essentials and my new policy is that only libraries that I can apt-get live in the global site-packages.

With all this in place, I symlinked the proper modules from domain1.com/src/ into $VIRTUALENV/lib/python2.5/site-packages and as long as I have a clean global python path, I’m good to go in either mod_wsgi or an interactive shell.

What About Static Files?

Not being a server guru, I was pretty proud of myself after I got all this running. There was one rather large wart though. My static files were living in domain1.com/src/domain1_src/domain1/static. For most sites, users were uploading files into a directory that was under version control and not writeable by the web server by default. Thanks to a suggestion from Vlada Macek, I put them where they belong in /var specifically, /var/www/domain1.com. I have a folder in there for uploads and symlinks to static, admin or any other site specific static files.

Suggestions?

I’m really happy with the new setup and it makes server management considerably easier than the mess I considered “stylish” earlier. What about you? If there are better techniques, I’d love to hear them. I’ll try to follow up with some Nginx and Apache configs soon.

Comments

• October 7, 2008 at 6:50 p.m. #
  gundam00 piped up:

  yes!
  i downloaded the virtualenv and it works quite good for me!
  thanks for sharing.

• October 9, 2008 at 3:11 a.m. #
  Gustaf Sjöberg chimed in with:

  This is our current repository layout at distrop:

  project_name/ media/ <- not versioned, used for development, could be anywhere on the filesystem really / static <- symlink to static content project_name/ __init__.py models.py <- project specific models settings.py templates/ templatetags/ urls.py views.py <- project specific views README <- details on dependencies etc static/ <- images, css, scripts

  The projects are deployed in a per-project user’s home directory (we’re serving content using nginx+fastcgi) like so:

  /home/project_name/projects/project_name/ <- the 'projects' directory is a little redundant, we know /home/project_name/media/ /home/project_name/media// /home/project_name/media/static <- symlink to the static directory in the project

  We install mature apps and dependencies with the help of setuptools and keep a site-packages directory on a per-user basis for less mature code.

• December 9, 2008 at 1:06 p.m. #
  Jökull added:

  Thanks for sharing this. This really made virtualenv click for me!

• July 9, 2009 at 1:10 p.m. #
  Martin Diers mentioned:

  I had already done something similar without using virtualenv.

  The basic idea is very simple:

  Inside libs/, I put any 3rd party packages I need, including Django. I never run setup.py on anything.

  Inside myapps/ I put any of my own shared packages. I can then update all my shared apps at once using Mercurial.

  I then symlink from projects/libs/ and projects/myapps/ into my project’s pythonpath directory.

  Each project is launched with it’s own env[‘PYTHONPATH’] pointing to the project/pythonpath/ directory.

  The same layout works in production. I simply set the project’s PYTHONPATH in my wsgi file.

Got something to say?