For our latest product, Ginger, we wanted to marry the real-time functionality we needed with the traditional Django stack we know and love. After some false starts and falling on our faces in the beginning, we ended with a stack we’re happy with and think will serve us well moving forward.
Back-end
This is our bread-and-butter. It’s Django and PostgreSQL running with Gunicorn behind Nginx. The biggest difference between this and your typical Django site is that it mainly serves as a REST API consumed by the front-end. We are using traditional Django views in a few places like login, logout, and user profiles, but the rest of the views are handled by the front-end.
Using Django, we were able to quickly build the nuts-and-bolts parts of our site that didn’t require any special real-time functionality. A few of the important libraries we’re using:
- tastypie for building the REST API
- haystack backed by solr for search
- celery backed by Redis for task queueing
- misaka and pygments for text formatting
- easy-thumbnails and PIL for image thumbnailing
Front-end
Our front-end uses Backbone.js to handle most of the routing (using HTML pushState) and interactions. We currently use jQuery templates, but the library is deprecated and we’ll likely switch to Underscore.js templates in the future.
Realtime Push/Pull
Our real-time layer probably gave us the most trouble as we got started. I really wanted to use Python and hit dead-ends at almost every turn with gevent and tornado. The libraries I found turned out to be more proof-of-concepts than ready-for-production. We finally decided on Node.js with socket.io. It seemed to have the most development activity and was keeping up with changing browser implementations well. The socket.io/websocket API is a breeze and a joy to work with.
Our Node.js server is very small, weighing in at less than 200 lines of code. Its primary purpose is to fan data out to all the connected clients. Almost all writes/pushes from the client go directly to our Django HTTP API. This let us keep our logic where our best skills are (Python) while getting the benefits of socket.io.
Redis serves as a Pub/Sub transport from Django to Node.js. As Django gets requests over the HTTP API, it places events on Redis queues which are immediately grabbed by Node.js and distributed to the clients. An interesting side note, Django puts serialized objects in Redis as one of the last things before returning the HTTP response to the client (after performing all the logic and hitting the database). Despite this, the clients consistently receive the event over the websocket before receiving an HTTP response. Redis to Node.js to client is blazingly fast.
Other Bits
- Munin for server monitoring
- Sentry for application exception logging (using our own JavaScript reporter)
- Jenkins for continuous integration testing and pylint and jshint for code checking
- Selenium for functional testing
- SendGrid for email
- Stripe for payments
- Chef for server configuration management
- Supervisord for process management
- Fabric for deployment-related tasks
- Rackspace cloud servers running Ubuntu
- Rackspace cloud files for GPG encrypted backups
- GitHub for repository hosting and issues
- NameCheap for DNS
- RapidSSL and StartSSL for certificates
- ZenDesk for support
It’s a shocking amount of stuff that goes into making what we hope is a “simple” product. We would probably make some different decisions if we rebuilt it all from scratch today, but, for the most part, I think the stack would look very similar. If you’re wondering why we made some of the decisions we did, ask away and we’ll do our best to answer.
