First off, realtime websites are hard. The current toolset is rudimentary. When I first started building Ginger, I thought I must be doing it wrong because of all the trial-and-error and pieces I was building from scratch. After watching Geoff Schmidt’s keynote at DjangoCon, I realized it’s not that I’m dumb, it’s that everyone is reinventing their own wheel. His co-creation, Meteor, promises to fix that, but until then, we’re stuck with the tools we have.
One thing nearly every realtime site needs is a replay log. Public networks are inherintly unstable (doubly so for mobile). If a client disconnects for a short period of time, users will expect to receive any data it missed when it reconnects. That’s where a replay log comes in. It keeps a log of all activity and, when a client reconnects, it streams all the data it is missing.
Step 1: Create the Replay Logs
Most realtime sites consist of a set of channels that users subscribe to and can push/pull data through. We want to maintain a replay log for each of those channels. The easiest way to do this is to put a thin wrapper around the code that publishes to the channel. For us, that looks something like this (Python):While this probably isn’t code you can copy/paste into your site, you should get the idea. The last line is our replay log for the channel as a sorted set, ordered by the timestamp (this is important for later).
Step 2: Unspool a Replay Log on Reconnect
zrangebyscore, read the docs. It does exactly what we need here (remember we set the score as the timestamp in Step 1).
Step 3: Prune the Replay Logs
Now that we’ve got it working, we need to make sure the replay logs don’t grow unbounded. We have a scheduled task that runs and removes everything that is older than a certain limit. In Python, that looks like this:
Step back and revel in the awesomeness of Redis. Sorted sets provide an elegant way to bolt-on replay logs to an existing site. I have a feeling a lot of people have built similar solutions, but my searches failed to provide any details.
Already have a solution for reconnecting realtime clients? I’d love to hear about it in the comments!