Primary image for Speed Comparison: Putting Pyston and PyPy to the Test

Speed Comparison: Putting Pyston and PyPy to the Test

The most famous quote to characterize python speed is:

It is fast enough

I have used this quote many times, and I still believe that it applies to most circumstances if you throw enough hardware at the problem.

Despite Python being “fast enough” for most applications, a few projects are working on making it even faster. The most well known is PyPy. It has been under active development for nearly a decade and is a production-ready implementation of the CPython API. In the last couple years, a competing project named Pyston has been seeing regular development and improvements. They’re backed by Dropbox who has every reason to prioritize speed (considering that they manage massive amounts of code). They’ve also employed Guido van Rossum, the creator of Python, since 2012.

Both Pyston and PyPy promise to deliver a faster Python Virtual Machine. They are doing so by applying a technique that has proven to be extremely efficient in JavaScript-land. This method is called “Just In Time compilation.”

Here’s what the Pyston team team writes to explain their initiative:

There are already a number of Python implementations using JIT techniques, often in sophisticated ways. PyPy has achieved impressive performance with its tracing JIT; Jython and IronPython are both built on top of mature VMs with extensive JIT support. So why do we think it’s worth starting a new implementation?

In short, it’s because we think the most promising techniques are incompatible with existing implementations. For instance, the JavaScript world has switched from tracing JITs to method-at-a-time JITs, due to the compelling performance benefits. Whether or not the same performance advantage holds for Python is an open question, but since the two approaches are fundamentally incompatible, the only way to start answering the question is to build a new method-at-a-time JIT. Another point of differentiation is the planned use of a conservative garbage collector to support extension modules efficiently. Again, we won’t know until later whether this is a better approach or not, but it’s a decision that’s integral enough to a JIT that it is difficult to test in an existing implementation.

Our Use Case

The hard thing when you try to compare python implementations is to find a use case and understand the side effects. A few months ago, we hired Maciej Fijalkowski to give us a better understanding of what PyPy could do for our Django-based projects. We put together a test suite comprised of a few simple high-level operations that are common in most Django projects:

  • Building ORM queries that filter on date
  • Render a template tree
  • Combinations of ORM queries that are taken from botbot.me

We built a simplistic “macro” benchmark to understand the disconnect between the stellar benchmark published by PyPy and the mediocre results we were getting ourselves.

Long story short, we learned that PyPy is, in fact, faster for our use case but requires several thousands of iterations to realize the speed improvement. Prior to the JIT “warming up”, PyPy performed significantly worse than the standard CPython implementation.

I decided to run Pyston through the same paces to see how it compared for our use case.

A Word About the Methodology

Django perf tester is a “macro” benchmark that abuses the Django test suite to run the same function a ridiculous number of times (~100,000). We print out the time taken to run it at different points during the iteration cycle. This gives us an idea of the time required to “warm up” before the code optimization kicks in.

I used Docker images pre-built for each Python implementation available at hub.docker.com:

yml@carbon$ (git: master) docker images
REPOSITORY                 TAG                 IMAGE ID            CREATED             SIZE
ubuntu                     latest              2fa927b5cdd3        4 days ago          122 MB
pyston/pyston              latest              eb4e00f6995e        6 days ago          827.3 MB
python                     latest              7471c1ec292d        2 weeks ago         674.4 MB
python                     2.7.11              11a8b7c7f0ca        2 weeks ago         664.9 MB
pypy                       latest              4adf58248d81        2 weeks ago         688.3 MB
pypy                       2-5.1.1             6046115a5c10        2 weeks ago         727.2 MB

I ran this test on my laptop (X1 carbon). As a consequence, all the numbers need to be taken with a grain of salt. My goal was to get an idea of where we’re at for the kind of code I am interested in. I want to see the order of magnitude of requests that need to hit a process before I can get the benefits from these alternative Python implementations.

The benchmark is run like this:

#!/usr/bin/env bash

set -ex

docker run -t --name=django-perf  --rm -v "$PWD":/usr/src/django-perf \
    -w /usr/src/django-perf pypy:2-5.1.1 \
    bash -c "ln -sf /usr/local/bin/pypy /usr/local/bin/python; bash test.sh" > result_test_pypy.txt

docker run -t --name=django-perf  --rm -v "$PWD":/usr/src/django-perf \
    -w /usr/src/django-perf python:2.7.11 bash test.sh > result_test_python2.7.11.txt

docker run -t --name=django-perf  --rm -v "$PWD":/usr/src/django-perf \
    -w /usr/src/django-perf pyston/pyston bash test.sh > result_test_pyston.txt

Lies, Damned Lies and Benchmarks

It is impossible to draw a definitive conclusion out of the number shown below. I have seen a huge volatility in performance gain that can be obtained by each JIT Python implementation depending on the use case. To make the problem even harder to solve, speed is not the only criteria that matters as the memory usage might also be drastically different.

pypy

Python 2.7.10 (b0a649e90b66, Apr 28 2016, 08:57:01)
[PyPy 5.1.1 with GCC 4.8.2]

In most benchmarks, you will see comparisons of Python implementations that show how fast each implementation goes after an unknown number of iterations. The result is that CPython is crushed by Python implementations that can JIT the code.

In the results below I try to give you an idea of the performance characteristic of each implementation over time/iterations.

cpython

Python 2.7.11 (default, May 12 2016, 18:24:08)
[GCC 4.9.2] on linux2

pyston

Pyston v0.5.0 (rev c005f5d8de6152c5b6262b244c052e2859c04a68), targeting Python 2.7.7

Conclusion

If your code base has a code path that is run several thousand times, it might be advantageous for you to try a JITed Python implementation as they can provide a substantial boost. Your code might end up being several times faster after a while. However, there is some code, like the Django ORM, that are less JIT friendly and not only require a lot more iterations to get faster, but also start way slower than CPython.

Pyston is still lagging behind PyPy in term of maximum speed increase, but it seems to have a smaller upfront cost on the first iterations.

Yann Malet

About the author

Yann Malet

Yann builds and architects performant digital platforms for publishers. In 2015, Yann co-authored High-Performance Django with Peter Baumgartner. Prior to his involvement with Lincoln Loop, Yann focused on Product Lifecycle Management systems (PLM) for several large …

View Yann's profile