This post covers portions of my talk, Containerless Django, from DjangoCon US 2018.
Deploying Python has improved significantly since I started working with it over a decade ago. We have virtualenv, pip, wheels, package hash verification, and lock files. Despite all the improvements, it still feels harder than it needs to be. Installing a typical large project has many steps, each one easy to trip up on:
- Install Python
- Install build tools (pip/virtualenv, pipenv, poetry, etc.)
- Install build dependencies (C compiler, development libraries, etc.)
- Download the code
- Run the build tools
- If you’re using Node to build client-side files for a website, repeat steps 1-5 for that
It’s normal for ops teams to repeat this process as part of the automated testing and then again on every server the project is deployed to. It’s no wonder Docker has become so popular because of the ease in which you can build-once and deploy-everywhere.
But Docker is a heavy-handed solution and doesn’t fit for every project. I envy the simplicity of languages like Go where you can compile your project down to a single binary that runs without any external dependencies. Even Java’s JAR file format which requires Java to be preinstalled, but otherwise only requires downloading a single file would be a huge improvement.
A JAR File for Python
Turns out there are already a few projects solving this problem. PEX from Twitter, XAR from Facebook, and more ambitious projects like PyOxidizer, but shiv from LinkedIn hits the sweet spot for us. It is simple, stable, does not require special tooling, and incurs no runtime performance hit. It creates a ZIP file of all your code and dependencies that is executable with only a Python interpreter. In a future post, we’ll do a deep-dive into how shiv works under-the-hood, but for brevity’s sake, we’ll treat it as a black box here.
Using Shiv with Django
Shiv works with any proper Python package. Since most Django developers don’t think about packaging their projects, we’ll give you a crash course here.
Packaging Your Project
Previously, the only viable packaging tool was setuptools, but since PEP-517 we now have a number of other options including flit and poetry. At the moment, setuptools is still the de-facto standard, so, despite it being a little cruftier than the other options we’ll use that in our example.
You can use our post, Using setup.py in Your (Django) Project, as a starting point, but we need to take a couple more steps to ensure non-Python files (static files, templates, etc.) are included.
The easiest way to do this is with a MANIFEST.in file. It might look something like this:
graft your_project/collected_static
graft your_project/templates
Note that these directories need to live inside the top-level Python module to be included. Also note that the static files directory should be your STATIC_ROOT not your STATICFILES_DIRS. If you define templates inside your individual apps, you’ll need to include those directories as well.
Dealing with Dependencies
These days, every project should include some sort of a lock file which is machine generated and defines the exact version of every dependency and the hash verifications for them. You can do this via poetry, pip-compile from pip-tools, or pipenv.
I typically let one of these tools handle the install and then pass its site-packages directory to shiv via the --site-packages option. In that case, you’ll also pass the pip flags --no-deps . to install your local project, but not include any defined dependencies for it.
Including an Entry Point
We need to provide shiv a Python function that it will run when the zipapp is executed. The most logical one is the equivalent, manage.py. You can use django.core.management.execute_from_command_line directly, but I recommend writing a small wrapper which also sets the default DJANGO_SETTINGS_MODULE environment variable. You could create a __main__.py in your project and include the following:
import os
from django.core.management import execute_from_command_line
def main():
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "your_project.settings")
execute_from_command_line()
if __name__ == "__main__":
main()
Putting it in __main__.py is a Python convention that also allows you to execute management commands via python -m your_project ....
The Shebang
While not necessary, you can customize the shebang of your zipapp to have it executed with a specific version of Python or a Python from a specific location. I typically use /usr/bin/env python3.7 (or whatever version the project expects).
Putting this altogether, you might have something like this in your CI script:
pipenv install --deploy
pipenv run manage.py collectstatic --noinput
shiv --output-file=your_project.pyz \
--site-packages=$(pipenv --venv)/lib/python3.7/site-packages \
--python="/usr/bin/env python3.7" \
--entry-point=your_project.__main__.main \
--no-deps .
Production Webserver
Astute readers may have noticed there’s not an easy way to run uwsgi or gunicorn when we want to deploy. Typically you execute your webserver, then point it at your project instead of the other way around. We created django-webserver so you’ll have access to your favorite WSGI server as a management command. We also sponsored work on uWSGI to package it as a wheel making it quick and easy to use in this setup. 🎺
You’ll also want to be sure that your zipapp can serve its own static files, either via whitenoise or by letting uwsgi handle your staticfiles (included by default in django-webserver).
Settings
People do all sorts of strange things with Django settings. You can still use the DJANGO_SETTINGS_MODULE environment variable to pick the settings to use at runtime. You can also use environment variables to set different values in your settings file, but that can be tedious and, potentially, a security problem.
Instead, I prefer to use a file that is easily machine readable (JSON or YAML) and also easily generated from configuration management or a secret manager like chamber or Hashicorp Vault.
We built another package, goodconf which lets you use a static configuration file (or environment variables) to adjust settings across environments. This lets us treat our zipapp more like a standard application and less like a special-case Django app. The people who handle your deployments should appreciate this.
Deployment
Once you have your zipapp, deployment is almost trivial:
- Install Python
- Create the configuration file
- Download the zipapp (we store ours in S3)
- Start server -
./myproject.pyz gunicornor./myproject.pyz pyuwsgi
Caveats
Zipapps created with shiv aren’t a perfect solution. You should be aware of a few things before you start using them.
Extraction
On first execution, shiv will cache the zip file contents into a unique directory in ~/.shiv (path is configurable at runtime). This creates a small delay on first run. It also means you may need to periodically clean out the directory if you’re doing lots of deploys.
System Dependencies
If you are using libraries which depend on system libraries, they will also need to be installed on the deployment target. For example, mysqlclient will require the MySQL library. Fortunately, the proliferation of the wheel format allows authors to bundle these libraries with their packages as is the case with Pillow, psycopg2-binary, lxml, and many others.
Flexibility
Your zipapp will define a single entry point. While it is possible to override it at runtime or even drop into a Python interpreter, I would save those options for debugging only. If you are used to running arbitrary scripts from your project or loading arbitrary files from your git repository, you’ll need to use some more discipline to make management commands for the things you want to run once deployed.
Portability
Pure Python projects should be very portable across different operating systems and potentially different Python versions. As soon as you start compiling dependencies however, your best bet is to build on the same OS and Python as you intend to run the project.
Isolation
Your project will run with the system’s site packages in sys.path, effectively the same as creating a virtualenv with --system-site-packages. For better isolation, use the -S flag for Python, e.g. python3.7 -S ./your_project.pyz. See this GitHub Issue for more details.
–
We’ve been successfully running this site and many of our client sites using shiv-generated zipapps for a few months now. We’re very happy with the simplicity and speed it lends to rolling out new software. If you’re using shiv or a similar technique to bundle your application, let us know in the comments below.
