Nowadays, it’s common to deploy multiple application servers, but it poses a very common problem; How are these application servers going to share the media files contributed by the users?
Cloud providers like Google, Rackspace or Amazon Web Services solve this problem by providing a cloud storage service. These services are comparable to an extremely slow hard disk that exposes an HTTP API. On the positive side, they are relatively cheap ($0.03/month per GB), provide redundant storage, and can be accessed from anywhere. Django’s storage abstraction layer, in combination with django-storages lets all your application servers manage files within the same bucket in the cloud. While emulating a server farm locally in containers, I looked into the options for sharing files without using a third-party service.
NFS – Network File System
Network File System (NFS) is a distributed file system protocol originally developed by Sun Microsystems in 1984, allowing a user on a client computer to access files over the network much like local storage.
This is the boring and battle-tested solution. In the simplest case, an NFS share can be created by installing the server from your package manager and adding a line to /etc/exports for each directory along the lines of:
/srv/photos *(rw,sync,no_root_squash)
On each client, the directory can be mounted with a single command (or a line in /etc/fstab):
sudo mount example.hostname.com:/srv/photos /srv/photos
As far as your application servers are concerned, the NFS mounted directory is just a directory. You do not need to do anything special to use it in your Django project.
NFS is extremely simple and well documented at first glance, but has left a few scars on the faces of many sysadmins. This page is a good write-ups of a few of the pitfalls.
Minio
Enter Minio, an object storage server built for cloud application developers and DevOps. Minio implements APIs compatible with Amazon S3 v4.
This is a new open source solution that gives you an API similar to the Amazon S3 cloud storage. Unlike NFS or Samba, you do not need to mount anything on the application servers. Django Minio storage provides a custom Django file storage backend that takes advantage of Minio’s Python API client.
Once the Minio server is up and running on your media server:
./minio server /srv/Photos/
AccessKey: <MINIO_ACCESS_KEY> SecretKey:<MINIO_SECRET_KEY> Region: us-east-1
Minio Object Storage:
http://127.0.0.1:9000
http://10.245.115.246:9000
Minio Browser:
http://127.0.0.1:9000
http://10.245.115.246:9000
To configure Minio Client:
$ wget https://dl.minio.io/client/mc/release/linux-amd64/mc
$ chmod 755 mc
$ ./mc config host add myminio http://127.0.0.1:9000 <MINIO_ACCESS_KEY> <MINIO_SECRET_KEY>
Django specific configuration is straightforward if you have ever used django-storages:
# Minio storage configuration in settings.py
MINIO_STORAGE_ENDPOINT = '10.245.115.246:9000'
MINIO_STORAGE_ACCESS_KEY = '<MINIO_ACCESS_KEY>'
MINIO_STORAGE_SECRET_KEY = '<MINIO_SECRET_KEY>'
MINIO_STORAGE_USE_HTTPS = False
MINIO_STORAGE_MEDIA_BUCKET_NAME = "django-media"
MINIO_STORAGE_AUTO_CREATE_MEDIA_BUCKET = True
MINIO_STORAGE_STATIC_BUCKET_NAME = "django-static"
MINIO_STORAGE_AUTO_CREATE_STATIC_BUCKET = True
DEFAULT_FILE_STORAGE = "minio_storage.storage.MinioMediaStorage"
STATICFILES_STORAGE = "minio_storage.storage.MinioStaticStorage"
Wrapping Up
In my opinion, the potential of having a self hosted Amazon S3 compatible storage service like Minio is huge step forward. If you choose to leave the boring (in the good sense), well understood POSIX technology that is NFS, you will be able to take advantage of the great Amazon S3 ecosystem that has flourished in the last decade. The great news is most of the tools dealing with data can now ship it to Amazon S3. With Minio, you are one line away to send it to any of your servers.
