How to Set Up Django with Docker, Postgres, Redis Cache, Redis Message Queue, and Celery: A Complete Guide

For developers building scalable web applications, integrating Docker with Django, Postgres, Redis, and Celery offers an efficient and robust setup.

This guide takes you step-by-step through creating a containerized development environment using Docker and Docker Compose to handle your database, caching, and asynchronous task processing needs. This is an advanced topic of the previous blog here - How to setup Django app with PostgreSQL using Docker

Why This Setup?

Combining these technologies provides:

Database Power: Postgres is a reliable, feature-rich relational database.
Efficient Caching: Redis reduces database load and speeds up request handling.
Asynchronous Processing: Celery handles long-running tasks like sending emails or processing images.
Portability: Docker ensures consistency across all environments.

Prerequisites

Before diving in, ensure you have:

Docker and Docker Compose installed on your system.
A Django project (new or existing).
Basic knowledge of Docker, Redis, and Celery.

Step 1: Set Up Your Project Directory

Organize your project for clarity and scalability:

django-docker-redis/

├── django_project/

├── Dockerfile

├── docker-compose.yml

├── requirements.txt

├── celery_tasks/

Step 2: Create the Dockerfile

The Dockerfile defines your application’s environment. Here’s a minimal setup for Django:

# Use a lightweight Python base image FROM python:3.10-slim

# Set the working directory in the container WORKDIR /app

# Copy and install dependencies COPY requirements.txt /app/ RUN pip install --no-cache-dir -r requirements.txt

# Copy the application code into the container COPY . /app/

# Expose port 8000 for the Django development server EXPOSE 8000

# Command to run the Django app CMD ["python", "manage.py", "runserver", "0.0.0.0:8000"]

Step 3: Configure Docker Compose

The docker-compose.yml file manages multiple services. Here’s how to define Django, Postgres, Redis, and Celery:

version: '3.8'

services: web: build: . command: python manage.py runserver 0.0.0.0:8000 volumes: - .:/app ports: - "8000:8000" depends_on: - db - redis

db: image: postgres:13 environment: POSTGRES_USER: glinteco POSTGRES_PASSWORD: glinteco POSTGRES_DB: glinteco

redis: image: redis:6

celery: build: . command: celery -A django_project worker --loglevel=info depends_on: - redis - db

Step 4: Update Django Settings

Configure the Database

In settings.py, update the DATABASES setting to connect to Postgres:

DATABASES = {

    'default': {

        'ENGINE': 'django.db.backends.postgresql',

        'NAME': 'glinteco',

        'USER': 'glinteco',

        'PASSWORD': 'glinteco',

        'HOST': 'db',

        'PORT': 5432,

    }

}

Set Redis as the Cache Backend

CACHES = {

    'default': {

        'BACKEND': 'django.core.cache.backends.redis.RedisCache',

        'LOCATION': 'redis://redis:6379/1',

    }

}

Configure Celery for Redis

CELERY_BROKER_URL = 'redis://redis:6379/0'

CELERY_ACCEPT_CONTENT = ['json']

CELERY_TASK_SERIALIZER = 'json'

Step 5: Create and Test Celery Tasks

Add Celery Configuration

In your Django project, create a file celery.py:

from celery import Celery

app = Celery('django_project') app.config_from_object('django.conf:settings', namespace='CELERY') app.autodiscover_tasks()

Create a Sample Task

In your app, add tasks.py:

from celery import shared_task

@shared_task def add(x, y): return x + y

Step 6: Build and Run the Containers

Build the Images

docker-compose build

Start the Containers

docker-compose up

Run Migrations

docker-compose run web python manage.py migrate

Test the Celery Task

Use the Celery shell to test your task:

docker-compose run celery celery -A django_project call tasks.add --args='[10, 20]'

Step 7: Testing the Setup

Access the Django app at http://localhost:8000.
Test Redis caching with an API endpoint.
Verify that Celery tasks are processed asynchronously.

Enhancements for Production

To make this setup production-ready:

Replace the development server with gunicorn.
Use nginx as a reverse proxy.
Monitor Celery tasks using Flower.
Secure sensitive credentials with a .env file.

Conclusion

This guide has equipped you to set up Django with Docker, Postgres, Redis Cache, Redis Message Queue, and Celery. With this architecture, your app can handle high traffic, optimize performance, and manage asynchronous tasks efficiently.

For further exploration: