Django Logging Basics to Production: Part 2

This article explains how to configure Python’s logging framework within Django for production use, covering formatters, handlers, filters, and best practices. You’ll learn how to set up structured, rotating logs that capture errors and warnings without overwhelming your storage or leaking sensitive data.

Background: Why Django Logging Needs a Part 2

Django’s default logging configuration is minimal—it prints errors to the console in development and, in production, you might see nothing at all if you don’t configure anything. The first part of this series likely covered the basics: using logging.getLogger(__name__), setting levels, and writing to a file. But production logging is a different beast. You need to manage log rotation, filter out noise, include request context, and ensure that log files don’t fill up your disk. This part dives into the LOGGING setting in settings.py, which is a dictConfig that gives you full control over loggers, handlers, filters, and formatters.

How Python Logging Works in Django

Python’s logging module uses a hierarchy of loggers. Each logger has a name, usually the module’s __name__, and it propagates messages up to its parent. Django defines several built-in loggers: django, django.request, django.server, django.template, django.db.backends, and django.security. When you call logger.info(), the message is passed to the logger’s handlers, and then up the chain to parent loggers’ handlers, unless propagation is disabled.

In production, you typically want to:

• Capture WARNING and above from your app and Django itself.
• Send ERROR and CRITICAL to a separate file or a monitoring service.
• Rotate log files daily or when they reach a size limit.
• Include timestamps, logger names, and perhaps request IDs.
• Avoid logging sensitive data like passwords or credit card numbers.

Configuring Loggers, Handlers, and Formatters

Your LOGGING dict in settings.py has four main keys: version (always 1), formatters, handlers, loggers, and optionally filters. Here’s a production-ready example:

LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,
    'formatters': {
        'verbose': {
            'format': '{levelname} {asctime} {module} {process:d} {thread:d} {message}',
            'style': '{',
        },
        'simple': {
            'format': '{levelname} {message}',
            'style': '{',
        },
    },
    'handlers': {
        'file': {
            'level': 'WARNING',
            'class': 'logging.handlers.RotatingFileHandler',
            'filename': '/var/log/django/app.log',
            'maxBytes': 1024 * 1024 * 5,  # 5 MB
            'backupCount': 5,
            'formatter': 'verbose',
        },
        'console': {
            'level': 'INFO',
            'class': 'logging.StreamHandler',
        },
    },
    'loggers': {
        'django': {
            'handlers': ['file', 'console'],
            'level': 'INFO',
            'propagate': False,
        },
        'myapp': {
            'handlers': ['file'],
            'level': 'DEBUG',
            'propagate': False,
        },
    },
}

Here, disable_existing_loggers is set to False to avoid disabling Django’s own loggers. The file handler rotates at 5 MB, keeping 5 backups. The console handler is useful for development but can be removed in production if you rely on files or external services.

Choosing the Right Handler: Rotating vs. TimedRotating

Two common file handlers are RotatingFileHandler and TimedRotatingFileHandler. The former rotates when the file reaches a size limit (maxBytes), while the latter rotates at a set interval (e.g., midnight). For high-traffic sites, size-based rotation is often safer because it prevents any single file from becoming enormous. Timed rotation can create many small files if traffic is low, but it makes log analysis by date easier. You can also use logging.handlers.WatchedFileHandler if you rely on an external log rotation tool like logrotate, which sends a signal to the process to reopen the file.

Adding Request Context with Filters

By default, log messages don’t include which URL or user triggered them. You can add this by creating a custom filter that injects request attributes into the log record. For example:

import logging

class RequestFilter(logging.Filter):
    def filter(self, record):
        record.url = getattr(record, 'url', '-')
        record.user = getattr(record, 'user', '-')
        return True

Then attach it to a handler and add {url} {user} to your formatter. To populate these attributes, you’d need to add them to the record in your view or middleware. A simpler approach is to use a middleware that adds a request ID to the logger’s context using logging.LoggerAdapter or contextvars (available in Python 3.7+). For example, you can set a context variable in middleware and read it in a filter.

Common Problems and Misconceptions

One frequent issue is that logs don’t appear because the logger’s level is set too high or propagation is disabled incorrectly. Another is that disable_existing_loggers is left as True (the default), which disables Django’s internal loggers. Also, people often forget that the django.request logger only logs 5xx responses by default; to log 4xx, you need to set its level to WARNING and add a handler. A common misconception is that setting level='DEBUG' on a handler will capture debug messages—actually, the handler’s level filters messages after the logger’s level, so both must be set appropriately. Another pitfall is logging sensitive data: ensure you never log request bodies or authentication tokens. Use a filter to redact fields like password or credit_card.

Structured Logging and External Services

For production, consider structured logging—outputting JSON instead of plain text. This makes it easier to parse logs with tools like Elasticsearch, Splunk, or cloud log services. You can create a custom formatter that outputs a JSON object with fields like timestamp, level, logger, message, and any extra context. For example:

import json
import logging

class JsonFormatter(logging.Formatter):
    def format(self, record):
        log_entry = {
            'time': self.formatTime(record, self.datefmt),
            'level': record.levelname,
            'logger': record.name,
            'message': record.getMessage(),
        }
        if hasattr(record, 'url'):
            log_entry['url'] = record.url
        return json.dumps(log_entry)

Then use 'class': 'myapp.logging.JsonFormatter' in your handler. Many production setups also send logs to a central service like Sentry for errors, but that’s beyond Python’s logging module; you’d typically add a handler that forwards to that service via its SDK.

Security and Performance Considerations

Logging can become a performance bottleneck if you log too much. In production, avoid logging at DEBUG level for high-traffic endpoints. Also, be careful with disk space: even with rotation, logs can fill up quickly if you log large objects. Use maxBytes and backupCount to limit disk usage. For security, never log passwords, tokens, or personal data. Use a filter to scrub sensitive fields. Also, ensure that log files are not publicly accessible—set appropriate file permissions and keep them outside the web root.

Frequently Asked Questions

Why don’t my Django logs appear in production?

Check that disable_existing_loggers is False and that your logger’s level is not higher than the messages you expect. Also, verify that the handler’s path is writable by the web server user. If you’re using Gunicorn or uWSGI, ensure that the logging configuration is loaded after the worker starts—sometimes you need to set --log-config or use django.setup() properly.

How do I log all requests, including 404s?

Add a handler to the django.request logger and set its level to WARNING or INFO. By default, it only logs 5xx responses at ERROR level. To log 404s, set the level to WARNING and it will capture 4xx responses as warnings.

What’s the difference between a logger and a handler?

A logger is what you call in your code (logging.getLogger('myapp')). It decides whether to process a message based on its level. Handlers are attached to loggers and actually write the message somewhere (file, console, network). A logger can have multiple handlers, and each handler can have its own level and formatter.

How can I avoid logging sensitive data?

Use a custom filter that inspects the log record’s message and redacts patterns like password=... or token=.... Also, never log request.body directly. For Django REST Framework, avoid logging request.data unless you’ve sanitized it. You can also set LOGGING_CONFIG to a function that adds a filter to all handlers.

Should I use JSON or plain text logs?

JSON is recommended for production because it’s machine-readable and works well with log aggregation tools. Plain text is easier for quick visual inspection. You can use both: JSON for files, plain text for console during development.

How do I rotate logs daily?

Use TimedRotatingFileHandler with 'when': 'midnight' and set backupCount to keep a certain number of days. For example, 'backupCount': 30 keeps a month of logs. Remember that this handler also has a utc option to use UTC time instead of local time.

Related

• Social Media Management for Higher Ed: Privacy & More
• Foreign Spies Recruit on Social Media? How It Works
• AI Skills vs Cybersecurity Skills: Hire First?