Python Logging Setup: Do You Need Your Own?

Yes, Python workers need their own logging setup. While the standard library’s logging module provides a solid foundation, a worker process—whether a Celery task, a background thread, or a queue consumer—requires explicit configuration to ensure logs are captured, formatted, and routed correctly. Without it, you risk missing errors, mixing log sources, or losing logs entirely when the worker crashes.

Why Default Logging Isn’t Enough

Python’s logging module is powerful but intentionally minimal out of the box. By default, it writes to stderr with a simple format and a default level of WARNING. This works for a simple script, but fails in worker contexts for several reasons:

• No structured output: Default logs are plain text, making them hard to parse for monitoring tools like Loki or CloudWatch.
• No rotation or retention: Without configuration, logs grow indefinitely, filling disk space.
• No propagation control: In a worker, logs from third-party libraries can double-print or get lost if the root logger isn’t configured.
• No context: Worker logs often need to include task IDs, queue names, or request IDs—something the default setup doesn’t provide.

For a worker that runs 24/7, these gaps are not just inconveniences; they are operational hazards.

How Logging Works in Python

The logging module uses a hierarchy of loggers. Each logger has a name (e.g., myapp.worker) and inherits settings from its parent. When you call logger.info(), the logger checks its effective level, creates a LogRecord, and passes it to handlers. Handlers are responsible for outputting the record to a destination: console, file, socket, etc. Each handler has a formatter that controls the log message structure.

Key components:

• Logger: The object you call info(), error(), etc. on.
• Handler: Sends the log record to a destination (e.g., StreamHandler, FileHandler).
• Formatter: Defines the layout of the log message (e.g., timestamp, level, message).
• Filter: Optionally modifies or suppresses records based on custom logic.

In a worker, you typically want at least one handler for console output (for container logs) and possibly a file handler for local debugging. You also want to set the root logger level to INFO or DEBUG to capture all relevant messages.

What a Worker Needs

A robust worker logging setup should address the following:

• Unified configuration: Use a dictConfig or fileConfig to define loggers, handlers, and formatters in one place. This avoids scattered basicConfig calls.
• Appropriate levels: Set the root level to INFO for production, but allow DEBUG via environment variable for troubleshooting.
• Structured output: Use JSON formatter (e.g., python-json-logger) to include fields like timestamp, level, logger name, and custom context.
• Log rotation: If writing to files, use RotatingFileHandler or TimedRotatingFileHandler to prevent unbounded growth.
• Exception logging: Ensure unhandled exceptions in the worker are logged with traceback using logger.exception() in except blocks or by setting sys.excepthook.
• Contextual information: Inject task IDs, queue names, or user IDs into log records using filters or a custom formatter.

Common Pitfalls

Even with a good setup, workers often suffer from these mistakes:

• Double logging: When a logger has both a handler and propagation to the root logger, messages can appear twice. Fix by setting propagate = False or ensuring the root logger doesn’t have a handler that duplicates.
• Forgotten third-party logs: Libraries like requests or urllib3 log at DEBUG level. If you set the root to DEBUG, you may drown in noise. Instead, set specific loggers to WARNING.
• No flush on exit: In a worker that shuts down abruptly, buffered logs may be lost. Use logging.shutdown() in a finally block or ensure handlers are flushed.
• Hardcoded paths: Log file paths should come from configuration or environment variables, not hardcoded strings.

How to Choose the Right Setup

The right logging setup depends on your deployment:

• Containers: Log to stdout/stderr and let the container runtime collect logs. Use a JSON formatter for easier ingestion.
• Traditional VMs: Use file handlers with rotation, and consider a centralized log aggregator (e.g., syslog, Logstash).
• Serverless: Use the built-in logging of the platform (e.g., CloudWatch for AWS Lambda) and avoid file handlers entirely.

Also consider the logging library. The standard library is sufficient for most cases, but structlog or loguru offer nicer APIs and better structured logging out of the box. However, they add dependencies and may not integrate seamlessly with all frameworks.

Costs and Trade-offs

A custom logging setup has costs:

• Maintenance: You must update and test the configuration as your application evolves.
• Performance: Logging every line synchronously can slow down high-throughput workers. Consider async handlers or batching (e.g., QueueHandler + QueueListener).
• Complexity: Over-engineering logging with too many handlers, filters, and formatters can make debugging harder, not easier.

But the cost of not having a proper setup is higher: lost logs, undiagnosed errors, and wasted time sifting through unstructured output.

Alternatives to a Custom Setup

If you don’t want to build your own, consider these alternatives:

• Use a framework’s built-in logging: Celery and Django have their own logging configurations. You can extend them rather than start from scratch.
• Adopt a logging library: structlog provides a high-level API that simplifies structured logging and integrates with the standard library.
• Use a cloud logging service: Services like AWS CloudWatch, Google Cloud Logging, or Datadog offer agents that capture logs from your worker without custom code.

Frequently Asked Questions

Should I use basicConfig in a worker?

No, basicConfig is a convenience for simple scripts. In a worker, it’s better to use dictConfig to have explicit control over loggers, handlers, and formatters. basicConfig also only configures the root logger, which can lead to unexpected behavior with library loggers.

How do I log to both file and console?

Add two handlers to your logger: a StreamHandler for console and a FileHandler for the file. Set the logger’s level to the minimum of the two, and each handler can have its own formatter and level if needed.

What’s the best format for worker logs?

JSON is generally best for production because it’s machine-readable and easy to search. A minimal JSON format includes timestamp, level, logger name, and message. For local development, a human-readable format like %(asctime)s %(levelname)s %(name)s: %(message)s is fine.

How do I include task IDs in logs?

Use a filter that adds the task ID to the LogRecord. In Celery, you can use the task_id from the current_task and add it to the record’s __dict__. Then include it in your formatter as %(task_id)s.

Why are my logs missing in production?

Common causes: the root logger level is set too high (e.g., WARNING), the handler is not attached to the correct logger, or the worker is running in a context where stderr is not captured. Check your configuration and ensure the worker’s logging is initialized before any log calls.

Related

• Hybrid Work Schedule: 12 Tips That Actually Work
• College Recruitment Games: Mobile-Friendly Essentials
• JBL Tune 770NC: ANC, EQ & Battery Claims