Kafka Consumer Retry Safety: A How-To Guide

Making Kafka consumers safe to retry means ensuring that when a message fails to process and is retried, it does not cause duplicate side effects, data corruption, or infinite loops. This guide walks through the steps to build idempotent, retryable consumers using standard Kafka features like consumer groups, offsets, and dead letter queues. You will need a working Kafka cluster, a consumer application (in any language, but examples use Java/Kafka Client), and an understanding of your message processing logic.

What You’ll Need

• A Kafka cluster (local or remote) with topics for main messages, retries, and dead letters.
• Kafka client library for your language (e.g., Apache Kafka Java client, librdkafka, etc.).
• Access to a database or storage for idempotency keys, if your processing is not naturally idempotent.
• Basic knowledge of Kafka concepts: topics, partitions, offsets, consumer groups.

Step-by-Step

1. Enable auto-commit off and manage offsets manually. Set enable.auto.commit=false in your consumer config. This prevents Kafka from automatically committing offsets before your processing is complete. You will commit offsets only after a message is successfully processed (or after you’ve decided to skip it).
2. Make your processing idempotent. The core of retry safety: the same message processed twice should have the same effect as processing it once. For example, if you are inserting a record, use a unique key (like a message ID) and perform an upsert. If you are sending an email, check a flag in your database that the email was already sent. If your processing is naturally idempotent (e.g., setting a value), you can skip this step.
3. Use a retry topic with a backoff strategy. Instead of retrying immediately in the consumer (which can block the partition), produce the failed message to a retry topic with a delay. Use Kafka’s max.poll.interval.ms and max.poll.records to control how long you can process before the consumer is considered dead. A common pattern: on failure, produce the message to a retry topic with a timestamp for when to retry, and consume from that topic with a separate consumer that only reads messages due for retry.
4. Implement a retry counter. Add a header to the message, like retry-count. Each time the message is retried, increment this count. When the count exceeds a threshold (e.g., 3), move the message to a dead letter topic (DLT) instead of retrying again. This prevents infinite retry loops.
5. Handle poison messages. If a message fails due to a permanent error (e.g., malformed JSON), do not retry it. Send it directly to the DLT. You can detect this by catching specific exceptions (e.g., SerializationException) and routing them to the DLT immediately.
6. Commit offsets only after successful processing. After you have processed a message (and possibly produced it to a retry topic), commit the offset. If your processing fails and you need to retry, do not commit the offset, so the message is redelivered. But be careful: if you produce to a retry topic, you should commit the original offset to avoid reprocessing the same message infinitely. The retry topic will handle the retry.
7. Use a separate consumer group for retry topics. This isolates retry processing from the main flow, so a retry backlog does not block new messages. The retry consumer can have a different group.id and a lower polling frequency.
8. Test with a simulated failure. Write a test that forces a failure (e.g., a message that always throws an exception) and verify that the message goes to the retry topic, then to the DLT after the max retries. Check that no duplicate side effects occur (e.g., no duplicate database records).

Common Mistakes

• Leaving auto-commit enabled. This can cause offsets to be committed before processing finishes, leading to message loss on a crash.
• Retrying indefinitely. Without a retry limit, a poison message will block the partition forever, halting all subsequent messages.
• Not making processing idempotent. Even with retries, if your processing is not idempotent, you will get duplicates. Retries amplify the problem.
• Retrying on permanent errors. If the message is malformed, retrying will never succeed. Send to DLT immediately.
• Committing offsets too early. If you commit before processing, you lose the message on failure. If you commit after producing to retry but before processing the retry, you might skip the retry if the process crashes.

Tips

• Use a unique message ID. Generate a UUID for each message and use it as an idempotency key. Store it in your database with a unique constraint.
• Consider using Kafka’s transactional API. If you need exactly-once semantics, use transactions to atomically process a message and commit the offset. This is more complex but provides stronger guarantees.
• Monitor retry and DLT topics. Set up alerts for messages landing in the DLT, as they indicate a problem that needs attention.
• Use exponential backoff. Instead of retrying every 1 second, retry after 1s, then 2s, 4s, etc. This reduces load on downstream systems.
• Keep retry topics with a short retention. Retry messages are temporary; set a retention of a few hours to avoid disk usage.

Troubleshooting

• Messages are not being retried. Check that you are producing to the retry topic correctly and that the retry consumer is polling. Verify the retry topic exists and has partitions.
• Duplicate side effects despite retries. Ensure your idempotency logic is correct. Test by processing the same message twice manually and see if the side effect occurs twice.
• Consumer keeps rebalancing. This often happens if processing takes longer than max.poll.interval.ms. Increase this value or reduce max.poll.records.
• Messages go to DLT too quickly. Check your retry count logic. Are you incrementing the header correctly? Is the header being lost when producing to retry topic? Use a serializer that preserves headers.
• Offsets are being committed for failed messages. Make sure you only commit after successful processing. If you are using a retry topic, commit after producing to retry, but ensure the retry is guaranteed to happen (e.g., produce with acks=all).

Frequently Asked Questions

Q: What is the difference between at-least-once and exactly-once semantics?

At-least-once means a message may be processed more than once, but never lost. Exactly-once means each message is processed exactly once. Retries naturally give at-least-once; to get exactly-once, you need idempotent processing or Kafka transactions.

Q: Can I use the same consumer group for retry and main topics?

You can, but it is not recommended. If the retry topic is in the same group, the same consumer instance may process both, and a retry backlog can block new messages. A separate group isolates them.

Q: How do I set the max retry count?

It depends on your use case. A typical value is 3-5 retries. After that, send to DLT. The count is stored in a message header, so you can adjust it per message type.

Q: What if my processing involves multiple side effects that are not all idempotent?

Make each side effect idempotent individually, or wrap them in a transaction with an idempotency key. For example, if you update a database and send an email, use the message ID as a unique key in the database, and check a flag before sending the email.

Q: Do I need a DLT for every retryable consumer?

Not strictly, but it is a best practice. Without a DLT, a poison message will be retried forever, blocking the partition. A DLT allows you to inspect and manually handle failed messages.

Q: How do I handle messages that fail due to a temporary outage of a downstream service?

That is exactly what retries are for. Use a retry topic with exponential backoff. If the outage lasts longer than your max retries, the message goes to DLT, and you can replay it later when the service is back.

Related

• Laser Drone Loses Charging Beam: What Happens?
• EEG Sleep Headbands vs Sleep Trackers: Which?
• Crunchyroll Watch Party: How It Works & How to Host One