keepcalculator

06 Numbers and Powers

Stop Planner Retries From Running the Same Tool Call Twice

Card No. 06·81496 · Updated · Standard formula, shown below

If your automation planner is retrying a failed tool call and accidentally re-running a side-effectful action (like sending an email or charging a card), the fix is to make the tool call idempotent or to deduplicate retries by a unique request ID. This article explains why retries duplicate work, how to prevent it with idempotency keys, and how to configure your planner’s retry policy so a retry never repeats a completed operation.

Why Retries Cause Duplicate Tool Calls

Planners (AI agents, workflow engines, or API orchestrators) retry a tool call when the first attempt fails—due to a network timeout, a crash, or a 5xx error. The problem is that the failure may have occurred after the tool actually executed its side effect. For example, an email sending service might accept the request, send the email, but the response is lost before the planner receives it. The planner sees a timeout, assumes the call never ran, and retries—sending a second email.

This is the classic “at-least-once” delivery problem. Most retry systems guarantee that a call is attempted at least once, not exactly once. To get exactly-once behavior, you must add a mechanism that recognizes a retry of the same logical request and prevents the side effect from happening again.

How Idempotency Keys Work

An idempotency key is a unique identifier that you attach to a tool call. The receiving service stores the key along with the result of the first execution. When a retry arrives with the same key, the service returns the stored result instead of executing the action again.

Here’s a typical flow:

  1. Planner generates a unique key (e.g., UUID) for the tool call.
  2. Planner sends the request with header Idempotency-Key: 123e4567-e89b-12d3-a456-426614174000.
  3. Service executes the action, stores the result keyed by that ID.
  4. Planner retries due to a timeout, sending the same key.
  5. Service sees the key already exists, returns the stored result without re-executing.

This requires the tool itself to support idempotency keys. Many modern APIs (Stripe, PayPal, Slack, etc.) do. If your tool doesn’t, you can implement a similar pattern on your side: before calling the tool, check a local database for the key; if present, return the cached result.

Configuring Retry Policies in Your Planner

Most planner frameworks let you customize retry behavior. The key is to limit retries to cases where the tool call is known to be safe to repeat, or to disable retries entirely for side-effectful operations.

  • Disable retries for non-idempotent calls – If a tool cannot be made idempotent, set max_retries=0 for that specific call. Instead, log the failure and handle it manually.
  • Use exponential backoff – Even with idempotency keys, backoff reduces load on the service and gives the network time to recover.
  • Only retry on certain errors – Retry on timeouts and 5xx, but not on 4xx (client errors) which indicate the request is invalid.
  • Set a retry budget – Limit total retries (e.g., 3) to avoid infinite loops.

Example in Python (pseudo-code):

from tenacity import retry, stop_after_attempt, wait_exponential, retry_if_exception_type

@retry(
    stop=stop_after_attempt(3),
    wait=wait_exponential(multiplier=1, min=2, max=10),
    retry=retry_if_exception_type(TimeoutError),
    before_sleep=log_retry
)
def call_tool_with_idempotency(tool, args, key):
    return tool.invoke(args, idempotency_key=key)

Common Mistakes When Implementing Deduplication

Even with idempotency keys, several pitfalls can still cause duplicate executions:

  • Key not unique per logical operation – If you reuse the same key for different calls, the service will incorrectly return a previous result. Always generate a fresh UUID for each new logical action.
  • Key not passed on retry – Some frameworks create a new request object on retry, losing the original key. Ensure the key is part of the retry request.
  • Storing results in a non-transactional way – If the service crashes after executing the action but before storing the result, the retry will execute again. Use atomic writes or a database that supports “insert if not exists”.
  • Assuming the tool is idempotent when it isn’t – A tool like “send email” is not idempotent by default. You must wrap it with a deduplication layer if the API doesn’t support keys.

Comparing Approaches: Idempotency vs. Exactly-Once Semantics

Idempotency keys are the most common solution, but there are alternatives:

  • Idempotent operations – Design the tool so that re-running it produces the same result. For example, “set status to paid” is idempotent; “increment counter” is not.
  • Exactly-once delivery – This is a stronger guarantee that the action is executed exactly once, often achieved through distributed transactions or message queues with deduplication (e.g., AWS SQS FIFO with message deduplication). However, it’s more complex and often overkill for a simple planner.
  • Manual reconciliation – For low-volume or critical actions, you might skip retries and instead log the attempt, then have a human verify and re-run if needed.

In practice, idempotency keys are the standard because they are simple, widely supported, and work across most APIs.

Safety and Reliability Considerations

When dealing with side-effectful tool calls, the cost of a duplicate can be high: double charges, duplicate emails, or corrupted data. Always consider the blast radius:

  • Financial operations – Always use idempotency keys if the payment provider supports them. Never retry a charge without a key.
  • Emails and notifications – Duplicate emails are annoying but not catastrophic. Still, use idempotency keys to maintain trust.
  • Database writes – If the tool writes to a database, make the write idempotent (e.g., use an upsert with a unique constraint).
  • Logging – Always log the idempotency key and the outcome so you can audit if a duplicate does occur.

If you cannot guarantee idempotency, the safest policy is to not retry at all. Instead, fail the task and alert a human.

Frequently Asked Questions

What if the tool doesn’t support idempotency keys?

You can implement a client-side deduplication layer. Before calling the tool, check a database (or in-memory store) for the key. If found, return the stored result. If not, execute the call, store the result, and then return it. This gives you the same effect even if the external service is not idempotent.

Should I retry on every error?

No. Only retry on transient errors like timeouts, connection resets, and 5xx responses. Do not retry on 4xx errors (e.g., 400 Bad Request, 401 Unauthorized) because the request is invalid and will fail again. Also, avoid retrying on errors that indicate a permanent condition.

How long should I keep idempotency keys?

Typically, you need to keep them for at least as long as you might retry. Most services expire keys after 24 hours. If you have a long-running workflow, consider storing keys in a persistent store with a TTL that matches your maximum retry window.

Can I use the same idempotency key for different tool calls?

No. The key must be unique per logical operation. If you reuse a key, the service will think it’s a retry and return the previous result, which could be wrong for a different action.

What’s the difference between idempotent and exactly-once?

Idempotent means that executing the same request multiple times has the same effect as executing it once. Exactly-once means the action is executed exactly one time, even if retries happen. Idempotency is a property of the operation; exactly-once is a delivery guarantee. With idempotency, you can achieve exactly-once behavior by having the service deduplicate retries.

Related

Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.

More cards in this drawer

The whole Numbers and Powers drawer →