LLM API 429 Errors: Make Calls Survive Overload

When an LLM API returns HTTP 429 or struggles under heavy load, a client that retries carefully can recover many temporary failures without overwhelming the service. This guide shows how to classify errors, use backoff and retry limits, control request volume, and preserve work when recovery takes longer.

What You’ll Need

• Your API provider’s documentation for rate limits, retry guidance, and response headers.
• The code that sends requests, plus a way to log status codes, elapsed time, and retry attempts without recording secrets or sensitive prompt content.
• A configurable timeout and a place to store or requeue work if requests cannot finish immediately.

Step-by-Step

1. Identify the failure before retrying. Handle HTTP 429 as a rate-limit response, but inspect the provider’s error body and documentation: it may mean too many requests, too many tokens, a quota or billing limit, or temporary capacity pressure. These cases do not all recover in the same way. A 5xx response can indicate a transient server problem; authentication, malformed-request, and permission errors generally need correction rather than retries. Do not infer that every 429 is temporary.
2. Honor the server’s retry instruction. If the response includes a valid Retry-After header, wait at least that long before retrying. It can specify a delay in seconds or an HTTP date. Some providers also return rate-limit reset information in provider-specific headers; follow their documented meaning. Parse headers defensively, and do not retry sooner just because a local timer expired. If no usable guidance is returned, use your own backoff.
3. Use exponential backoff with jitter. Start with a short delay, increase it after each failed attempt, and cap the delay. Add random jitter so many clients do not retry in sync. For example, choose a random delay between zero and the smaller of a configured cap and base delay × 2^attempt. Tune the base and cap to the provider’s limits and your latency needs; there is no universal safe value. When a server specifies a longer wait, honor it rather than clipping it to a shorter local cap.
4. Set a retry budget and an overall deadline. Limit the number of attempts and the total time spent on one operation. Include connection and response time in the deadline, not just sleep time. If the budget expires, return a clear retryable failure or place the work into a durable queue. Avoid unlimited retries: they can tie up workers, create duplicate work, and intensify an outage. Make attempt limits configurable and observable.
5. Retry only operations that are safe to repeat. A request may have been processed even if the client timed out before receiving its response. Check whether the API supports idempotency keys or another deduplication mechanism, and use it as documented for operations where duplicates matter. If it does not, consider whether repeating the request could create duplicate side effects. For ordinary generation calls, repeated attempts can still incur additional usage, so track them and avoid automatic replay when the outcome is uncertain unless your application accepts that risk.
6. Control concurrency and request rate. Retrying alone does not prevent overload. Set a maximum number of in-flight calls, and use a rate limiter appropriate to the provider’s documented request and token limits. Apply limits across all workers that share the same account or project; a per-process limit may be insufficient when several instances run at once. If available, use a queue to smooth bursts and process work at a sustainable pace.
7. Reduce the demand that caused the limit. Check whether limits apply to requests per minute, tokens per minute, concurrent requests, or a combination. Reduce burst size, avoid unnecessary duplicate calls, and keep prompts and requested output lengths within what the task needs. If a request exceeds a documented per-request token limit, shorten or split the work appropriately; retries will not make an oversized request valid. Request a limit increase only through the provider’s normal process.
8. Keep failures visible and recoverable. Log the status, provider request or correlation ID when supplied, retry count, wait duration, and final outcome. Do not log API keys or unrestricted user data. For queued jobs, use bounded retries and a dead-letter or failed-job path so persistent failures can be inspected and deliberately replayed. Expose a useful message to users instead of leaving them waiting indefinitely.
9. Test the behavior without creating a real traffic spike. In a test or mocked client, simulate 429 responses, a Retry-After value, transient 5xx responses, timeouts, and a permanent 4xx error. Verify that waits grow as intended, retries stop at the deadline, permanent errors are not retried, and duplicate-sensitive work is handled safely. Keep tests from sending repeated load to a production endpoint.

Common Mistakes

• Retrying every error, including invalid input, authentication failures, or exhausted quota. Classify the response and follow the provider’s guidance.
• Retrying immediately or using the same fixed delay for every client. This creates synchronized retry bursts; use backoff and jitter.
• Ignoring Retry-After, or interpreting a provider-specific reset header without checking its documented units and meaning.
• Setting a high retry count with no total deadline. A long chain of attempts can consume workers and make an outage worse.
• Assuming a timeout means the server did not process the request. The result may be uncertain, so account for duplicate calls and usage.
• Limiting each application instance independently while many instances share one account. Coordinate limits at the appropriate shared scope.

Tips

• Keep retry policy in one reusable client layer so different parts of the application behave consistently.
• Use separate limits for concurrency and request rate: one controls simultaneous work, the other controls how quickly calls begin.
• Track 429s, retry success rate, wait time, queue depth, and final failures. A rising queue can signal that incoming work exceeds available capacity.
• Make the policy configurable. Provider limits and application latency requirements differ, and a deployment change should not require rewriting request logic.
• Respect cancellation. If a user cancels a task or its deadline passes, stop waiting and avoid starting another attempt.

Troubleshooting

429s continue after retries: Check the response body and provider dashboard or documentation to distinguish a temporary rate limit from quota exhaustion, a token ceiling, or an account restriction. Reduce traffic or correct the account issue; waiting and retrying cannot resolve a hard limit.

Calls fail in bursts despite backoff: Confirm that jitter is actually applied and that all application instances share an appropriate rate limit. Check whether scheduled jobs or other services use the same account.

Retries take too long: Review the overall deadline, per-call timeout, server-directed wait, and queue policy together. If the required wait exceeds the interactive request’s deadline, return a pending or retryable status and continue asynchronously where your application supports it.

Users receive duplicate results or charges: Check whether a timed-out call may have completed, and use documented idempotency or deduplication support. Avoid automatically replaying uncertain operations until the duplicate risk is addressed.

Only large prompts fail: Determine whether the provider’s limit concerns input tokens, total context, or output tokens. Measure or estimate token use using provider-supported methods, then reduce or divide the work within documented limits.

Frequently Asked Questions

Should every 429 be retried?
No. Retry when the provider indicates a temporary limit or capacity issue. A quota, billing, or account restriction needs a different fix.

How many times should I retry?
There is no universal count. Set a small, bounded attempt budget that fits your overall deadline and provider guidance, then send unfinished work to an appropriate failure or queue path.

Is exponential backoff enough?
Not by itself. Add jitter, honor server retry instructions, limit concurrency and request rate, and stop at a deadline.

Should I retry HTTP 5xx errors too?
Often a transient 5xx can be retried with the same bounded policy, but check provider guidance and consider whether the request is safe to repeat. Do not treat every 5xx as proof that no work was done.

What if the API gives no retry header?
Use capped exponential backoff with jitter and a finite retry budget. Monitor outcomes and adjust the policy to the documented limits and your service’s latency needs.

Can retries guarantee a request will succeed?
No. They can recover from temporary failures, but not from persistent overload, invalid requests, hard quotas, or unavailable services. Preserve unfinished work and report failure clearly.

Related

• How to Secure a Raspberry Pi Exposed to the Internet
• Unable to Initialize Steam API: Causes & Fixes
• Sony WH-1000XM6 Setup & Pairing Guide