retry logic

Retry Logic with Exponential Backoff is the resilience pattern that automatically re-attempts failed API requests with progressively increasing wait times — the fundamental strategy for handling transient failures in AI API integrations where rate limits (429), server errors (500-503), and network timeouts are common and expected failure modes requiring graceful recovery rather than immediate hard failure.

What Is Retry Logic with Exponential Backoff?

• Definition: A retry strategy where failed requests are automatically re-attempted after a waiting period that doubles with each successive failure — starting short (1 second) and growing exponentially (2s, 4s, 8s, 16s) to reduce load on the recovering service while giving it time to stabilize.
• Problem Solved: AI APIs (OpenAI, Anthropic, Google) regularly return transient errors — rate limit exceeded, server overloaded, network timeout — that resolve themselves within seconds. Without retry logic, these transient failures cause application-visible errors that could have been silently recovered.
• Jitter: Random noise added to backoff wait times — prevents the "Thundering Herd" problem where all clients that failed simultaneously retry at exactly the same moment, creating a retry spike that overwhelms the recovering server again.
• Max Retries: Retry logic must have a ceiling — infinite retries create applications that hang indefinitely on non-transient failures. Typical: 3-5 retries with exponential backoff.

Why Retry Logic Matters for AI APIs

• Rate Limits Are Expected: OpenAI, Anthropic, and Google enforce per-minute and per-day token and request rate limits. Applications approaching limits regularly receive 429 responses — retry with backoff is the designed response.
• Server Load Variability: AI inference is computationally expensive — API providers experience load spikes where 503 responses signal temporary capacity constraints that resolve in seconds.
• Network Reliability: Long-running LLM inference requests (10-60 seconds for large generations) are vulnerable to network timeouts, connection resets, and proxy failures.
• Production SLA Requirements: User-facing AI applications cannot display API error messages to end users — transparent retry logic maintains application availability during transient failures.
• Cost Efficiency: Retrying transient failures is dramatically cheaper than adding error handling paths, fallback systems, or manual re-submission workflows.

Exponential Backoff Algorithm

Core algorithm:

wait_time = base_delay × (2 ^ retry_count) + random_jitter

Retry 1: 1 × 2^0 + jitter = 1.0 ± 0.5 seconds
Retry 2: 1 × 2^1 + jitter = 2.0 ± 0.5 seconds
Retry 3: 1 × 2^2 + jitter = 4.0 ± 0.5 seconds
Retry 4: 1 × 2^3 + jitter = 8.0 ± 0.5 seconds
Retry 5: 1 × 2^4 + jitter = 16.0 ± 0.5 seconds (then give up)

Which Errors to Retry

Implementation Examples

Python with tenacity library (Recommended):

from tenacity import retry, stop_after_attempt, wait_exponential, retry_if_exception_type
import openai

@retry(
    stop=stop_after_attempt(5),
    wait=wait_exponential(multiplier=1, min=1, max=60),
    retry=retry_if_exception_type((openai.RateLimitError, openai.APIStatusError))
)
def call_llm(prompt: str) -> str:
    return client.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": prompt}]
    ).choices[0].message.content

Manual Implementation with Jitter:

import time, random

def call_with_backoff(prompt: str, max_retries: int = 5) -> str:
    for attempt in range(max_retries):
        try:
            return llm.generate(prompt)
        except (RateLimitError, ServerError) as e:
            if attempt == max_retries - 1:
                raise  # Last attempt — propagate error
            wait = (2 ** attempt) + random.uniform(0, 1)  # Exponential + jitter
            time.sleep(wait)

Rate Limit Header Handling (Advanced):
OpenAI returns headers indicating when the rate limit resets:

except RateLimitError as e:
    reset_time = e.response.headers.get("x-ratelimit-reset-requests")
    if reset_time:
        wait = max(float(reset_time), 1.0)  # Wait until reset, not just backoff
        time.sleep(wait)

Production Considerations

• Circuit Breaker: After N consecutive failures, stop retrying for a cooldown period — prevents cascading failures where retries amplify overload.
• Async Retry: For high-throughput applications, use async retry to avoid blocking threads during backoff waits.
• User Feedback: For user-facing applications with long retry queues, provide progress indication — "Processing your request..." — rather than silent delays.
• Monitoring: Track retry rates, backoff durations, and ultimate failure rates — high retry rates indicate systematic issues requiring architectural response.
• Budget Accounting: Retries multiply API costs — ensure retry behavior is accounted for in cost modeling.

Retry logic with exponential backoff is the foundational resilience pattern that separates brittle AI prototypes from production-grade AI applications — by automatically recovering from the transient failures that are inevitable when calling AI APIs at scale, retry logic with jitter transforms occasional API hiccups from user-visible errors into seamless, transparent recovery that maintains application reliability and user trust.