client gave up after: 1.54s. Nothing about that line looks alarming until you know the server had just told the client to wait 120 seconds before trying again. Between those two numbers sits most of what goes wrong with retries in a language model application, and the code that produced it was a stock SDK client with settings nobody had touched.

Part 5 gave the documentation assistant a meter. It can count tokens before it sends and reconcile against what came back. This part points that meter at the failure path, because a call that fails is not free and a retry that succeeds is not free either. Getting this wrong is how a feature that worked all week falls over on the first Monday morning traffic spike.

Failure modes of a model call

A model call can fail in ways that look identical from the outside and need opposite responses. Retrying a malformed request wastes latency and never succeeds. Failing fast on an overloaded provider throws away a request that would have worked on the second attempt. Sorting failures into retryable and terminal is the whole job, and the SDK exception hierarchy already does most of the sorting for you if you catch the typed classes rather than matching on message strings.

Here is the lookup I keep open while writing error handling. Class names are the Anthropic Python SDK; OpenAI uses the same names for everything except OverloadedError, which arrives as a plain InternalServerError on a 5xx status.

One column deserves attention now rather than later. A 429 returns no tokens, so a rate limited retry costs nothing beyond latency and rate limit headroom. A client side timeout is different: if the model finished generating and your client had already walked away, you were billed for that generation and threw it away. Most engineers I meet are worried about the wrong one.

Retry behaviour built into the SDKs

Before writing a single line of retry code, find out what you already have. Both SDKs ship retry and timeout defaults, and both expose them as importable constants, which means you can print them instead of trusting a blog post.

# tested with openai 2.46.0, anthropic 0.117.0, httpx 0.28.1, Python 3.10
from openai._constants import DEFAULT_MAX_RETRIES, DEFAULT_TIMEOUT
print('openai  ', DEFAULT_MAX_RETRIES, DEFAULT_TIMEOUT)

from anthropic._constants import DEFAULT_MAX_RETRIES as A_R, DEFAULT_TIMEOUT as A_T
print('anthropic', A_R, A_T)

# ---- output ----
# openai   2 Timeout(connect=5.0, read=600, write=600, pool=600)
# anthropic 2 Timeout(connect=5.0, read=600, write=600, pool=600)

Two retries means three total attempts. Retries fire on connection errors, 408, 409, 429 and anything at 500 or above, and on a non standard x-should-retry header if the provider sends one. That covers the right status codes. Timing is where it comes apart.

Point a stock client at a server that returns 429 with retry-after: 2 and watch what happens. Everything here runs against a local handler, so no key and no spend:

import os, time, traceback
from openai import OpenAI, RateLimitError

# local stub returns 429 with retry-after: 2 on every call
client = OpenAI(base_url='http://127.0.0.1:8931/rl',
                api_key=os.environ['OPENAI_API_KEY'])  # read from env, never hardcode

started = time.perf_counter()
try:
    client.chat.completions.create(
        model='gpt-4o-mini',
        messages=[{'role': 'user', 'content': 'hi'}],
    )
except RateLimitError as exc:
    print('attempts made:', ATTEMPTS)
    print('elapsed: %.2fs' % (time.perf_counter() - started))
    print('status:', exc.status_code)
    print('retry-after:', exc.response.headers.get('retry-after'))
    print(traceback.format_exc().strip().splitlines()[-1][:110])

# ---- output ----
# attempts made: 3
# elapsed: 4.29s
# status: 429
# retry-after: 2
# openai.RateLimitError: Error code: 429 - {'error': {'message': 'Rate limit reached

That is the good case. Server named a wait, client honoured it, and 4.29 seconds is 2 plus 2 plus request overhead. Anyone reading that output would reasonably conclude the defaults are fine.

Where default backoff falls short

Change one thing. Have the server ask for 120 seconds instead of 2, which is an entirely normal response when you have exhausted a per minute token budget on a large request.

# same client, same call, server now sends retry-after: 120

# ---- output ----
# server asked for retry-after: 120
# attempts made: 3
# client gave up after: 1.54s

Cause sits in one line of SDK source: a retry-after value is honoured only when it falls in the range 0 < retry_after <= 60. Anything larger is discarded and the client falls back to its own exponential schedule, which starts at 0.5 seconds and doubles. So the header that carried the most useful information the provider had is the one the client ignored, and instead of waiting the client added two more requests to a service that was already refusing them.

Once the header is out of play, everything depends on the backoff schedule, which is min(0.5 * 2^n, 8.0) seconds with jitter. Totalled up, the retry budget looks like this:

Set that against the window you are trying to survive. Rate limits replenish continuously under a token bucket, but a badly timed burst against a per minute budget can leave you shut out for the better part of a minute. Even nine attempts cannot cover it:

Timeout settings that matter

Connect timeout is 5 seconds, which is sensible. Read timeout is 600 seconds, which is a decision made for long generations and inherited by every trivial call you make. A classification prompt that normally returns in 700 milliseconds will, on a bad day, hold a worker thread for ten minutes before anyone finds out. Under a synchronous web server with a fixed worker pool, a handful of those will take the whole service down while your model dashboards stay green.

Set timeouts per call, not per client, because a summarisation over a long document and a yes or no classification have nothing in common:

import httpx, os, time, traceback
from openai import OpenAI, APITimeoutError

client = OpenAI(
    api_key=os.environ['OPENAI_API_KEY'],
    max_retries=0,                                   # we will own retries ourselves
    timeout=httpx.Timeout(connect=5.0, read=60.0, write=10.0, pool=5.0),
)

# per call override for a short classification
fast = client.with_options(timeout=8.0)

started = time.perf_counter()
try:
    fast.chat.completions.create(
        model='gpt-4o-mini',
        messages=[{'role': 'user', 'content': 'hi'}],
    )
except APITimeoutError:
    print('elapsed: %.2fs' % (time.perf_counter() - started))
    print(traceback.format_exc().strip().splitlines()[-1])

# ---- output (against a server that never responds, timeout set to 2.0) ----
# elapsed: 2.01s
# openai.APITimeoutError: Request timed out.

Resist the urge to make timeouts aggressive across the board. Cutting a generation off at 8 seconds when it needed 11 does not save money, it spends the money twice: the provider generated and billed those tokens, your client discarded them, and the retry pays again. For anything expected to run long, streaming is the correct answer rather than a shorter timeout, because tokens arrive continuously and the read timeout applies between chunks instead of to the whole response. Deployment shape matters here too, and the trade offs are the same ones covered in serving models batch and real time.

Cost of a retry storm

A nightly job of mine used to reindex changed documentation pages and generate a short summary for each. Forty workers, a queue, and default client settings. One night the provider tightened a limit and the job began taking 429s. Forty workers each turned one question into three requests inside 1.5 seconds, then failed, and the queue handed them fresh work to fail on. Request rate tripled at precisely the moment the provider was asking for less traffic, and the job that normally finished in 18 minutes ran for 2 hours 40 before I killed it.

What surprised me next was the bill. It barely moved. Every one of those thousands of extra requests was a 429, and a 429 generates no tokens, so the storm cost me an evening and no money at all. Real spend was hiding somewhere else. Two weeks earlier I had set a 20 second timeout on the summarisation call to stop it blocking workers. Around 8 percent of pages were long enough to exceed it, so 8 percent of summaries were generated, billed, thrown away and generated again. Nobody noticed, because the job succeeded.

Put numbers on it using the assistant from Part 5, which costs 13.13 dollars per 1,000 requests at standard rates. At an 8 percent client side timeout rate you send 1,080 billable calls for every 1,000 answers delivered, so the true cost is 14.18 dollars per 1,000, of which 1.05 dollars bought nothing. Push the timeout tighter and 15 percent of calls start missing it, and you are at 15.10 dollars per 1,000. Compare that against zero for the far more dramatic looking rate limit storm. The generative AI cost breakdown covers the pricing model underneath these figures.

Wiring retries into the documentation assistant

Assistant so far answers staff questions from a corpus of public Kubernetes documentation, using the token meter from Part 5. Adding a retry layer means one decision repeated at every failure, and it is worth drawing before coding it:

Turning off the SDK retries entirely with max_retries=0 and owning the loop is the part people hesitate over. Do it anyway. Two layers of retry multiply: an outer loop of 5 wrapped around an SDK default of 2 sends 18 requests, not 6, and that is how a small incident becomes a large one.

# tested with openai 2.46.0, httpx 0.28.1, Python 3.10
import os, time, random, httpx
from openai import (OpenAI, RateLimitError, APITimeoutError,
                    APIConnectionError, InternalServerError)

RETRYABLE = (RateLimitError, APITimeoutError, APIConnectionError, InternalServerError)

def call_with_budget(client, *, deadline_s=45.0, max_attempts=6, **kwargs):
    started = time.monotonic()
    for attempt in range(1, max_attempts + 1):
        try:
            return client.chat.completions.create(**kwargs)
        except RETRYABLE as exc:
            elapsed = time.monotonic() - started
            if attempt == max_attempts or elapsed >= deadline_s:
                raise
            wait = None
            resp = getattr(exc, 'response', None)
            if resp is not None and resp.headers.get('retry-after'):
                wait = float(resp.headers['retry-after'])   # obey it in full
            if wait is None:
                wait = min(0.5 * 2 ** (attempt - 1), 20.0) * random.random()  # full jitter
            if elapsed + wait > deadline_s:
                raise
            print('  attempt %d failed (%s), sleeping %.2fs'
                  % (attempt, type(exc).__name__, wait))
            time.sleep(wait)

client = OpenAI(
    api_key=os.environ['OPENAI_API_KEY'],
    max_retries=0,
    timeout=httpx.Timeout(connect=5.0, read=60.0, write=10.0, pool=5.0),
)

reply = call_with_budget(
    client,
    model='gpt-4o-mini',
    messages=[{'role': 'user', 'content': 'How do I scale a deployment?'}],
)
print('tokens:', reply.usage.prompt_tokens, 'in /', reply.usage.completion_tokens, 'out')

# ---- output (stub returned 429 with retry-after: 3 twice, then 200) ----
#   attempt 1 failed (RateLimitError), sleeping 3.00s
#   attempt 2 failed (RateLimitError), sleeping 3.00s
# succeeded on attempt 3 after 6.29s
# tokens: 412 in / 38 out

Three details carry most of the value. Deadline is checked before sleeping, so the loop never sleeps past the point where the answer stops being useful. Header value is obeyed in full with no 60 second ceiling. Backoff multiplies by random.random() rather than shaving 25 percent off, which spreads concurrent workers across the whole interval instead of bunching them.

One caution before you generalise this. Retrying assumes the operation is safe to repeat, which is true for a read only question over documentation and false the moment the assistant starts calling tools that write. A retried call that opens a support ticket opens two. Tool calling arrives in Part 14 and that constraint comes with it, so keep the retry wrapper on the read path for now.

Limiting concurrency before the provider does it for you

Everything above treats a 429 as weather. Much of the time it is self inflicted, because nothing on the client side caps how many requests are in flight at once. Forty queue workers will happily open forty simultaneous connections and discover your requests per minute ceiling on your behalf. A semaphore around the call site costs four lines and removes the problem at source.

I ran 40 workers against a stub that starts refusing above 8 concurrent requests, once unbounded and once behind a semaphore of 8:

import threadingnnGATE = threading.Semaphore(8)   # match this to your provider concurrency, not your worker countnndef ask(question):n    with GATE:n        return call_with_budget(n            client,n            model='gpt-4o-mini',n            messages=[{'role': 'user', 'content': question}],n        )nn# ---- output, 40 worker threads against a stub that refuses above 8 concurrent ----n# 40 workers unbounded   peak_inflight= 9  429s=14  ok=22  wall=2.64sn# 40 workers, sem(8)     peak_inflight= 8  429s= 0  ok=40  wall=1.48s

Read those two lines again, because the result is the opposite of what most people expect. Unbounded, 14 requests were refused and only 22 of 40 questions got answered, in 2.64 seconds. Gated at 8, nothing was refused, all 40 were answered, and it finished in 1.48 seconds. Restricting concurrency made the system both more reliable and roughly 44 percent faster, because every 429 is wasted round trip time that buys nothing. Sizing the semaphore is the only real work: start from the provider concurrency your tier supports rather than from how many workers you happen to run, and treat it as config so you can change it without a deploy.

Retry and timeout settings I would ship on Monday

Set max_retries=0 on the client and own the loop. Bound it by a deadline in seconds rather than an attempt count, obey retry-after in full however large it is, and use full jitter. Set a read timeout that reflects the call, somewhere near 8 to 15 seconds for a classification and 60 or more for a long generation, and reach for streaming instead of a tighter timeout when a response is genuinely slow. Leave connect at 5 seconds.

If you do one thing this week, count attempts rather than calls in your logging and compare the two numbers. Gap between them is money you are spending on answers nobody reads, and until it is on a chart it stays invisible. A single counter that increments per attempt, tagged with the exception class, would have found my 8 percent problem in an afternoon instead of a fortnight.

With the raw call now reliable, metered and bounded, Part 7 steps back to a design question that decides the shape of everything after it: why retrieval beats fine tuning for most business problems, and what that means for an assistant whose corpus changes every week.

References

• Claude Platform docs, API errors, for the full status code list used in the lookup table, the 32 MB Messages API request ceiling, the advice to catch typed exception classes rather than string matching, the confirmation that SDKs retry twice by default while honouring retry-after, and the _request_id property used for logging.
• Claude Platform docs, Rate limits, for the token bucket replenishment model, the guidance that a 60 RPM limit may be enforced as 1 request per second, and the retry-after and anthropic-ratelimit response headers worth logging.
• openai/openai-python, whose _base_client and _constants modules are the source for every default quoted here: max retries of 2, the 0.5 and 8.0 second backoff bounds, the 60 second retry-after ceiling and the jitter formula. Version 2.46.0 was installed and read directly rather than quoted from documentation.
• anthropics/anthropic-sdk-python, version 0.117.0, confirming identical retry constants and timeout defaults plus the extra OverloadedError and RequestTooLargeError classes.
• kubernetes/website, the public Markdown corpus standing in for product documentation across this series.

About The Author

Dr. Pranay Jha

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

See author's posts