My first useful call against our documentation assistant returned a two sentence answer and this usage line:

ResponseUsage(input_tokens=1163, output_tokens=94, total_tokens=1257)

Question asked was forty words long. Everything else in that 1,163 was instructions I had written once and forgotten about, plus three paragraphs of product documentation I had pasted in to give the model something to work from. Twelve times more input than the user typed, on every single request, forever. That ratio is the first thing worth internalising about calling a model from code: you are not billed for the question, you are billed for the entire conversation you reconstruct and resend each time.

Shape of a model request

Every major provider exposes roughly the same thing: an HTTPS endpoint that accepts a list of messages and returns one more message. Each message carries a role and some content. Roles differ slightly by vendor but three concepts recur. Developer or system content sets standing behaviour and is not part of the conversation. User content is what the person typed. Assistant content is what the model said on previous turns, which you resend so it can see its own history.

Statelessness is the part that surprises backend developers, because it inverts a habit. HTTP sessions taught us that the server remembers and the client sends a token. Here it is reversed: your process is the memory. If a user asks a follow up question, nothing on the provider side knows what came before unless you paste it back in. Some APIs now offer server side conversation storage as an option, but the billing model underneath is unchanged, and the input still gets re tokenised on every call. Context length is a hard ceiling on how much history you can carry, which context window explained covers properly.

First call from Python

Our running project through this series is an internal documentation and support assistant for a mid sized SaaS company. Right now it does not exist. By the end of this part it will be a single function that answers a question from a chunk of documentation you pasted in by hand, which is a deliberately poor design that Parts 8 to 13 replace with real retrieval. Getting the call correct first means the retrieval work later has something reliable to sit on.

Install the SDK and pin it. Provider SDKs change signatures more often than most libraries you depend on, and an unpinned install will break a working script three weeks later with no code change on your side. Set your key as an environment variable and never put it in the file. Every code block below reads from the environment.

# tested against: Python 3.12, openai 2.46.0
# pip install 'openai==2.46.0'
# export OPENAI_API_KEY=sk-...   (never hardcode this)

import os
from openai import OpenAI

client = OpenAI()  # reads OPENAI_API_KEY from the environment

DOC = '''Rate limits on the Starter plan are 60 requests per minute per API key.
Exceeding them returns HTTP 429. Limits reset on a rolling 60 second window,
not on the minute boundary.'''

resp = client.responses.create(
    model='gpt-5',
    instructions='You answer staff questions from the product documentation provided. '
                 'If the documentation does not contain the answer, say so plainly.',
    input=f'Documentation:n{DOC}nnQuestion: If I hit the limit at 10:00:30, when can I retry?',
    max_output_tokens=400,
)

print(resp.status)
print(resp.output_text)
print(resp.usage)

completed
Because the window rolls rather than resetting on the minute, capacity frees up
60 seconds after each individual request. A request made at 10:00:30 falls out
of the window at 10:01:30, so retry from then rather than at 10:01:00.

ResponseUsage(input_tokens=131, output_tokens=61, total_tokens=192)

Three things in that snippet deserve attention. Separating instructions from input keeps standing behaviour out of the user turn, which matters later when you start logging and replaying user inputs on their own. Reading the key from the environment means the file is safe to commit. Printing status and usage rather than only output_text is the habit that saves you in the section after next.

On the Anthropic side the same call is client.messages.create with a system argument instead of instructions, messages as an explicit list of role and content dictionaries, and max_tokens rather than max_output_tokens. Text arrives at resp.content[0].text and the finish reason at resp.stop_reason. Concepts map one to one; only the field names move.

Parameters that matter, and parameters that do not

Here is where a lot of received wisdom has gone stale. Nearly every introduction to LLM APIs opens with temperature, presents it as the main dial, and moves on. On current reasoning models that parameter is not merely discouraged, it is rejected outright. Sampling controls were removed because the model no longer generates a single pass answer that you can usefully perturb; it produces internal reasoning first, and the knob that governs quality and cost is how much of that reasoning you are paying for.

resp = client.responses.create(
    model='gpt-5',
    input='Summarise the rate limit policy in one sentence.',
    temperature=0,
)

Traceback (most recent call last):
  File "first_call.py", line 14, in <module>
    resp = client.responses.create(
openai.BadRequestError: Error code: 400 - {'error': {'message':
"Unsupported parameter: 'temperature' is not supported with this model.",
'type': 'invalid_request_error', 'param': 'temperature',
'code': 'unsupported_parameter'}}

Fix is to drop the parameter, not to set it differently. Reach for reasoning effort and verbosity instead. Effort governs how much internal thinking the model does before answering, which drives both latency and the output token bill. Verbosity governs how long the visible answer is, independently of how hard the model thought. Conflating those two is a common and expensive mistake: teams turn effort up because answers felt thin, when what they wanted was a longer final answer at the same effort.

Below is the reference I keep pinned. It is worth reading once properly rather than discovering each row through a production incident.

Temperature, measured rather than assumed

Plenty of models still accept temperature, so it is worth knowing what it actually buys. Second piece of received wisdom to retire: setting temperature to zero does not give you a deterministic system. It makes sampling greedy, which is not the same thing. Batching on the serving side, floating point non associativity across different hardware, and silent model updates all move output under you. I measured this rather than argue about it, asking one fixed question twenty times at four settings and counting distinct answers.

# tested against: Python 3.12, anthropic 0.116.0
# pip install 'anthropic==0.116.0'
# export ANTHROPIC_API_KEY=sk-ant-...

import anthropic

client = anthropic.Anthropic()  # reads ANTHROPIC_API_KEY
Q = 'In one sentence, what happens when a Starter plan key exceeds 60 requests per minute?'

for temp in (0.0, 0.3, 0.7, 1.0):
    seen = set()
    for _ in range(20):
        r = client.messages.create(
            model='claude-sonnet-5',
            max_tokens=120,
            temperature=temp,
            messages=[{'role': 'user', 'content': Q}],
        )
        seen.add(r.content[0].text.strip())
    print(f'temperature={temp}  distinct answers={len(seen)}/20')

temperature=0.0  distinct answers=2/20
temperature=0.3  distinct answers=6/20
temperature=0.7  distinct answers=14/20
temperature=1.0  distinct answers=20/20

Practical consequence: do not build a cache key, a test assertion or a diff on the assumption that identical input yields identical output. Assert on structure and on meaning, which is what Parts 20 through 22 build properly. For the underlying reason variation and confident wrongness travel together, why AI hallucinates and what temperature really does is the GenAI series companion to this section.

Truncation and other silent failures

Here is the mistake that cost me the most embarrassment on this project. Early version of the assistant summarised support tickets, and I set max_output_tokens to 512 because summaries are short and I was being frugal. Everything looked fine in testing. Three weeks later a customer success manager forwarded a summary that ended mid clause, at the words "the customer confirmed that the refund was", and asked what the rest said. There was no rest. Roughly four percent of summaries had been truncated for three weeks, because longer tickets produced longer summaries and I had only ever tested on short ones.

Nothing raised. No exception, no warning, no non zero exit. Response came back with a status of incomplete and a reason of max_output_tokens, and my code read output_text and ignored everything else. That is the default shape of a first implementation and it is why the very first code block in this part printed status. Reasoning models make this sharper still, because invisible reasoning tokens count against the same ceiling, so a request can burn its entire budget thinking and return an empty string.

LONG_TICKET = open('ticket_8841.txt').read()  # 2,900 words

resp = client.responses.create(
    model='gpt-5',
    instructions='Summarise this support ticket for a handover note.',
    input=LONG_TICKET,
    max_output_tokens=512,
)

if resp.status != 'completed':
    reason = getattr(resp.incomplete_details, 'reason', 'unknown')
    raise RuntimeError(f'model response incomplete: {reason}')

print(resp.output_text)

Traceback (most recent call last):
  File "summarise.py", line 12, in <module>
    raise RuntimeError(f'model response incomplete: {reason}')
RuntimeError: model response incomplete: max_output_tokens

On Anthropic the same check reads resp.stop_reason and compares against max_tokens rather than inspecting an incomplete_details object. Same defect, same fix, different field name. Set the ceiling generously and treat it as a circuit breaker rather than a budget, because output tokens are billed on what is generated, not on what you allowed.

Streaming and perceived latency

Generation is sequential, so a long answer takes a long time. Streaming does not change that, it changes when the user sees the first word. On our assistant a typical grounded answer took 5.1 seconds end to end. Streamed, first visible text appeared at 410 milliseconds, and total completion drifted marginally later at around 5.3 seconds because of per event overhead. Users described the streamed version as much faster. Same work, thirteen times better first impression.

import time

start = time.perf_counter()
first_token_at = None

with client.responses.stream(
    model='gpt-5',
    instructions='Answer from the documentation provided.',
    input=f'Documentation:n{DOC}nnQuestion: How does the rolling window work?',
    max_output_tokens=400,
) as stream:
    for event in stream:
        if event.type == 'response.output_text.delta':
            if first_token_at is None:
                first_token_at = time.perf_counter() - start
            print(event.delta, end='', flush=True)
    final = stream.get_final_response()

print(f'nnTTFT: {first_token_at*1000:.0f} ms')
print(f'Total: {time.perf_counter()-start:.2f} s')
print(f'Status: {final.status}')

Rolling means capacity frees up 60 seconds after each request rather than at
the top of the minute, so a burst at 10:00:30 clears at 10:01:30.

TTFT: 412 ms
Total: 5.31 s
Status: completed

Cost of streaming is not latency, it is that you lose the ability to inspect a complete response before showing it. Anything you plan to validate, filter or parse must not be streamed to the user directly, because you cannot un show a sentence you have already rendered. Structured output in Part 4 and guardrails in Part 23 both need the whole response in hand. My rule: stream prose answers, never stream anything that a schema or a filter has to approve first.

Keeping the call site provider neutral

Every field name that differs between providers is a place your application can grow a dependency it did not intend. Instructions against system, max_output_tokens against max_tokens, status against stop_reason, output_text against content[0].text. None of those differences are interesting, and all of them will end up scattered through your codebase if you call the SDK directly from twenty places. Twenty minutes of work now prevents a genuinely tedious refactor later, and Part 29 builds on exactly this seam.

from dataclasses import dataclass

@dataclass
class Completion:
    text: str
    input_tokens: int
    output_tokens: int
    complete: bool
    reason: str

def ask(system: str, user: str, provider: str = 'openai', max_out: int = 1500) -> Completion:
    if provider == 'openai':
        r = openai_client.responses.create(
            model='gpt-5', instructions=system, input=user, max_output_tokens=max_out
        )
        return Completion(
            text=r.output_text,
            input_tokens=r.usage.input_tokens,
            output_tokens=r.usage.output_tokens,
            complete=(r.status == 'completed'),
            reason=getattr(r.incomplete_details, 'reason', '') or 'completed',
        )
    r = anthropic_client.messages.create(
        model='claude-sonnet-5', system=system, max_tokens=max_out,
        messages=[{'role': 'user', 'content': user}],
    )
    return Completion(
        text=r.content[0].text,
        input_tokens=r.usage.input_tokens,
        output_tokens=r.usage.output_tokens,
        complete=(r.stop_reason != 'max_tokens'),
        reason=r.stop_reason,
    )

Resist the urge to make this abstraction clever. It should normalise names and nothing else. Every team I have seen grow a general purpose LLM abstraction layer ended up with something that supported the intersection of provider features rather than the union, and quietly blocked adoption of whatever shipped next. Frameworks such as LangChain and LlamaIndex offer a richer version of this seam and are a reasonable choice when you want their retrieval and agent pieces too, but for a first project a thirty line dataclass is easier to debug and hides nothing from you.

Log status and usage before you write another prompt

If you take one action from this part, make it this: every call site records the finish status and both token counts alongside the response, from the first commit. Truncation rate, cost per request and the effect of any prompt change all become measurable the moment those three fields are in your logs, and all three become archaeology if they are not. Costs nothing. Nobody does it until it has already bitten them.

Second recommendation, narrower: pin your SDK version, drop temperature from any call to a reasoning model, and put reasoning effort at the lowest setting you can defend rather than the highest you can afford. Our assistant now has a working call. It is expensive, it knows only what you paste into it, and it has no idea whether its answers are any good. Part 3 tackles the prompt itself, and specifically the patterns that hold up when real users start typing things you did not anticipate.

References

• OpenAI, Migrate to the Responses API, for the recommendation that new projects start on Responses rather than Chat Completions, and for the reasoning model behaviour differences between the two surfaces.
• OpenAI API reference, Responses, for the definitions of input, instructions, max_output_tokens and temperature used in the parameter table.
• openai-python releases, for version 2.46.0, released 17 July 2026, which is the version every OpenAI code block here was tested against.
• Claude Platform docs, Python SDK, for messages.create, the system argument, max_tokens and stop_reason as used in the provider neutral wrapper.
• OpenAI Cookbook, GPT-5 new params and tools, for reasoning effort and verbosity as the replacements for sampling controls on reasoning models.

About The Author

Dr. Pranay Jha

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

See author's posts