TL;DR
watsonx.ai gives you two inferencing calls. The chat endpoint takes a list of role messages and is the one to reach for first. The text generation endpoint takes a single string and hands you finer control over decoding.
Output length drives latency more than anything else, so cap max_new_tokens and turn on streaming. Streaming does not make generation faster, it just shows the first token in a fraction of a second instead of after the whole answer.
A prompt template turns a working prompt into a versioned, reusable asset with variables, so the wording lives in one place instead of being pasted across services.
One line decides whether your first request works or returns a wall of red: the version query string on the endpoint. Post to /ml/v1/text/generation without ?version=2024-05-31 and the service rejects the call before it ever reaches a model. It is a small thing, and it is the first of several small things that stand between a secured endpoint and a useful answer.
Parts 4 through 7 put an instance on a map, sized its GPUs, and locked down who can reach it. None of that returns a token. Inferencing is where you finally send text in and get text back, and it is also where the cost meter starts and where prompt engineering earns or wastes its keep. This part walks the request itself, the parameters that shape the output, streaming, and the two places IBM wants you to keep your prompts: Prompt Lab for building them by hand, and prompt template assets for reusing them.
From a secured endpoint to your first token
Inferencing is the act of sending a prompt to a foundation model and reading back what it generates. On watsonx.ai every inference call is an HTTPS POST to the regional endpoint you settled on in Part 7, so a Dallas instance calls https://us-south.ml.cloud.ibm.com and a Frankfurt instance calls the eu-de host instead. The model runs where your instance lives, which is exactly why the region choice was a one way door.
Before any of that, you need to prove who you are. You do not send your API key on the request. You exchange the key for a short lived IAM bearer token, a signed string that expires after about an hour, and you put that token in the Authorization header. Every request also carries a project_id or a space_id, which tells watsonx.ai which workspace to bill and which assets you are allowed to touch. Miss the token and you get a 401. Miss the project and you get a 400. Neither error is about the model, and both trip up people on day one, so the flow below draws the whole path once.
flowchart LR
K[API key] --> T[IAM bearer token]
T --> R{Which call}
R -->|one string| G[POST /ml/v1/text/generation]
R -->|role messages| C[POST /ml/v1/text/chat]
G --> M[Foundation model in your region]
C --> M
M -->|stream on| S[SSE token chunks]
M -->|stream off| B[Single JSON response]
Two ways to call a model, generation and chat
watsonx.ai exposes two text inferencing surfaces, and picking the wrong one makes simple things awkward. The text generation endpoint, /ml/v1/text/generation, takes a single input string and returns generated_text. It is the raw surface: whatever you put in the string is exactly what the model sees, with no formatting added on your behalf. The chat endpoint, /ml/v1/text/chat, takes a list of messages tagged with roles such as system, user, and assistant, and applies the model specific chat template for you so the conversation is framed the way the model was trained to expect.
Use chat for anything that looks like a conversation or an instruction to an instruct tuned model such as granite-3-3-8b-instruct, because it inserts the right control tokens and keeps multi turn context in order. Use generation when you need the string to be literal: classification where you feed a rigid template, structured extraction, or a case where you have already formatted the prompt yourself and do not want the service second guessing it. Both endpoints have a streaming twin, /ml/v1/text/generation_stream and /ml/v1/text/chat_stream, which return the same output as a sequence of chunks. The table lines up the choice.
| Endpoint | Input shape | Reach for it when |
|---|---|---|
| /ml/v1/text/chat | List of role messages | Conversations and instruct models, the default |
| /ml/v1/text/generation | Single input string | Literal prompts, classification, extraction |
| /ml/v1/text/chat_stream | Role messages, SSE out | Chat UIs that show the reply as it forms |
| /ml/v1/text/generation_stream | Single string, SSE out | Long single prompt completions, live output |
Endpoint paths and the version date are from the IBM watsonx.ai REST API reference; confirm the current version query value for your account before you build.
Anatomy of a text generation request
Strip a generation call to its parts and there are only four things in the body. The model_id names the model, using the provider prefixed form such as ibm/granite-3-3-8b-instruct. The project_id names the workspace. The input holds the prompt. The parameters object holds everything about how the model should decode, and it is the part most people leave empty and then wonder why the output rambles. Here is the whole request as curl, which is worth sending by hand once before you wrap it in an SDK.
curl -X POST '{watsonx_url}/ml/v1/text/generation?version=2024-05-31'
-H 'Authorization: Bearer {iam_token}'
-H 'Content-Type: application/json'
-d '{
"model_id": "ibm/granite-3-3-8b-instruct",
"project_id": "{project_id}",
"input": "Reply in one sentence: what is watsonx.ai?",
"parameters": {
"decoding_method": "greedy",
"max_new_tokens": 60,
"min_new_tokens": 1,
"stop_sequences": ["nn"]
}
}'Expected response: a JSON body whose results array holds one object with a generated_text field and a generated_token_count. Failure modes: drop the version query and you get a 400 before the model runs; send an expired token and you get a 401; name a model that is not hosted in your region and you get a model_not_supported error, which sends you back to the region check from Part 7.
Decoding parameters that steer the output
Decoding is how the model turns its next token probabilities into an actual choice, and the decoding_method parameter picks the strategy. Greedy takes the single most likely token every step, so it is repeatable and factual and a little flat, which is what you want for extraction, classification, and anything you will parse. Sampling draws from the distribution, so it is more varied and more creative, and you shape it with temperature, top_p, and top_k. Higher temperature flattens the distribution and raises surprise; top_p keeps only the smallest set of tokens whose probabilities sum past a threshold; top_k keeps only the k most likely. For most production work I start greedy and only move to sampling when the task genuinely wants variety.
The parameter that decides your bill and your latency is max_new_tokens. Generation time is dominated by how many tokens the model produces, not by how long your prompt is, because each output token is a separate forward pass while the input is processed in one go. So a cap of 1024 tokens can take roughly eight times as long as a cap of 128, and if you forget to set it the model runs to its own limit and you pay for tokens you never wanted. Set min_new_tokens to stop it quitting early, and set stop_sequences to end cleanly on a delimiter. The chart shows the shape of the max_new_tokens effect.
| Parameter | What it does | A sane starting point |
|---|---|---|
| decoding_method | greedy is repeatable, sample is varied | greedy for parsed output |
| max_new_tokens | Hard cap on output length | Set it to the shortest useful answer |
| temperature | Raises randomness when sampling | 0.7 for creative, unused for greedy |
| repetition_penalty | Discourages repeating tokens | 1.0 to 1.1, nudge up if it loops |
| stop_sequences | Ends generation on a delimiter | A newline pair or a closing tag |
Parameter names follow the watsonx.ai generation parameters schema. Available parameters differ slightly between the generation and chat endpoints, so check the reference for the one you call.
Why streaming changes what the user feels
Streaming is often sold as a speed feature. It is not. A streamed response takes the same total time to finish as a non streamed one, because the model does the same work either way. What streaming changes is when the first words appear. With a plain call the user stares at a spinner until the entire answer is ready, so a 512 token reply that takes about five seconds is five seconds of nothing. With the streaming endpoint the server sends server sent events, SSE, and each event carries the next chunk of text, so the first words land in a fraction of a second and the rest flows in as it is generated.
That gap between time to first token and time to full answer is the whole reason chat interfaces stream. The perceived wait drops from the full generation time to the time to first token, which is close to constant regardless of how long the answer will be. For a batch job that writes to a database, streaming buys you nothing and adds SSE parsing you do not need. For anything a person watches, it is the difference between an app that feels stuck and one that feels alive. The chart puts the two experiences side by side for the same 512 token answer.
Prompt Lab, from clicking to code
You do not have to write curl to try a prompt. Prompt Lab, which you met briefly in Part 2, is the visual workbench inside watsonx.ai Studio where you pick a model, type a prompt, move the same decoding sliders this part just described, and read the output next to the token count and the estimated cost. It has three editing modes: chat for turn by turn messages, structured for a form with separate instruction, examples, and input fields, and freeform for a single raw text box that maps directly onto the generation endpoint.
The feature that makes Prompt Lab more than a toy is the export. Once a prompt behaves, you can save it as a prompt template asset in the project, or generate the equivalent Python or curl so the settings you tuned by hand become code you can paste into a service. This is the honest workflow: tune in the Lab where feedback is instant, then export so the exact model, parameters, and wording travel into your application unchanged. Tuning in code first, then guessing why the output differs from the Lab, is the slow way round.
Prompt templates as reusable assets
A prompt template is a saved pattern for a prompt with named variables punched into it, stored as a governed asset in your project rather than pasted into three services that then drift apart. Instead of hardcoding the instruction and few shot examples in every caller, you write them once, mark the parts that change as variables such as {input} or {topic}, and callers supply only those values at request time. The template carries its own model_id and default parameters too, so the whole recipe, wording, model, and decoding, lives in one versioned place.
In the Python SDK you manage these with PromptTemplateManager from ibm_watsonx_ai.foundation_models.prompts, which stores, lists, loads, and deletes template assets. A stored template can even be deployed on its own, giving it a deployment id you call directly, so the prompt becomes a small service that product teams consume without seeing its internals. The payoff is governance: when legal asks you to change a disclaimer in every generated reply, you edit one asset instead of grepping a codebase. The structure below is what a template actually holds.
A Python client that generates and streams
Curl is good for one call. For an application you use the SDK, and the class that matters is ModelInference. The script below authenticates with your API key, builds a model handle with parameters, runs one blocking generation, then runs a streaming generation that prints tokens as they arrive. Fill in the three placeholders and run it. I flag its limits under the output.
# pip install ibm-watsonx-ai
from ibm_watsonx_ai import Credentials
from ibm_watsonx_ai.foundation_models import ModelInference
creds = Credentials(
url='https://us-south.ml.cloud.ibm.com', # match your region from Part 7
api_key='{your_api_key}',
)
params = {
'decoding_method': 'greedy',
'max_new_tokens': 200,
'min_new_tokens': 1,
'repetition_penalty': 1.05,
'stop_sequences': ['nn'],
}
model = ModelInference(
model_id='ibm/granite-3-3-8b-instruct',
credentials=creds,
project_id='{your_project_id}',
params=params,
)
# 1) one blocking call, whole answer at once
answer = model.generate_text(prompt='Reply in one sentence: what is watsonx.ai?')
print(answer)
# 2) streaming, print each chunk as the model produces it
for chunk in model.generate_text_stream(prompt='List three uses for a governed LLM.'):
print(chunk, end='', flush=True)Expected output: the first print shows one full sentence about watsonx.ai; the second prints a short list that appears word by word rather than all at once. Failure modes: a wrong region url in Credentials returns a 404 with no clear message, so match it to your Part 7 region; an unset project_id raises a validation error before any request; and generate_text_stream yields text chunks, so do not try to index them as if each were a full JSON object. Swap generate_text for chat if you move to role messages.
Worked example
A support tool classifies incoming tickets into three labels, then drafts a reply. The classify step wants one word, so it uses the generation endpoint, greedy decoding, max_new_tokens of 5, and a stop sequence on a newline. That call returns in well under a second and is cheap, because the output is tiny and Figure 2 shows output length is what costs you.
The draft step wants a paragraph a human reads on screen, so it uses the chat endpoint with streaming, max_new_tokens of 512, and greedy decoding for a consistent tone. From Figure 3 the agent sees the first words at about 0.4 seconds even though the full draft takes around 5, which is what makes the tool feel responsive. Same model family, two endpoints, two parameter sets, chosen by what each step needs rather than by habit.
If you are coming from another cloud, the same generation versus chat split shows up everywhere, and the way AWS draws the line in Amazon Bedrock Converse API vs InvokeModel is a close parallel to watsonx.ai chat versus generation. The general craft of writing the prompt itself, independent of any vendor, is covered in Prompt Engineering That Actually Works, which pairs well with the template asset idea here.
Chat API first, raw generation when you need the reins
My recommendation is plain. Reach for the chat endpoint by default, because with an instruct model it applies the right chat template and keeps multi turn context straight, and drop to the text generation endpoint only when you need the input string to be literal, for classification, extraction, or a template you have already formatted yourself. Set max_new_tokens on every call to the shortest useful answer, keep decoding on greedy for anything you parse, and turn to sampling only when the task actually wants variety. Stream every response a person watches and skip streaming for batch jobs that write to storage.
Then get your prompts out of your code. Tune in Prompt Lab where the feedback is instant, save the winner as a prompt template asset with variables, and let callers pass only the values. That is the difference between a prompt you can govern and a string that lives in six services and drifts. Region and security decided where and who in Part 7; decoding, streaming, and templates decide how well it answers here.
Do one thing before Part 9. Send a single generation call to your endpoint by hand, with max_new_tokens set and the version query in place, and confirm you get a generated_text back. Once one request works, everything else is shaping it. Part 9 moves from a bare prompt to a grounded one, building retrieval augmented generation on watsonx.data and vector search so the model answers from your documents instead of its memory.
References
IBM Cloud: watsonx.ai REST API reference
IBM watsonx Developer Hub: Text generation
IBM watsonx.ai Python SDK: ModelInference
IBM watsonx.ai Python SDK: Prompt Template Manager
About The Author
DrJha