The first multimodal call I shipped on Azure failed with a 400 and a one line message: this deployment does not support image input. The model was right. I had pointed a text only gpt-4o-mini deployment at a request that carried a base64 image, and Azure told me so before I burned a token. That error is the whole lesson of this part. On Azure, multimodal is not one switch you flip, it is four different services with four different deployment shapes, and the first job is knowing which one owns each kind of input.
Multimodal on Azure splits across four homes. Vision (image in, text out) rides the same chat models you already deploy, gpt-4o, gpt-4.1, and o series reasoning models. Voice (speech in, speech out) is a separate model family, gpt-realtime, spoken over WebRTC or a WebSocket, not the chat endpoint. Image generation is its own deployment, the gpt-image models. And documents, video, and long audio go through Azure Content Understanding, a managed extraction pipeline that went generally available on the 2025-11-01 API. Pick the wrong home and you either get a 400 or a bill three times bigger than it needed to be.
What multimodal means on Azure, and the services that do it
Multimodal just means a model that takes in, or gives back, something other than text: an image, a chart, a photo of a receipt, a spoken sentence, a video. On a single vendor like OpenAI you tend to think of this as one model that does everything. Azure surfaces it differently, because the deployment, the pricing, and the network path are not the same for a photo as they are for a live phone call. Getting that split clear up front saves you the debugging I did the hard way.
Four homes, and I will use these names for the rest of the part. Vision is image understanding: you send an image alongside text and the chat model reads it. This runs on the ordinary chat completions and Responses API you met in Part 11, no new endpoint. Voice is the Realtime family, a low latency speech in, speech out channel that keeps a session open. Image generation is the gpt-image models, deployed like any other but called through an images endpoint. And Content Understanding is the batch style service for files: PDFs, scanned forms, recorded calls, and video, returning structured fields and transcripts instead of a chat reply.
How a vision request reaches the model
Vision is the easy one, because it reuses the chat path you already know. You add an image content part to the same message array you send for text. The image goes either as a public URL or as a base64 data URL inside the request body. The model reads it and answers in text. My failing 400 from the intro happened because the deployment behind my endpoint was a text only model. The fix was one line: deploy a vision capable version, gpt-4o or gpt-4.1, and point the same code at it.
The part people miss is how images are billed. An image is not free context, it is converted into tokens based on its resolution and a detail setting. A high detail 1024 by 1024 photo can cost well over a thousand input tokens on its own. Send ten of them in one prompt and your input token count jumps by five figures before the user has typed a word. Set detail to low when you only need the gist, keep the resolution modest, and you cut that cost by more than half. This is the single biggest surprise for teams moving a text app to vision.
Realtime voice with gpt-realtime and WebRTC
Voice is where Azure looks nothing like the chat API. The current production model is gpt-realtime, which reached general availability in 2025, with a lighter gpt-realtime-mini for cost sensitive work. It keeps a bidirectional session open so audio streams both ways with sub second latency. You reach it two ways: WebRTC when the audio starts in a browser or on a device, and a WebSocket when a server holds the mic. The older gpt-4o-realtime-preview still exists, but I start new builds on gpt-realtime and treat the preview as legacy.
Two design decisions matter before you write any code. First, put the model deployment in a region close to your users, because voice is unforgiving about round trip time in a way a chatbot never is. Second, decide early whether the browser talks to the model directly over WebRTC with a short lived key, or whether your server sits in the middle. Direct WebRTC is lower latency and cheaper to run. A server in the middle gives you control over logging and moderation. I default to the server path for anything a regulator will ask about later, and direct WebRTC for a consumer demo.
In practice
Realtime bills audio as tokens, not minutes, and the two directions are not symmetric. User audio counts at roughly one token per 100 ms, assistant audio at one token per 50 ms. So a minute of you talking is about 600 tokens, but a minute of the assistant talking back is about 1,200. The model that answers at length costs you double on the way out. Trim the assistant response length and you cut the expensive half of the bill.
Generating and editing images with the gpt-image models
Image generation is a separate deployment. The base model is gpt-image-1, with gpt-image-1.5 now generally available and a newer gpt-image-2 in the catalog. All of them do text to image and image to image, so you can generate from a prompt or edit an uploaded picture, including inpainting where you mask a region and describe the change. On Azure the output dimensions are fixed to one of three shapes: 1024 by 1024, 1024 by 1536, or 1536 by 1024. Square renders fastest. There is no arbitrary size, so design your UI around those three.
Access is gated. The gpt-image series needs limited access registration on Azure before you can deploy it, which trips up teams expecting to click deploy and go. Request access early in the project, not the week you plan to ship. Pricing is per generated image and scales with quality tier and size, so a low quality square is a fraction of a high quality tall image. If you are rendering thumbnails at volume, the quality setting is where the money is, not the model choice.
| Modality | Service | Model or analyzer | Call path |
|---|---|---|---|
| Image understanding | Azure OpenAI chat | gpt-4o, gpt-4.1 | chat / Responses API |
| Live voice | Azure OpenAI Realtime | gpt-realtime, gpt-realtime-mini | WebRTC or WebSocket |
| Image generation | Azure OpenAI images | gpt-image-1, 1.5, 2 | images endpoint |
| Docs, video, audio files | Content Understanding | prebuilt or custom analyzers | analyze REST + SDK |
Table 1. Which service owns which input. Model names verified against the Foundry catalog, July 2026.
Content Understanding for documents, video, and audio
When the input is a file rather than a chat turn, you want Content Understanding. It is a managed extraction service that ingests documents, images, audio, and video and returns structured fields, transcripts, and summaries you can feed into a RAG index or an agent. It went generally available with the 2025-11-01 API version, and ships prebuilt analyzers so you do not start from scratch: document analyzers for forms and PDFs, an image analyzer for visual descriptions, an audio analyzer with speaker diarization and multilingual transcription, and a video analyzer that does scene detection and segment summaries. SDKs exist for Python, .NET, Java, and JavaScript.
The reason this is a separate service and not a vision call is scale and shape. A chat model reads one image in a prompt. Content Understanding is built to run over ten thousand invoices or a library of recorded calls, extract the same fields from each, and hand you rows. If you tried to do that by looping a chat model over every page you would pay more, hit rate limits, and get less consistent structure. Reach for Content Understanding whenever the job is turn a pile of files into data, and reach for the chat model when the job is answer a question about this one image. That line is the whole decision.
Worked example
Take a support desk that runs all four modalities in a month. Voice: 8,000 minutes of calls on gpt-realtime at about 0.25 per minute uncached, roughly 2,000 dollars. Documents: 50,000 pages through Content Understanding at about 0.01 per page, roughly 500 dollars. Image generation: 5,000 rendered images at about 0.04 each, roughly 200 dollars. Vision: 20,000 photos read by the chat model at about 0.004 each, roughly 80 dollars. Total is about 2,780 dollars, and voice alone is 72 percent of it.
The lesson I take from that split every time: optimize the voice channel first, and do not waste a sprint shaving vision cost that was never the problem. Rates are tagged VERIFY, plug in your own from the pricing page, but the shape holds.
Which service handles which input?
Here is the rule I give teams so they stop guessing. If the input is an image and you want an answer about it, send it to the chat model as a vision request. If the input is live speech and latency matters, open a Realtime session on gpt-realtime. If you need to produce a picture, deploy a gpt-image model and call the images endpoint. If the input is a file, or a batch of files, and you want structured data out, run Content Understanding. When two paths could work, pick the one whose billing model fits your volume: per token for interactive, per file for batch.
A minimal two modality script: read an image with the chat model, then generate one. It uses the OpenAI SDK pointed at your Azure endpoint. Swap in your deployment names.
import base64, os
from openai import AzureOpenAI
client = AzureOpenAI(
azure_endpoint=os.environ["AZURE_OPENAI_ENDPOINT"],
api_key=os.environ["AZURE_OPENAI_API_KEY"],
api_version="2025-04-01-preview", # confirm in portal
)
# 1) vision: read an image, answer in text
r = client.chat.completions.create(
model="gpt-4o", # a vision capable deployment
messages=[{
"role": "user",
"content": [
{"type": "text", "text": "What is on this receipt? Total only."},
{"type": "image_url",
"image_url": {"url": "https://example.com/receipt.jpg",
"detail": "low"}},
],
}],
)
print(r.choices[0].message.content)
# 2) generation: make one 1024x1024 image
img = client.images.generate(
model="gpt-image-1", # gated, request access first
prompt="a flat vector icon of a paper receipt, red accent",
size="1024x1024",
n=1,
)
open("out.png", "wb").write(base64.b64decode(img.data[0].b64_json))
print("saved out.png")Expected output: the receipt total prints, then saved out.png appears in your folder. Failure mode: a 400 with image input not supported means the chat deployment is text only, redeploy on gpt-4o. A 403 or DeploymentNotFound on the image call almost always means the gpt-image access request has not been approved yet, or the deployment name does not match.
What multimodal costs, and where it surprises you
Three cost traps catch teams new to this. First, vision tokens: a handful of high detail images can quietly become your largest input cost, so set detail low unless you need fine text in the image. Second, asymmetric audio: the assistant speaking costs about twice what the user speaking does, so a chatty voice agent is a chatty bill. Third, gated access lead time: gpt-image approval is not instant, and a launch that assumed same day access slips a week. None of these show up in a quick demo. All three show up in the first real invoice.
My take
Do not build a voice agent because the demo is charming. Build one when a phone or hands free workflow genuinely beats typing, because it is the most expensive and most latency sensitive path on this whole list. For most business problems, vision plus Content Understanding does the real work at a fraction of the cost, and a text chat handles the conversation. Voice is the modality I add last and remove first.
Content Understanding for the pipeline, gpt-realtime only when latency pays
If you are standing up multimodal on Azure this quarter, here is where I would put the effort. Start with vision on the chat model you already run, because it is a one line change and covers a huge share of real requests: read this photo, this chart, this screenshot. Add Content Understanding next, because turning files into structured data is where multimodal earns its keep for most businesses, and it feeds cleanly into the RAG work from Part 12. Add image generation only when the product genuinely renders pictures, and request that gated access on day one. Add voice last, and only when a hands free or phone workflow is worth its price and its latency budget. Compare this against how AWS bundles the same job in the Bedrock Data Automation part of the AWS series before you commit a design.
Next, take one image capable deployment, wire the vision snippet above into a real request from your own app, and watch the token count. That single number tells you more about your multimodal bill than any pricing page will.
References
- Use the GPT Realtime API for speech and audio, Microsoft Learn
- Use image generation models from OpenAI, Microsoft Learn
- Azure Content Understanding is now generally available, Microsoft Foundry blog
- Azure OpenAI Service pricing
About The Author
DrJha