Four shapes cover almost every generative app teams actually ship on Azure: a chat front end, a retrieval app that grounds answers in your own content, an agentic system that calls tools and other agents, and a batch job that scores millions of rows overnight. Everything else is a variation on one of those, or a combination of two. Pick the shape by the requirement that forces it, not by the diagram that looks impressive, and most Azure GenAI projects get simpler.

Four shapes, and which requirement pushes you to each

The reason to name these four is that each one answers a different requirement, and naming the requirement first keeps you from over building. A chat architecture answers "a person needs to talk to the model and get a fast reply." Retrieval, usually called RAG for retrieval augmented generation, answers "the answer has to come from my documents, not the model’s training data." An agentic architecture answers "the model needs to take actions, call APIs, or coordinate steps, not just produce text." Batch answers "I have a huge pile of inputs and nobody is waiting on any single one." Read the requirement, and the shape is usually obvious.

Most real systems combine two of these. A support assistant is a chat architecture with retrieval bolted in. A document pipeline is batch with retrieval for classification. The mistake I see most is teams reaching for the agentic shape first, because it sounds like the future, when a retrieval chat app would have shipped in a week and answered the same questions. Complexity is a cost you pay every day in operations, so add an agent only when a tool call or a multi-step task actually needs one.

flowchart TD
  R[Driving requirement] --> Q{What drives it}
  Q -->|Simple Q and A| CB[Chatbot]
  Q -->|Needs your data| RG[Retrieval]
  Q -->|Multi step actions| AG[Agentic]
  Q -->|Bulk offline| BT[Batch]

The baseline chat architecture, component by component

Microsoft publishes a baseline reference architecture for enterprise chat, and it is the right starting point even if you never need the full network isolation, because it names every component you will eventually want. A user hits a chat UI hosted on Azure App Service, the managed web hosting platform. Traffic reaches App Service through Azure Application Gateway, a layer 7 load balancer, with Azure Web Application Firewall inspecting requests for common web attacks before they land. App Service calls a prompt agent hosted on Foundry Agent Service, the managed runtime that holds your agent’s instructions, model choice, and connected tools. The agent queries Azure AI Search for grounding, calls the Azure OpenAI model, and persists the full exchange to Azure Cosmos DB, a low latency document database, so the conversation survives across turns and sessions.

The security posture is the part worth copying. Every hop between components runs over a private endpoint, so traffic never touches the public internet, and outbound calls from the agent, like a web search tool, route through Azure Firewall which enforces which destinations are even reachable. Azure Key Vault holds the TLS certificates for the gateway. Microsoft Entra ID handles identity, and the agent authenticates to the model and to Cosmos DB with a managed identity rather than a stored key. Application Insights and Azure Monitor watch the whole thing. Here is the request path.

flowchart LR
  U[User] --> AGW[Application Gateway]
  AGW --> AS[App Service]
  AS --> AG[Foundry agent]
  AG --> SR[AI Search grounding]
  AG --> MD[Model]
  AG --> DB[(Cosmos DB state)]

There are three tiers of this architecture, and you should know which one you are copying. The basic tier drops the network isolation and is fine for a proof of concept or an internal demo. The baseline tier adds the private endpoints, firewall, and zone redundancy for production. The landing zone tier layers on enterprise governance for organizations that already run an Azure landing zone. Start at basic to prove the idea, then move to baseline the moment real users or real data show up. Skipping straight to the landing zone tier for a pilot is how a two week project becomes two months.

How does retrieval change the picture?

It barely changes the boxes and completely changes the answers. Retrieval augmented generation means you fetch relevant chunks of your own content and hand them to the model as context, so the answer is grounded in your documents instead of the model’s general training. In the Azure baseline this is Azure AI Search doing the work: it holds a chunked, vectorized index of your content, and the agent calls it as a tool at query time. AI Search does hybrid retrieval, combining keyword and vector search, then applies semantic ranking to reorder the top results by meaning. The model only sees the handful of chunks that survive that ranking.

The architectural weight in RAG is not the model, it is the indexing pipeline behind AI Search: how you chunk documents, which embedding model you use, and how often you reindex. I covered that pipeline in Part 12 and the data prep in Part 20, so I will not repeat it here. The point for this part is where it sits: AI Search is a first class component of the baseline chat architecture, not a separate system, and Foundry Agent Service can manage the index for files uploaded during a chat. Retrieval quality lives or dies in the index, and no amount of prompt tuning rescues a bad one.

Agentic architecture and multi-agent orchestration

An agent is a model that can call tools and decide what to do next, rather than only returning text. The Azure baseline already runs a single agent on Foundry Agent Service, so the step up to agentic is about adding tools and, eventually, more agents. Tools connect through two standard protocols: Model Context Protocol, MCP, for tools hosted as their own servers, and Agent to Agent, A2A, for handing work to another agent. From the calling agent’s view both are external HTTP endpoints, which means each one needs a firewall egress rule and its own authentication. That last sentence is the operational cost people forget when they sketch a clean agent diagram.

When one agent is not enough, you orchestrate several. Microsoft documents a set of orchestration patterns, sequential, concurrent, group chat, handoff, and magentic, and you can run simple ones inside Agent Service while more involved ones need self hosted orchestration with a framework like the Microsoft Agent Framework or Semantic Kernel, which I covered in Part 21. My guidance here is blunt: a single agent with good tools beats a committee of agents for almost every workload I have shipped. Multi-agent designs multiply the failure modes, the token bill, and the debugging surface. Reach for them when the tasks are genuinely parallel or need real separation of responsibility, and not before.

Batch inference when latency does not matter

Batch is the architecture teams skip and then wish they had built. When you have a large volume of inputs and no user waiting on any one result, classify a million tickets, summarize a document archive, extract fields from a warehouse of PDFs, you do not run them through the interactive endpoint one at a time. You use a global batch deployment. You write all your requests into a single JSONL file, upload it to Blob Storage, submit it as a batch job against the batch deployment, and collect the output file when the job finishes. Azure targets a 24 hour turnaround and charges 50 percent less than the global standard price for the same model, with a separate quota so batch work does not eat your interactive capacity.

The shape is an offline pipeline, not a service. A queue or a scheduled job builds the input file, submits the batch, and a second step reads results when they land. Two facts change how you write the consumer. Responses do not come back in the order you submitted them, so every line carries a custom id you join on. And the 24 hour figure is a target, not a guarantee, so the pipeline waits and retries rather than assuming a fixed completion time. Here is a minimal submission you can adapt.

from openai import AzureOpenAI
client = AzureOpenAI(azure_endpoint=ENDPOINT, api_version='2025-04-01-preview',
                     azure_ad_token_provider=token_provider)

# 1. upload the JSONL of requests; each line has a unique custom_id
batch_file = client.files.create(file=open('requests.jsonl', 'rb'), purpose='batch')

# 2. submit against the GLOBAL BATCH deployment, not a standard one
job = client.batches.create(input_file_id=batch_file.id,
                            endpoint='/chat/completions',
                            completion_window='24h')

# 3. poll until terminal, then download the output file
import time
while job.status not in ('completed', 'failed', 'cancelled', 'expired'):
    time.sleep(60)
    job = client.batches.retrieve(job.id)
print(job.status, job.output_file_id)

Expected result: status moves through validating, in_progress, then completed, and output_file_id names a JSONL you download and join to your inputs on custom_id. Failure mode: point the job at a standard deployment instead of a global batch one and it is rejected, because batch requires a deployment created with the global batch type. A whole file also fails validation if any line is malformed, so validate the JSONL before you upload it, not after a wasted hour.

Put a gateway in front when apps multiply

One app calling one model needs no gateway. The day a second or third app shares the same model deployments, you want Azure API Management sitting in front as an AI gateway, because the model quota is measured in tokens per minute and one runaway app will starve the rest. The gateway gives you a token limit policy that caps tokens per minute or a longer quota per consumer, enforced at the gateway so you are not paying for requests it rejects. It load balances across several model backends with round robin, weighted, priority, or session aware strategies, so you can send most traffic to a reserved provisioned throughput backend and spill overflow to pay as you go ones. A circuit breaker pulls an unhealthy backend out of rotation and honors the Retry-After header when it comes back.

Two more capabilities earn the gateway its place. Semantic caching stores previous completions and serves a cached answer when a new prompt is close enough in meaning, cutting model calls and cost, backed by Azure Managed Redis. And token metrics emitted per consumer give you one place to see who is spending what, which is the report finance always asks for and nobody can produce without a gateway. This policy shows the two pieces most teams start with, a per subscription token cap and a load balanced backend.

<inbound>
  <!-- cap each consumer so one app cannot drain the shared TPM quota -->
  <llm-token-limit counter-key='@(context.Subscription.Id)'
      tokens-per-minute='500' estimate-prompt-tokens='false'
      remaining-tokens-variable-name='remainingTokens' />
  <!-- emit per consumer token metrics to Application Insights -->
  <llm-emit-token-metric namespace='llm-metrics'>
    <dimension name='Subscription' value='@(context.Subscription.Id)' />
  </llm-emit-token-metric>
  <!-- route to a load balanced pool of model backends -->
  <set-backend-service backend-id='model-pool' />
</inbound>

Expected result: each subscription is held to 500 tokens per minute, token usage shows up in Application Insights split by subscription, and requests spread across the backends defined in the model-pool load balancer. Failure mode: set the limit far below real demand and legitimate users get throttled, so derive the number from measured usage before you enforce it, and start with metrics only until you trust the figure.

Size the baseline before you build it

Before you provision anything, turn the requirement into three numbers: peak requests per minute, tokens per request, and the latency your users will tolerate. Those three decide almost everything downstream. Peak requests times tokens per request gives the tokens per minute you must provision on the model deployment, and the latency tolerance decides whether any of the work can move to the batch path. In the worked example above, 40 requests per minute at 2,000 tokens each set an 80,000 token per minute floor, which is why the deployment was sized at 100,000. Get those three numbers wrong and every other choice inherits the error.

This table maps the four patterns to the numbers that drive them, so you can price a design on paper before you deploy a single resource. Treat the cost index as relative, anchored to the interactive standard path at 100.

The AWS half of this series lays out the same four shapes on Bedrock, and the parallels are close enough to lift if you run both clouds. See Bedrock reference architectures for chatbot, RAG, agentic, and batch for the Amazon version, and the guide for where each Azure piece connects.

Start with the baseline chat shape, add only what a requirement forces

If you take one thing from this part, make it the sequencing. Stand up the baseline chat architecture first, App Service behind Application Gateway, a single prompt agent on Foundry Agent Service, Cosmos DB for state, all on private endpoints. Add Azure AI Search the moment answers must come from your own content, which for most business apps is immediately. Add agents and tools only when the model has to take an action, not just answer, and prefer one well equipped agent to a crowd of them. Split off a global batch path for any high volume offline work, because half price for a 24 hour wait is a trade most bulk jobs will take. And introduce Azure API Management as an AI gateway when a second app starts sharing your model quota.

When this is the wrong amount of architecture: a single internal tool with ten users does not need private endpoints, a firewall, or a gateway. Ship the basic tier and move up only when real data or real scale arrives. What to validate first: write down your peak requests per minute, tokens per request, and latency tolerance, then size the model deployment from those three numbers before you provision anything, so your first bill is not a surprise.

Your move this week: take one workload, decide which of the four shapes it is, and sketch it with real numbers for load, tokens, and latency. A named shape with three honest numbers beats a whiteboard full of boxes. Part 30 closes the series by weighing Azure GenAI against the field and giving the verdict.

References

• Baseline Microsoft Foundry chat reference architecture, Azure Architecture Center
• AI gateway capabilities in Azure API Management, Microsoft Learn
• Global batch processing with Azure OpenAI, Microsoft Learn
• AI agent orchestration patterns, Azure Architecture Center
• Related: Azure AI Search and RAG (Part 12) and Azure AI Foundry Agent Service (Part 13)

About The Author

Dr. Pranay Jha

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

See author's posts