Open the Azure portal, type ‘AI’ in the search bar, and you get back a wall of services. Azure AI Foundry, Azure OpenAI, Azure AI Search, Azure Machine Learning, Content Safety, a dozen more. I have watched capable teams stall for a week just deciding which door to walk through. This first part draws the map so the other 29 parts have somewhere to hang. I will name every service, say what it does in one plain clause, and show how a single chat request travels through the whole thing.
TL;DR
Azure AI Foundry (Microsoft has begun calling it Microsoft Foundry) is the front door: one place to find a model, deploy it, ground it on your data, wrap it in an agent, and watch it in production. Under Foundry sit the models (Azure OpenAI plus a catalog of open and partner models), three deployment types that decide your bill (Standard, Global Batch, Provisioned), a retrieval layer (Azure AI Search), an agent runtime (Foundry Agent Service), safety (Content Safety), and the raw compute (ND-series GPU VMs and Microsoft’s own Maia accelerators). Learn the six layers once and the rest of this series is detail.
What sits in the Azure GenAI stack
Start with the word that trips everyone up. Azure AI Foundry is a platform, not a single model or a single API. It is the studio where you browse models, the SDK you call from code, and the management plane that holds your deployments, indexes, and agents together. Microsoft renamed the classic Azure AI Studio to Foundry and folded Azure OpenAI into it, so you will see both names in the docs for a while. When I say Foundry in this series, I mean that unified surface.
Everything else is a layer you reach through or alongside Foundry. A model is the thing that generates text or images. A deployment is a named, billable instance of that model in your resource. An index is your data, chunked and embedded so a model can retrieve it. An agent is a model plus tools plus memory that can take multi-step actions. Guardrails are the filters that sit in front of the model to block unsafe input and output. Keep those five nouns straight and the portal stops looking like alphabet soup.
Here is the blunt version I give new teams. You do not need most of these services on day one. You need a Foundry resource, one model deployment, and a way to call it. The retrieval layer, the agent runtime, and the custom compute come later, when a real requirement forces them. Build the smallest thing that answers a question, then add layers as the pain appears.
Six layers, from silicon to agent
The layer diagram is the spine of the whole series, so let me walk each one with the service that owns it and the part number where I go deep. This table is the index you will come back to.
| Layer | Main service | What it does | Deep dive |
|---|---|---|---|
| Portal and SDK | Azure AI Foundry | Find, deploy, evaluate and manage models and agents | Part 2 |
| Agents | Foundry Agent Service | Run a model with tools, memory and identity | Part 13 |
| Retrieval | Azure AI Search | Index your data and feed it to the model at query time | Part 12 |
| Models | Azure OpenAI and the catalog | Generate text, images, speech and embeddings | Parts 3 to 6 |
| Safety | Content Safety, Entra ID | Filter unsafe I/O, control who can call what | Parts 10 and 14 |
| Compute | ND-series VMs, Maia | The GPUs and accelerators that run the math | Part 7 |
Notice how few of these you touch to ship a first prototype. A Foundry resource and a model deployment cover it. The retrieval and agent layers are what turn a demo into something a business trusts, and that is where most of the real work in this series lives.
Where do the models come from?
The model catalog inside Foundry is the shelf you shop from. Microsoft groups it into two buckets, and the split matters for billing and support. The first bucket is Foundry Models sold by Azure. That includes every Azure OpenAI model plus a set of hand-picked models from providers such as Meta, Mistral, Cohere, and NVIDIA. Azure bills you directly, backs the SLA, and runs the infrastructure. The second bucket is Foundry Models from partners and community, thousands of open models from the Hugging Face ecosystem and others that you deploy onto your own compute or a managed endpoint.
The Azure OpenAI models are the ones most teams start with. The frontier GPT models carry very large context windows (Microsoft advertises up to a million tokens on the top tier [VERIFY]) and there are smaller, cheaper variants for high-volume or latency-sensitive work. There are also speech models for transcription and voice, and image models for generation and editing. I will not pin exact version strings here because they roll forward every few months, and a version I name today will be wrong by the time you read a later part. Part 3 covers how to read the catalog and pick a model without chasing the newest label.
My take after doing this for a living: pick the smallest model that clears your quality bar, then only move up when an evaluation says you must. Teams reach for the biggest model by reflex and pay for it on every single call. The catalog makes that reflex expensive.
Deployment types decide your bill
Choosing a model is half the decision. The other half is the deployment type, and it drives cost more than almost anything else you do. Azure gives you three, and they trade latency, throughput guarantees, and price against each other.
Standard is pay-per-token, best-effort service. You send a request, you get a response, you are billed for the tokens. It is right for development, spiky traffic, and anything where an occasional queue is fine. Global Batch takes a file of requests and returns results within a 24-hour target window, and Microsoft prices it at roughly half the cost of Global Standard. Use it for overnight jobs: bulk summarization, classification, embedding a back catalog. Provisioned is the reserved tier. You buy a fixed number of Provisioned Throughput Units, or PTUs, and pay an hourly rate whether or not you use them. In return you get predictable latency and reserved capacity that does not compete with anyone else. Provisioned comes in global, data zone, and regional flavors depending on where you need the traffic to land.
| Deployment type | Billing | Latency | Best for |
|---|---|---|---|
| Standard | Per token, best effort | Variable | Dev, prototypes, spiky traffic |
| Global Batch | Per token, about 50% of Standard | Up to 24 hours | Bulk offline jobs |
| Provisioned (PTU) | Hourly per PTU, fixed | Predictable, reserved | Steady high-volume production |
The chart below shows the shape of the trade-off. Batch and Standard cost scales with how many tokens you push. Provisioned is a flat hourly line that does not care about volume, which is exactly why it wins once your traffic is high and steady, and loses badly when it is not.
Worked example
Say a support assistant handles 200,000 chats a month, each about 1,500 input tokens and 500 output tokens. That is 400 million tokens a month, steady through business hours. On Standard you pay for every token at list rate. If a right-sized PTU commitment covers your peak of roughly 12 chats a second and costs less per hour than the equivalent token spend at that rate, Provisioned wins and gives you flat, predictable latency on top. If the same assistant only runs during a nightly report, Global Batch at half the token price beats both. Same workload, three different right answers, all decided by the traffic shape. I plot these numbers in Figure 3.
How a request actually flows
Layers are easier to trust once you see one request move through them. A grounded chat call, the kind that answers from your own documents, touches most of the stack in a fraction of a second. Your app authenticates with Microsoft Entra ID and gets a token. It sends the user question to Azure AI Search, which retrieves the most relevant chunks from your index. Your app packs those chunks into the prompt and calls the model deployment. Content Safety screens the input and the output. The model returns an answer grounded in your data, and Azure Monitor logs the whole trip. That is retrieval augmented generation, and Part 12 builds it properly.
You can see your deployments from the command line, which is the fastest way to check what a resource actually holds. The snippet below lists every deployment in a Foundry resource and its SKU, so you can tell Standard from Provisioned at a glance.
az cognitiveservices account deployment list
--name my-foundry-resource
--resource-group rg-ai-prod
--output table
# Expected output
# Name Model Sku.Name Capacity
# gpt-chat gpt-4 family Standard 30
# gpt-batch gpt-4 family GlobalBatch 50
# gpt-prod gpt-4 family ProvisionedManaged 100Failure mode: if the resource or group name is wrong you get a ResourceNotFound error, and if your account lacks the Cognitive Services role you get AuthorizationFailed. Both are permission and naming issues, not model issues, and Part 10 covers the RBAC side.
Compute underneath: ND GPUs and Maia
When you call a serverless model you never see a GPU, and that is the point. But it helps to know what is down there, because the compute layer shows up the moment you train your own model or need dedicated hardware. Azure runs generative AI on two kinds of silicon. The ND-series virtual machines carry the datacenter GPUs, historically NVIDIA parts and more recently AMD Instinct MI300X in the ND MI300 line. These are what you rent when you fine-tune an open model or run a large training job.
The second kind is Microsoft’s own. Maia 100, announced in late 2023, is the first AI accelerator Microsoft designed in house for training and inference in Azure. It carries 64 GB of HBM2E memory and about 1.8 TB per second of bandwidth on a large TSMC N5 die, and Microsoft runs its own Copilot and Azure OpenAI traffic on it. A second generation is already in design. You do not pick Maia directly today, it sits under the managed services, but it is why Azure can keep frontier models fed at the scale it does. Part 7 puts the GPU and accelerator options side by side.
Where Azure sits next to the other clouds
This series is the Azure lens on purpose. If you want the vendor-neutral foundations, how tokens and embeddings and RAG work regardless of cloud, that lives in the general GenAI series linked below. And if you are weighing Azure against AWS, the AWS Bedrock stack maps onto these same six layers with different names: Bedrock plays the Foundry role, SageMaker plays Azure ML, Titan and Nova play the first-party model role. The shape is the same across all three big clouds. What differs is the model catalog, the pricing knobs, and how opinionated each platform is about agents. I keep pointing at those differences as we go, because knowing the shape is common makes each vendor easier to learn.
One honest caveat before we move on. Azure’s naming churn is real and it is annoying. Between the Foundry rename, the classic portal, and the government variants, you will hit docs that disagree with each other. When they do, trust the newest Foundry page and the API version string over any blog, including older parts of this one. I flag version-sensitive claims as we go. See the AWS Gen AI series for the same six layers under Bedrock.
Start with one resource and one deployment
Here is the recommendation, not a shrug. If you are new to Azure generative AI, create a single Foundry resource, deploy one mid-tier Azure OpenAI model on the Standard type, and call it from a small script. Do not stand up Azure AI Search, an agent, private networking, or a PTU commitment until a concrete requirement asks for it. Standard is cheap to experiment on because you only pay for the tokens you actually send, and it teaches you the request path in an afternoon. Add the retrieval layer when your answers need to come from your own documents. Add PTUs when a real traffic pattern shows steady, high volume and Figure 2 tells you the flat line wins. Add the agent runtime when a single model call stops being enough and you need tools and multi-step work.
That order is deliberate. Every layer you add is a thing to secure, monitor, and pay for. Adding them in response to need, rather than up front, is the difference between a bill you understand and one you dread. Part 2 opens the Foundry portal and does exactly this first deployment step by step.
Your move: open the portal, find the model catalog, and just look at what is on the shelf. You do not have to deploy anything yet. Seeing the catalog is the first real step, and it makes Part 2 click.
References
- Microsoft Foundry Models overview
- Understanding deployment types in Microsoft Foundry Models
- Azure Maia for the era of AI
- What is Microsoft Foundry Agent Service
About The Author
DrJha