Key takeaways
- An open model on Azure Machine Learning runs on managed compute, a dedicated GPU virtual machine you rent by the hour, not a token meter. You own the box while the deployment is alive.
- The model catalog offers the same weights two ways. Serverless bills per token and Microsoft runs the box. Managed compute bills per core hour and you run the box.
- Open weights from the Hugging Face registry download to your endpoint at deploy time. The weights are not stored on Azure, and the registry is not covered by Microsoft support.
- Managed compute only beats serverless once utilization is high and steady, or when you need a model or an isolation guarantee serverless does not offer.
Here is the line that separates this part from the seventeen before it: azureml://registries/HuggingFace/models/Meta-Llama-3.1-8B-Instruct/labels/latest. With Azure OpenAI you called a model Microsoft was already running for you, and you paid per token. Point a deployment at a registry string like that one and something else happens. Azure provisions a GPU virtual machine in your subscription, pulls the weights onto it, and starts charging by the hour the moment the endpoint reports healthy, whether or not a single request has arrived. That is the whole trade of open models on Azure ML. You get control and isolation, and you take on the meter that never sleeps.
What open models mean on Azure ML
An open model, sometimes called an open weight model, is one whose trained parameters you can download and run yourself. Llama, Mistral, Phi, and the thousands of transformer models on the Hugging Face hub are open in this sense. That is different from Azure OpenAI models, where you never see the weights and only ever call an API. The Foundry model catalog, which now lists more than 1,900 models, carries both kinds side by side, and the filter that matters here is deployment option.
Two of those options put the model on hardware you control. Managed compute deploys the weights to an Azure Machine Learning managed online endpoint, which is a virtual machine, or a set of them, provisioned on your behalf. The endpoint hands you a REST API and takes care of serving, scaling, and monitoring, so you are not building a container host from scratch. The open models from Hugging Face show up in the catalog through a community registry that Hugging Face itself maintains, and their weights are not hosted on Azure at all. They download straight from the Hugging Face hub onto your endpoint when the deployment runs. Azure ML is the catalog and the plumbing, not the warehouse.
Managed compute or serverless, which one owns the box
This is the decision that drives everything else, so pin it down before you write a line of code. With a serverless deployment, Microsoft hosts the model in its own infrastructure and gives you an API. You are billed for input and output tokens, the box scales itself, and you never think about a virtual machine. With managed compute, the deployment lives on virtual machines in your subscription, you are billed for the core hours those machines run, and idle time still costs money. One is a taxi meter, the other is a leased car.
The reason to choose managed compute is rarely price at low volume. It is control. Your prompts and completions stay on infrastructure you can wrap in a managed network, you can pick the exact open model you want including ones no provider offers serverless, and you get a stable box for latency-sensitive traffic. The catch is that content safety is not wired in for you the way it is on serverless language models. On managed compute you call the Azure AI Content Safety APIs yourself, and you are billed for that separately. Nothing screens output unless you build the screening.
| Dimension | Managed compute | Serverless |
|---|---|---|
| What you rent | GPU or CPU virtual machines | An API, no machine |
| Billing unit | Core hours, idle included | Tokens in and out |
| Batch inference | Yes, except Hugging Face registry | Batch option on select models |
| Content safety | You call the APIs yourself | Default filters integrated |
| Network isolation | Managed network on the hub | Follows workspace access flag |
| Support for HF models | Community registry, not Microsoft support | Varies by provider |
My take
Start on serverless for any open model that offers it, and treat managed compute as the deliberate upgrade you make for a reason you can name. Data isolation, a model no one hosts serverless, or sustained high traffic are good reasons. A vague wish to own the stack is not, because the endpoint you forget to delete on a Friday bills all weekend at GPU rates. I have paid that tuition.
From catalog entry to a live endpoint
The path is short once you know the pieces. You find a model in the catalog, copy its registry asset id, create a managed online endpoint, then create a deployment on that endpoint that names the model and an instance type. The instance type is the SKU, the virtual machine size, and it must be one you hold quota for. The catalog quick-deploy dialog filters the SKU list down to sizes the model fits in, which saves you from guessing whether a model lands in memory. One deployment can back a single endpoint, or you can run several deployments behind one endpoint and split or mirror traffic between them for safe rollout.
For Hugging Face models the asset id points at the community registry, and the format is fixed. You reference either an explicit version or a moving label. A model must meet a few conditions to deploy cleanly: it needs a Transformers, Diffusers, or Sentence-Transformers tag, a supported task such as chat completion or embeddings, weights in the Safetensors format, and it must not require remote code. Miss any of those and the deployment fails with an error that reads like a Python traceback rather than an Azure message, which is your first clue that you are in community-registry territory now.
Deploying an open model with the SDK
Here is a full deployment in the Python SDK. It creates an endpoint, deploys one open model to a single A100 virtual machine, sends all traffic to it, and prints the scoring URI you would call. The endpoint name gets a timestamp because names must be unique per region.
from azure.ai.ml import MLClient
from azure.ai.ml.entities import ManagedOnlineEndpoint, ManagedOnlineDeployment
from azure.identity import DefaultAzureCredential
import time
ml_client = MLClient(
credential=DefaultAzureCredential(),
subscription_id='SUB_ID',
resource_group_name='RG_NAME',
workspace_name='WS_NAME',
)
# open model pulled from the HuggingFace community registry
model_id = 'azureml://registries/HuggingFace/models/Meta-Llama-3.1-8B-Instruct/labels/latest'
name = 'oss-ep-' + str(int(time.time())) # unique per region
ml_client.online_endpoints.begin_create_or_update(
ManagedOnlineEndpoint(name=name)
).wait()
ml_client.online_deployments.begin_create_or_update(
ManagedOnlineDeployment(
name='blue',
endpoint_name=name,
model=model_id,
instance_type='Standard_NC24ads_A100_v4',
instance_count=1,
)
).wait()
ep = ml_client.online_endpoints.get(name)
ep.traffic = {'blue': 100}
ml_client.online_endpoints.begin_create_or_update(ep).result()
print(ep.scoring_uri)Expected output: a scoring URI like https://oss-ep-1751...azureml.online... printed after several minutes, since pulling weights and warming a GPU is not instant. Failure mode: no quota for the A100 SKU and the deployment stops before it starts, so raise quota first. Second failure mode: the model asset name is wrong or gated, and you get a KeyError or a not-found rather than a clean message. [VERIFY: exact Hugging Face registry asset name for the current Llama 3.1 8B Instruct listing]
Sizing the GPU and finding the quota
Two numbers decide the SKU: how much GPU memory the model needs to load and run, and how much throughput you want. Weights alone take roughly two bytes per parameter at 16-bit precision, so an 8 billion parameter model is about 16 GB before you add the key-value cache that grows with context length and concurrency. That is why an 8B model fits comfortably on a single A100 with 80 GB, and a 70B model does not, and needs either quantization or several GPUs. The catalog dialog hides this math by pre-filtering SKUs, but knowing it tells you when a cheaper card is a false economy.
Quota is the wall people hit first. Managed compute needs virtual machine quota for the specific GPU family in your subscription and region, and fresh subscriptions start at zero for the big cards. Some catalog models let you deploy to a temporary shared quota pool for a quick test, which is the fastest way to try a model without a quota request, but it is for testing, not production. For anything real, raise a quota request for the GPU family before you plan the rollout, because approval is not instant and a demo with no quota is a demo that does not happen.
| SKU | GPUs | GPU memory | Typical fit |
|---|---|---|---|
| Standard_NC24ads_A100_v4 | 1x A100 | 80 GB | 7B to 13B chat models |
| Standard_NC48ads_A100_v4 | 2x A100 | 160 GB | Higher concurrency or longer context |
| Standard_NC96ads_A100_v4 | 4x A100 | 320 GB | Around 70B with headroom |
| Standard_ND96isr_H100_v5 | 8x H100 | 640 GB | Largest open models, top throughput |
Worked example
Put an 8B chat model on one Standard_NC24ads_A100_v4 and run it around the clock. At an assumed 3 dollars an hour, that is about 3 times 720, near 2,160 dollars a month, flat, whether you serve one request or a million. Now price the same traffic serverless at an assumed blended 0.30 dollars per million tokens. The two costs meet when serverless tokens reach 2,160 divided by 0.30, which is 7,200 million, about 7.2 billion tokens a month. Below that, serverless is cheaper and you skip the ops. Above it, the always-on GPU pulls ahead and keeps pulling as volume climbs, because its cost stops growing while the token bill does not. [VERIFY: current Standard_NC24ads_A100_v4 hourly rate and the serverless token rate for your chosen model]
Gated models and the code you cannot run
Two walls trip up first deployments from the Hugging Face registry. The first is gated models, the ones whose authors make you request access and accept terms before you can pull the weights. To deploy one you create a custom key connection named HuggingFaceTokenConnection holding your Hugging Face token as a secret under the key HF_TOKEN, then create the endpoint with secret store access enabled so the token injects at deploy time. Skip that and the deployment fails with a KeyError, because the weights never came down.
The second wall is by design. Models that run custom code from their own repo, the ones that need trust_remote_code set to true, are not supported on managed compute, and the deployment refuses them on purpose. Running arbitrary code pulled from a model repo on your GPU is exactly the supply-chain risk Azure will not take for you. If a model you want needs remote code, that is a signal to package it yourself as a custom container rather than force it through the catalog path. Missing tokenizers, non-Safetensors weights, and models that need extra Python libraries fail the same way, with a traceback rather than a portal error.
What managed compute leaves on your plate
Serverless hides a long list of chores that come back once you own the box. Content safety is the big one, since managed compute does not screen output, so you route responses through the Azure AI Content Safety APIs and pay for that call. Scaling is yours to tune, though the platform gives you autoscale on utilization metrics or a schedule, so an endpoint can add nodes past 70 percent GPU use or scale up for business hours and down overnight. Batch inference from the Hugging Face registry is simply not offered today, so a nightly bulk job needs a different path. And the weights come from outside Azure, which means a model can be updated or pulled upstream in ways a Microsoft-hosted model would not be.
None of this makes managed compute a bad choice. It makes it a choice with an operations bill attached, and the bill is worth paying when isolation, model selection, or steady scale justify it. If you have run open models on other clouds, the shape rhymes: AWS puts the same idea behind SageMaker JumpStart and private hubs, which I cover in the SageMaker JumpStart part, and the trade there is the same one, managed convenience against owning the endpoint. Next in this series, Part 19 takes the single endpoint you just built and scales it into distributed training on ND clusters, where the quota and networking lessons here start to compound.
Where I reach for open weights, and where I don’t
My rule is short. If a capable model is available serverless, I start there and stay there until a real constraint pushes me off. The constraints that push me are concrete: data that cannot leave an isolated network, a specific open model no one hosts serverless, or traffic steady and heavy enough that the always-on GPU comes out cheaper on the break-even math from earlier in this part. When one of those is true, managed compute is the right tool and the operations bill is money well spent. When none of them is true, reaching for open weights on your own GPU is buying a leased car to sit in a garage.
So price it before you build it. Take your real monthly token volume, your target GPU SKU, and the serverless rate for a comparable model, and find your own crossing point. If your volume sits well above it and stays there, deploy the endpoint and wire the content safety and autoscale you now know you owe. If it sits below, book the savings and revisit when traffic grows. Your next step: open the catalog, filter to managed compute, and deploy one small open model to the shared quota pool so the endpoint, the registry id, and the SKU stop being abstract.
References
- Deploy Hugging Face hub models to Azure ML online endpoints
- Explore Microsoft Foundry Models in Azure Machine Learning
- Deploy models to managed compute with Foundry
About The Author
DrJha