A hosted model API bills you for tokens. A self hosted one bills you for a GPU that costs the same whether it answered ten requests last hour or ten thousand. That single change, from paying per token to paying per hour of silicon, is the reason Red Hat AI exists, and it is why standing up your own inference is a platform decision long before it is a model decision.
Inference just means running a trained model to get an answer, as opposed to training it in the first place. Nobody in this series trains a foundation model. What we do instead is take an open weight model somebody else trained, put it on hardware we control, and make it answer questions reliably, cheaply and safely. Red Hat sells three products for that job, they overlap in confusing ways, and the first useful thing I can do is tell you which one you actually need.
Key takeaways
Red Hat AI is not one product. It is three: RHEL AI for a single server, OpenShift AI for a cluster, and the Red Hat AI Inference Server as the shared engine underneath both.
All three run the same open source vLLM engine and the same open weight Granite models. What you pick decides how much operational machinery sits around that engine, not what the model can do.
The open source thesis is that your model, your serving stack and your data stay portable across your own hardware and any cloud. Red Hat adds hardening, support and a tested path, not a proprietary model.
Most first projects belong on a single RHEL AI box, not a cluster. The vendor funnel points at OpenShift AI too early.
Before choosing anything, count two numbers: how many models you will serve, and your peak concurrent requests.
What Red Hat AI actually is
In October 2025 Red Hat put a single banner, Red Hat AI 3, over three products that had been shipping separately. Reading the announcement you could be forgiven for thinking Red Hat AI is one thing you install. It is not. It is a family, and the members solve the same problem at three different scales. Underneath all of them sits vLLM, an open source inference engine from the vLLM project that serves models efficiently by packing many requests onto a GPU at once. Red Hat did not write vLLM. What Red Hat ships is a tested, supported, security patched path to running it, plus the Granite models, IBM and Red Hat open weight models released under the Apache 2.0 licence, that you can tune and redistribute without asking anyone.
So the three products are best read as three amounts of surrounding machinery. Red Hat Enterprise Linux AI, RHEL AI, is a bootable operating system image with vLLM, the Granite models and InstructLab tuning tools baked in, meant for one server. Red Hat OpenShift AI is a platform layer on top of OpenShift, which is Red Hat Kubernetes, for running the same workloads across a cluster of machines with scheduling, pipelines and multi tenancy. The Red Hat AI Inference Server is just the hardened vLLM engine on its own, shipped as a container you can drop onto either of the above or onto a Linux box you already run. One engine, three delivery sizes.
| Product | What it is, in one line | Upstream it builds on | What Red Hat adds | Reach for it when |
|---|---|---|---|---|
| RHEL AI | A bootable Linux image that serves and tunes a model on one server | RHEL, vLLM, InstructLab, Granite | One tested image, support, a tuning workflow that runs offline | One team, one or two models, modest concurrency |
| OpenShift AI | A cluster platform for training, serving and governing many models | OpenShift, KServe, Kubeflow, vLLM | Scheduling, GPU sharing, pipelines, registry, tenancy, support | Many teams or models, autoscaling, shared GPUs |
| AI Inference Server | The hardened vLLM engine as a standalone container | vLLM, plus Neural Magic compression work | Security patched builds, LLM Compressor, an optimised model repo | You already have Linux or Kubernetes and only need serving |
Open source thesis, and where Red Hat adds value
Every argument for this stack reduces to one claim: the model, the engine that serves it and the data it reads should all stay yours, and yours means movable. A hosted API gives you none of that. You rent access to a model you cannot inspect, on hardware you cannot see, under terms that can change the model under you between Tuesday and Wednesday. For a support assistant answering staff questions that is often fine. For a bank, a hospital or a company whose contracts forbid sending customer data to a third party, it is a non starter, and that constraint, not performance, is what drives most teams to self host.
Red Hat did not invent the open pieces and does not pretend to. vLLM is a Berkeley project. Kubernetes and KServe, the component that turns a model into a running service on a cluster, are community efforts. Granite is IBM and Red Hat, but released openly. What you pay Red Hat for is the unglamorous middle: a build of vLLM that has been security scanned and will get a patch when a CVE lands, a version of the Granite models that has been validated on named accelerators, and a support contract so that when serving falls over at 2am there is a number to call. That is the honest value proposition, and it is the same one Red Hat has sold for Linux for twenty years. Self hosting like this is the far end of the portability argument I made in the AI Engineering series on choosing and switching providers without a rewrite: instead of swapping one hosted vendor for another, you own the endpoint outright.
Where common advice is wrong
Red Hat AI as a name implies one product, and the sales motion points you at OpenShift AI first because a cluster platform is the larger deal. For a first model with a handful of users that is the wrong starting point. A single RHEL AI server, or the Inference Server on a Linux box you already run, serves the same Granite model and gives the same answers with a fraction of the operational cost. Moving up to OpenShift AI is a scaling decision you make when concurrency, many models or shared GPUs force it, not a choice you make on day one.
Dividing one problem across RHEL AI, OpenShift AI and the Inference Server
Picture one request travelling through the stack. A staff member asks the assistant a question. The application assembles a prompt and sends it to an inference endpoint. Behind that endpoint sits vLLM, running a Granite model on a GPU. Everything Red Hat sells is about where that endpoint lives and how much sits around it. On RHEL AI the endpoint is a process on one server you booted from an image. On OpenShift AI the same endpoint is a KServe service that the cluster can scale, schedule onto shared GPUs and place behind tenancy rules. The Inference Server is the engine common to both, and you can also run it alone.
Read the diagram from the bottom and the point lands: the Granite model and the vLLM engine are identical no matter which product wraps them. That is why moving between them later is a migration of operations, not of behaviour. Your model answers the same. What changes is whether you are managing one box by hand or letting a cluster place, scale and share work for you. The positioning below plots that trade off directly, control and scale against operational effort, with a hosted API included as the thing you are leaving behind.
Where the support assistant starts, and where it ends up
One project runs the length of this series so that each part builds on the last rather than restarting. Our employer is a mid sized company that cannot send its data to a hosted API, for the ordinary reason that its contracts forbid it. It wants an internal assistant that answers staff and customer questions from three sources: the product documentation, an archive of resolved support tickets and a changelog. That is a retrieval problem at heart, retrieval meaning the assistant looks up relevant text and answers from it rather than from memory, and the GenAI explainer on what RAG is covers the idea we will lean on from Part 27.
The assistant starts small and grows the way a real one does. In Phase 2 it is a single Granite model on one RHEL AI server, tuned on the company docs with InstructLab, InstructLab being Red Hat and IBM tool for improving a model from example questions and answers rather than raw retraining. In Phase 3 it moves onto OpenShift AI so several teams can share GPUs and ship their own versions. In Phase 4 it is served through the Inference Server and llm-d, a project for spreading one large model across several machines, so it stays fast and cheap under load. In Phase 5 it gets guardrails, retrieval grounding and a GPU budget somebody has to defend. Standing up the serving path is the same problem the Data Science series framed in serving machine learning models, now specialised to large language models on Red Hat.
What self hosting costs you
Owning the endpoint is not free, and the costs are different in kind from an API bill, not just smaller or larger. Five habits change the day you stop renting inference. Read the right column as the work you are signing up for.
| With a hosted API | When you host it yourself on Red Hat |
|---|---|
| You pay per token, and an idle app costs nothing | You pay per GPU hour whether it serves one request or none, so utilisation becomes the number that matters |
| Capacity is the provider problem | You size the accelerator, and a model that does not fit fails to start rather than queueing |
| Model updates arrive automatically | You choose when to move, which is a feature for stability and a chore for security |
| Your data leaves your network on every call | Your data never leaves, which is usually the entire reason you are here |
| Latency is their published SLA | Latency is yours to tune, through batching, quantisation and hardware choice |
Row one deserves a moment because it inverts the economics people arrive with. A hosted API rewards low, spiky traffic, since you only pay when you call it. A self hosted GPU rewards steady, high traffic, since the machine costs the same all day and cheap inference comes from keeping it busy. There is a crossover point in requests per day below which renting is simply cheaper, and pretending otherwise is how self hosting projects lose their budget in month two. Phase 5 puts real numbers on that crossover.
Choosing an entry point
Here is the decision I wish someone had handed me, reduced to the two numbers that actually decide it: how many models you will serve, and your peak concurrent requests. Everything else is detail you can change later. Match your situation to a row and start there.
| Your situation | Start here | Why |
|---|---|---|
| One team, one or two models, tens of users | RHEL AI on one server | A cluster buys you nothing here and costs weeks of setup |
| You already run Linux and only need a private endpoint | Inference Server, standalone | You get portable serving without adopting a new operating system |
| You already run OpenShift and only need serving | Inference Server on the cluster | The engine slots into what you have without the full AI platform |
| Many teams or models, autoscaling, shared GPUs, tenancy | OpenShift AI | This is the one problem the cluster platform is actually built for |
A word on versions, because Red Hat product numbers move and this series states what it tested against. As of mid 2026 the relevant baselines are RHEL AI 1.5, which carries RHEL 9.4, Python 3.11 and the Granite 3.1 starter models, and OpenShift AI in its 3.x line, which needs OpenShift 4.19 or later. Granite 4.0 models are shipping in the model catalogue. Where a later part shows a command, it names the version it ran against, since a flag that works on one release can vanish on the next.
Start on one server, move when concurrency forces it
If you take one thing from this part, take the order: model portability first, cluster machinery last. The Granite model and the vLLM engine you run are identical across all three products, so the choice between them is purely about how much operational weight you want to carry now. Start on the smallest thing that fits your load, prove the assistant is useful, and let real concurrency, not a slide, tell you when to graduate to OpenShift AI. Teams that pick the cluster first spend their first month operating Kubernetes instead of improving answers, and the answers are the point.
Your action for Monday costs a spreadsheet cell and a conversation. Write down the two numbers, how many models you truly need to serve and your realistic peak concurrent requests, and match them to the entry point table above. If both are small, RHEL AI on one server is your platform and Phase 2 is where you begin. Part 2 picks the actual Granite size for the assistant and explains the Apache 2.0 licence that lets you tune and ship it, which is the first real decision once you have chosen where to run.
References
- Introducing Red Hat AI, the official overview of how the three products relate
- Red Hat AI Inference Server technical deep dive, on the hardened vLLM engine
- RHEL AI 1.5 release notes, the single server product and its Granite and InstructLab versions
- OpenShift AI 3.0 release notes, the cluster platform and its OpenShift version floor
- vLLM documentation, the upstream engine all three Red Hat products build on
About The Author
