A compliance officer once stopped a project I was on with a single sentence: show me, on a diagram, every place a customer support ticket touches disk. We could not draw it, and the pilot slipped a quarter. That question, not a benchmark score, is what decides where Red Hat AI runs.
Where Red Hat AI can physically run
Red Hat sells the hybrid cloud idea hard, so it helps to say what it means in concrete terms. Red Hat AI is a single software stack that can land on four kinds of surface: a bare metal server in a room you control, a virtual machine in your own private cloud, a GPU enabled instance rented from a public cloud, and a disconnected site with no route to the internet at all. Same models, same tools, different address. Hybrid cloud is just the practice of running some of those surfaces at once and treating them as one estate.
Below is how the three products in the stack map onto those surfaces. RHEL AI targets a single tuned server, OpenShift AI targets a cluster, and the Red Hat AI Inference Server is a container that drops into any of them. Keep this shape in your head as the rest of the part fills it in.
Hybrid cloud without the marketing gloss
Vendor decks make hybrid cloud sound like a capability you switch on. In practice it is a set of placement decisions you make one workload at a time, and each decision trades data gravity against convenience. A model that answers from a corpus wants to sit next to that corpus, because moving several terabytes of documents to wherever the GPU happens to be is slow, costly and, for regulated data, sometimes illegal.
What Red Hat actually gives you here is portability of the runtime, not of the data. Because the same container and the same model formats run on every surface, you can move where inference happens without rewriting the application, in the same spirit as keeping an application portable across model providers. That portability is worth real money the day a cloud contract turns sour or a datacentre fills up. It is not a licence to scatter one workload across three clouds; doing that multiplies your egress bill and your attack surface for no gain.
A second cost hides in latency. Inference that runs three hundred milliseconds away across a network link feels slow to a support agent typing a question, and every retrieval round trip to a distant vector store piles on top of it. Co locating the model, the index and the application on one side of the boundary removes that tax, which is a quieter reason on prem placement often reads faster than a cloud instance that is technically larger. Placement, put plainly, is a performance decision as much as a compliance one.
RHEL AI deployment targets
RHEL AI ships as a bootable image, a whole operating system with the model server, the ilab tooling and a Granite model baked in. Red Hat provides that image for bare metal and for the major clouds: you can buy it from the AWS and Azure marketplaces and run it on their GPU enabled instances, and there are images for IBM Cloud and Google Cloud as well. One image, one server, one GPU footprint to start. That single node shape is the point of RHEL AI, and it is why it does not appear as a managed cluster service.
Where it lands still decides what hardware you get. A single accelerator card gives you tens of gigabytes of GPU memory; a full node gives you hundreds. That range is the difference between serving one small model and serving a large one with room for its KV cache, the running memory a model uses to hold the conversation so far. The chart shows the three steps you will actually climb as the project grows.
OpenShift AI, self managed or managed service
OpenShift AI comes in two shapes, and the difference is who runs the Kubernetes underneath. Self managed OpenShift AI installs on any OpenShift cluster you run, whether that cluster sits on bare metal in your datacentre, on virtual machines, or on a public cloud. You own the control plane and you own the upgrades. The managed cloud service is instead an add on to a Red Hat managed cluster: ROSA (Red Hat OpenShift Service on AWS), ARO (Azure Red Hat OpenShift), or OpenShift Dedicated. There the provider and Red Hat run the cluster jointly and bill it through the cloud, and you get scale without staffing a Kubernetes team.
Cost models differ as sharply as the residency picture. Self managed means you carry the cluster, the hardware, the operators and the upgrade nights. Managed means you rent that operational burden from Red Hat and the cloud, paying a premium per hour in exchange for not staffing a platform team at two in the morning. For a small team shipping one assistant that premium can be cheaper than the head count it replaces, and for a large group that already runs OpenShift it is close to pure markup.
That split matters for data residency. Self managed on your own bare metal keeps every byte inside your walls. A managed service keeps your data in whichever cloud region the cluster lives in, under that provider's shared responsibility model, which is fine for a lot of workloads and disqualifying for a few. Here is the full surface, product by product. This table is the one to bookmark.
Data residency and disconnected deployments
For a company that cannot send its data to a hosted API, residency is the whole game. Two levers control it. First, disconnected installs: OpenShift AI has a documented air gapped path where you mirror the container images into a registry inside the isolated network, and the object storage the platform reads must sit in that same network. Nothing phones home. RHEL AI can be stood up the same way behind a local mirror. Second, egress cost, which is the quiet tax on any design that lets data leave.
People underestimate egress because the per gigabyte number looks tiny. At roughly 0.09 US dollars per gigabyte to the public internet on a major cloud, before any volume discount, a design that streams responses and logs out of the region adds up faster than the compute in some months. Rates vary by provider and by destination, and cross region transfer is cheaper than transfer to the open internet, so treat these as order of magnitude anchors, not quotes.
Worth naming the reverse case too. A workload whose data is already public, say a documentation assistant that only reads pages you publish on the open web, carries no residency constraint at all, and for that a public cloud instance or a managed service is the sane default because it ships fastest. Residency is a property of the data, not of Red Hat AI, so the same stack lands in opposite places for two projects that look identical on a slide. Sorting your sources by that property first is what keeps the decision honest.
Placing the support assistant
Last part we decided the running project, an internal support assistant that answers staff and customer questions from product docs, a support ticket archive and a changelog, starts life on a single RHEL AI server and graduates to OpenShift AI once it needs scale. This part places those two stages physically. The corpus is the deciding fact: the ticket archive holds customer personal data that the company's own policy says cannot leave its network. That single sentence removes the public cloud instance as a resting place before we compare anything else.
The assistant will ground its answers with retrieval over that corpus, the pattern covered in what RAG is, so the model and the documents want to sit on the same side of the network boundary. When it later moves to a cluster, the placement question becomes the same one you face serving any machine learning model: where does the request enter, and where does the data it needs already live. Use this table to place a workload once you know its residency tier.
Deployment choice I would make here
For this project I would start the assistant on an on prem RHEL AI box the company owns, tune it there, and graduate the tuned model to self managed OpenShift AI in the same datacentre when it needs to serve more than one team at once. I would use public cloud GPU instances only for burst, a short lived load test or a one off tuning run on synthetic data that carries no customer records, and never as the resting place for the corpus. The option I would avoid is the one that looks fastest on a Friday: spinning up a cloud GPU instance to demo quickly, then quietly leaving production there. That is the exact move that cost me six weeks and a rebuild.
One thing to do on Monday: for your actual corpus, write down which residency tier each source falls in, public, internal, or regulated, before anyone picks an instance type. That one page settles ninety percent of the placement argument and stops the demo from becoming production in the wrong place. Next in the series we stop choosing and start building, standing up alignment with InstructLab and the LAB method so the assistant learns your own documents.
References
- Red Hat documentation, Introduction to Red Hat AI, the products and where they run
- Red Hat documentation, Installing RHEL AI, bare metal, AWS, Azure, IBM Cloud and GCP images
- Red Hat documentation, deploying OpenShift AI self managed in a disconnected environment
- Red Hat OpenShift AI product page, self managed and cloud service options
About The Author
