Generative AI · Infrastructure · Kubernetes · Solution Architecture

One AI application, followed end to end from a solution architect’s chair: how it runs today without a GPU, what VRAM and quantization really mean, what a GPU would and would not change, why token limits disappear, where Kubernetes fits, and the sizing worksheet, bill of materials and three-year cost an architect actually has to produce.

I built a tool called DrJhaGPT Pro. It answers questions about VMware Cloud Foundation, Kubernetes and AI infrastructure by first reading my own articles and manuals, then writing an answer. The industry calls this pattern RAG, retrieval-augmented generation. It uses a 70-billion-parameter Llama model, runs 24×7, costs me nothing, and I do not own a GPU.

That contradiction bothered me. I design infrastructure for a living, and here was an AI application whose infrastructure I could not draw. So I asked the questions I would normally be embarrassed to ask in a customer meeting. This post is those questions, in the order I asked them, arranged along the lifecycle of one application.

When someone says “you need an H100 for AI”, they are usually mixing these two up.

Map Start hereThe lifecycle of one AI application

A solution architect does not think in products; they think in phases with a deliverable at the end of each. Here is how the rest of this post lines up with that lifecycle. If you only read one diagram, read this one.

Q1 DesignIs a RAG tool “an application”? Where does it sit in the stack?

Yes. It sits at the top. An enterprise AI stack is a set of layers, and a RAG tool lives in the application layers and consumes everything underneath. Once I laid my own tool over those layers, the whole picture became obvious, and so did what I was renting rather than running.

By code, the tool is already about 60% open source. What is SaaS is the serving (Groq), the database and identity (Supabase) and the hosting (Streamlit Cloud). Keep that split in mind; it is the answer to almost every later question.

A side note on “open source”

If your goal is “open source at every layer”, check the model licence. For an internal tool it rarely matters; for something you publish as “fully open source”, it does.

Q2 DesignHow is it running at all if I don’t have an A100 or H100?

Because the model is not running on my hardware. It is running on Groq’s.

When someone asks my tool a question, the app sends the prompt as text over the internet to Groq’s API. Everything heavy happens in their building. My side only runs the small pieces: the web page, the search over my index, and a tiny embedding model that is happy on a CPU.

Q3 DesignWhat do “VRAM” and “quantized” actually mean?

VRAM is the memory on a GPU card. To answer even one question, the entire model must sit in that memory at the same time. That is the whole constraint.

What the “70B” actually is

The 70B in Llama-3.3-70B means 70 billion parameters. A parameter is one learned number inside the neural network; “weights” means the same numbers. The whole model is, quite literally, a very long list:

parameter 1              →   0.0134
parameter 2              →  -0.2841
parameter 3              →   1.7283
parameter 4              →   0.0047
   ...
parameter 70,000,000,000 →  -0.9182

Training is the process that finds those 70 billion values. Inference is using them. Nothing else is stored in the model file; when you download “the model” you are downloading this list.

How big is the list?

How many bytes you spend per number is called the precision. Models are trained and normally served at 16-bit (FP16 or BF16), 2 bytes each:

Pedantic but useful: 140 GB here is decimal gigabytes (140 × 10⁹ bytes). GPU spec sheets quote binary gibibytes; 140 GB is about 130 GiB. The gap is 7%, roughly the difference between “fits” and “does not fit” on a tight card, so keep the units straight.

No single card holds 140 GB. An A100 or H100 has 80 GB, so you need two, wired together, which is where the “you need A100s” folklore comes from.

The weights are only the first line of the memory bill

A common mistake is to stop here and say “I need a 140 GB GPU”. The weights are the fixed cost. Serving also needs memory that grows with use:

The red block, the KV cache, is the one that decides the GPU count in real deployments. Question 8 puts numbers on it.

Quantization is storing each weight in 4 bits instead of 16: a rounded, compressed copy of the same model. You lose a little precision; in document Q&A it is usually hard to notice.

Q4 SizeIf I buy a GPU, what will improve? How will it differ from today?

So the honest reasons to self-host are four, and “better answers” is not one of them:

The break-even nobody shows you

Groq’s paid tier for a 70B model is on the order of $0.60 to 0.80 per million tokens (check the current price sheet; it changes). A single-GPU server with an L40S-class card is roughly $18,000 to 24,000, before power, cooling and the person who looks after it.

How long would a $21,000 server take to pay for itself against the API bill it replaces?

Most internal tools never get near the middle row. If the reason to buy is cost, do this sum first. If the reason is privacy, the sum does not matter, and that is a perfectly good reason.

RAG changes which model you need

Because a RAG tool hands the model the relevant paragraphs, the model does not have to know the answer; it has to read and write well. That lowers the bar considerably.

An 8B model needs about 6 GB of VRAM quantized. That runs on a laptop.

Q5 SizeIf I buy a GPU, will I still hit token limits?

Groq is sharing a fixed pool of hardware among thousands of users. To stop one person hogging it, and to nudge people toward paid plans, they cap each key: so many requests per minute, so many tokens per minute, so many per day. When the limit “expires after a while”, that is simply their meter resetting. Paid plans raise the caps; they never remove them.

On your own card there is no meter. The only limit is physics: how much work the card can do per second. The experience flips:

Two things feel like limits but are not quotas:

Q6 BuildWhere does Kubernetes come into this?

Self-hosted, my tool becomes five programs that must run all the time: the Streamlit app, the vLLM model server, Postgres with pgvector, Keycloak for login, and later Grafana and Langfuse. Each is a container, a packaged program. Something has to start them, restart them when they crash, connect them to each other, and put each one on a machine with the right resources.

That something is Kubernetes. If you know vSphere, you already know the model:

You describe what you want in YAML (“one vLLM pod with one GPU, one Streamlit pod, a Postgres pod with a 50 GB disk, expose Streamlit on this hostname”) and Kubernetes makes it true and keeps it true. That is its entire job.

But you do not need Kubernetes yet

Q7 Build and runWhat are the rules for running GPUs on Kubernetes?

Kubernetes does not understand GPUs out of the box. You add a few pieces once, then follow a handful of rules for every workload.

Setup, once per cluster

That is why the GPU Operator appears in every Private AI reference architecture, and why my stack builder warns you if you pick NVIDIA GPUs and Kubernetes without it.

Eight rules for every GPU workload

Rules 1, 3, 4, 5, 6 and 8 in one place:

apiVersion: apps/v1
kind: Deployment
metadata: { name: vllm }
spec:
  replicas: 1                          # rule 6: one GPU, one server
  selector: { matchLabels: { app: vllm } }
  template:
    metadata: { labels: { app: vllm } }
    spec:
      nodeSelector: { gpu: l40s }      # rule 4: land on the 48 GB card
      tolerations:                     # rule 3: allowed onto the tainted GPU node
        - key: nvidia.com/gpu
          operator: Exists
          effect: NoSchedule
      containers:
        - name: vllm
          image: vllm/vllm-openai:latest
          args: ["--model", "<a 4-bit checkpoint of Llama-3.3-70B>", "--max-model-len", "16384"]
          resources:
            limits: { nvidia.com/gpu: 1 }   # rule 1: ask for the GPU
          volumeMounts:
            - { name: weights, mountPath: /root/.cache/huggingface }
          startupProbe:                # rule 8: minutes, not seconds
            httpGet: { path: /health, port: 8000 }
            failureThreshold: 60
            periodSeconds: 10
      volumes:
        - name: weights
          persistentVolumeClaim: { claimName: model-weights }   # rule 5

And the one line that changes in the application: the base URL that used to point at Groq now points at http://vllm:8000/v1. vLLM and Ollama both speak the same OpenAI-style API, so the rest of the code does not know the difference.

Q8 SizeWhat does a solution architect need to size and price this?

Everything above is the why. This section is the worksheet: the calculations I would put in front of a customer before a single purchase order. One worked example runs all the way through so the arithmetic is visible; change the inputs and the method stays the same.

The chain to keep in your head

Every sizing conversation about an LLM platform, whether the vendor is NVIDIA, Nutanix, VMware or nobody, walks the same chain left to right. If you remember one line from this post, remember this one:

Each arrow is a decision that constrains the next. Model and precision make the memory arithmetic. Concurrency and context make the KV cache arithmetic. Memory plus measured throughput gives the GPU count. The GPU count decides the node pool and scheduling rules. The serving engine (vLLM, NIM, NAI) turns the GPUs into a URL. The application only ever sees the URL.

How to answer it in a room

Someone asks: “how would you size GPUs for a 70B model?”

Step 1Tokens per day (drives the API comparison)

At about $0.70 per million tokens on a paid API that is ≈ $110 to 140 per month, ≈ $1,300 to 1,700 a year. Keep that number; it is the yardstick everything else is measured against.

Step 2GPU memory: weights

These are the Question 3 figures with the 1.1 runtime factor applied, which is why 140 GB shows as 154 GB here. Use this table for sizing and the earlier bars for explaining.

Step 3GPU memory: the KV cache (the part everyone forgets)

Every conversation being processed keeps a running memory of its tokens on the GPU, the KV cache. Its size per token depends on the model’s architecture:

For the worked example, 20 concurrent conversations averaging 3,500 tokens each:

Step 4Add them up, pick the card

That does not fit on one 48 GB card. The options, in order of cost (bar = the card’s memory; black = weights, red = room left for conversations):

Step 5Throughput and latency (measure, don’t guess)

A GPU’s real throughput depends on model, precision, batch size and context length, and the numbers change every quarter. Do not size from a vendor slide. Rent the exact card for an hour, load the exact model, and run vLLM’s own benchmark (vllm bench serve) with your token profile. If the measured aggregate tokens per second at your concurrency beats the required figure with 30% margin, the card count from Step 4 stands; if not, add GPUs or reduce the model. One hour of rental replaces a year of regret.

Step 6Storage

The lesson: RAG storage is trivial. A million chunks is a few gigabytes. The only storage that matters for performance is the NVMe the weights load from.

Step 7CPU, RAM, network, power

Step 8Availability

One server means downtime for every kernel patch, driver update and model swap. If that is not acceptable, the BOM doubles (a second server, N+1) and Kubernetes starts to earn its keep: it drains one node while the other serves. For a training team a maintenance window is usually fine. For a customer-facing product, budget for two.

Step 9Software: what is licensed by what unit

For the example I take the open-source route everywhere and add NVIDIA AI Enterprise as an option line. It is what a customer buys when they want a phone number to call at 2 a.m., and it is what the VMware / Nutanix / HPE / Dell bundles include.

Step 10The bill of materials

Step 11Three-year TCO versus the API

Step 12The cheap variant, for contrast

Same organisation, but the team accepts an 8B model for document Q&A after a two-week trial on the API. Weights 5 GB + KV 9 GB = 14 GB, so one 24 GB card (an RTX 6000-class workstation GPU, or one L40S with room for a second model). Server ≈ $7,000 to 11,000 all-in.

The model choice moved the bill by $32,000; nothing else in the design changed. Run the quality trial before the sizing exercise, not after.

The Enterprise AI Stack Builder produces the product-level bill of materials (the “what”) for any design you click together; this section is the sizing behind it (the “how many”). Together they are the two halves of a solution document.

Q9 Run and evolveSo what would I actually do, and tell a client?

ollama run llama3.1:8b

docker run --gpus all -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -e HUGGING_FACE_HUB_TOKEN=$HF_TOKEN \
  vllm/vllm-openai:latest \
  --model <a 4-bit AWQ/GPTQ checkpoint of Llama-3.3-70B> \
  --max-model-len 16384

When you do self-host: drivers, operators, storage, backup, monitoring, patching. That is not a complaint; that is the job, and it is the job infrastructure people already know how to do.

Day 2 and beyond: what “run” and “evolve” look like

Glossary, with the everyday equivalent

Disclosure: DrJhaGPT Pro is my own tool. Groq’s free tier, pricing and limits are as observed at the time of writing and change frequently; treat every dollar figure above as an order of magnitude, not a quote. Hardware and people costs are Indian list prices converted at roughly ₹85 to the US dollar. The KV-cache formula uses Llama-3’s published architecture and differs for other model families.