Generative AI · Infrastructure · Kubernetes · Solution Architecture

One AI application, followed end to end from a solution architect’s chair: how it runs today without a GPU, what VRAM and quantization really mean, what a GPU would and would not change, why token limits disappear, where Kubernetes fits, and the sizing worksheet, bill of materials and three-year cost an architect actually has to produce. Every term is explained with a plain example or a VMware equivalent.

I built a tool called DrJhaGPT Pro. It answers questions about VMware Cloud Foundation, Kubernetes and AI infrastructure by first reading my own articles and manuals, then writing an answer. The industry calls this pattern RAG, retrieval-augmented generation. It uses a 70-billion-parameter Llama model. It runs 24×7. It costs me nothing. And I do not own a GPU.

That contradiction bothered me. I design infrastructure for a living, and here was an AI application whose infrastructure I could not draw. So I asked the questions I would normally be embarrassed to ask in a customer meeting. This post is those questions, in the order I asked them, arranged along the lifecycle of one application: discover, design, size, build, run, evolve.

MapThe lifecycle of one AI application

A solution architect does not think in products; they think in phases with a deliverable at the end of each. Here is how the rest of this post lines up with that lifecycle. If you only read one table, read this one.

Question 1 · DesignIs a RAG tool “an application”? Where does it sit in the stack?

Yes. It sits at the top. An enterprise AI stack is a set of layers, and a RAG tool lives in the application layers and consumes everything underneath. Once I laid my own tool over those layers, the whole picture became obvious, and so did what I was renting rather than running.

By code, the tool is already about 60% open source. What is SaaS is the serving (Groq), the database and identity (Supabase) and the hosting (Streamlit Cloud). Keep that split in mind; it is the answer to almost every later question.

A side note on “open source”

If your goal is “open source at every layer”, check the model licence. Llama’s licence is not OSI-approved open source. It is “open weights” with usage conditions. Mistral, Qwen, IBM Granite and OpenAI’s gpt-oss ship under Apache-2.0, which is. For an internal tool it rarely matters; for something you publish as “fully open source”, it does.

Question 2 · DesignHow is it running at all if I don’t have an A100 or H100?

Because the model is not running on my hardware. It is running on Groq’s.

When someone asks my tool a question, the Streamlit app finds the relevant paragraphs from my articles, bundles them with the question into a prompt, and sends that text over the internet to Groq’s API. Groq’s data centre runs Llama-3.3-70B on its own processors (custom chips they call LPUs, not even GPUs) and returns the answer as text. My side only runs the small pieces: the web page, the search over my index, and a tiny embedding model that is happy on a CPU.

Question 3 · DesignWhat do “VRAM” and “quantized” actually mean?

VRAM is the memory on a GPU card. To answer even one question, the entire model must sit in that memory at the same time. That is the whole constraint.

What the “70B” actually is

The 70B in Llama-3.3-70B means 70 billion parameters. A parameter is one learned number inside the neural network. When people say “weights” they mean these same numbers. The whole model is, quite literally, a very long list:

parameter 1              →   0.0134
parameter 2              →  -0.2841
parameter 3              →   1.7283
parameter 4              →   0.0047
   ...
parameter 70,000,000,000 →  -0.9182

Training is the process that finds those 70 billion values. Inference is using them. Nothing else is stored in the model file; when you download “the model” you are downloading this list.

How big is the list?

Each number has to be stored somehow, and how many bytes you spend per number is called the precision. At the precision models are trained and normally served in (16-bit, called FP16 or BF16), each number takes 2 bytes:

70,000,000,000 × 2 bytes ≈ 140 GB

Pedantic but useful: 140 GB here is decimal gigabytes (140 × 10⁹ bytes). GPU spec sheets quote binary gibibytes; 140 GB is about 130 GiB. The gap is 7%, which is roughly the difference between “fits” and “does not fit” on a tight card, so keep the units straight.

No single card holds 140 GB. An A100 or H100 has 80 GB, so you need two, wired together, which is where the “you need A100s” folklore comes from.

The weights are only the first line of the memory bill

A common mistake is to stop here and say “I need a 140 GB GPU”. The weights are the fixed cost. Serving also needs memory that grows with use:

GPU memory
├── Model weights            fixed: parameters × bytes (the 140 GB above)
├── KV cache                 grows with context length × concurrent users
├── Activations / temporaries  small, per request
├── CUDA + inference engine    a few GB of fixed overhead
└── Headroom                 you want some, or the server rejects requests

The second line, the KV cache, is the one that decides the GPU count in real deployments. Question 8 puts numbers on it.

Quantization is storing each weight in 4 bits instead of 16: a rounded, compressed copy of the same model. The 70B model shrinks to roughly 35 to 40 GB. Add working space for the conversation being processed (the “KV cache”, explained in Question 8) and you arrive at about 48 GB, which fits on one L40S or one RTX 6000-class card. You lose a little precision; in document Q&A it is usually hard to notice.

Question 4 · SizeIf I buy a GPU, what will improve? How will it differ from today?

This was the answer I least expected: the answers will not get better. Same model, same weights, same output. A GPU changes where the model runs and who controls it, not how good it is.

So the honest reasons to self-host are: privacy and compliance (a client whose manuals cannot leave the building; think DPDP Act, RBI, defence), no quotas, offline operation, and, for infrastructure people specifically, learning and credibility. An architect who has stood up Private AI end to end is a different profile from one who calls an API.

The break-even nobody shows you

Groq’s paid tier for a 70B model is on the order of ₹50 to 70 per million tokens (check the current price sheet; it changes). A single-GPU server with an L40S-class card is roughly ₹15 to 20 lakh, before power, cooling and the person who looks after it.

Most internal tools never get near the middle row. If the reason to buy is cost, do this sum first. If the reason is privacy, the sum does not matter, and that is a perfectly good reason.

RAG changes which model you need

Because a RAG tool hands the model the relevant paragraphs, the model does not have to know the answer; it has to read and write well. That lowers the bar considerably.

An 8B model needs about 6 GB of VRAM quantized. That runs on a laptop.

Question 5 · SizeIf I buy a GPU, will I still hit token limits?

No. Token limits are a business rule, not a property of the model.

Groq is sharing a fixed pool of hardware among thousands of users. To stop one person hogging it, and to nudge people toward paid plans, they cap each key: so many requests per minute, so many tokens per minute, so many per day. When the limit “expires after a while”, that is simply their meter resetting. Paid plans raise the caps; they never remove them.

On your own card there is no meter. The only limit is physics: how much work the card can do per second. The experience flips:

Two things feel like limits but are not. First, VRAM is finite: the conversation being processed also lives in GPU memory, so a 48 GB card running a quantized 70B model handles a few long conversations at once, not dozens; push past that and the server queues or rejects the request. That is a memory ceiling, not a quota, and it never “resets” because it was never counting. Second, a rented cloud GPU has no token limit either, but it has an hourly bill; you are limited by how long you leave it switched on.

Question 6 · BuildWhere does Kubernetes come into this?

Self-hosted, my tool becomes five programs that must run all the time: the Streamlit app, the vLLM model server, Postgres with pgvector, Keycloak for login, and later Grafana and Langfuse. Each is a container, a packaged program. Something has to start them, restart them when they crash, connect them to each other, and put each one on a machine with the right resources.

That something is Kubernetes. If you know vSphere, you already know the model:

You describe what you want in YAML (“one vLLM pod with one GPU, one Streamlit pod, a Postgres pod with a 50 GB disk, expose Streamlit on this hostname”) and Kubernetes makes it true and keeps it true. That is its entire job.

Question 7 · Build and RunWhat are the rules for running GPUs on Kubernetes?

Kubernetes does not understand GPUs out of the box. You add a few pieces once, then follow a handful of rules for every workload.

Setup, once per cluster

1. GPU nodes: servers with the cards, joined to the cluster, usually in their own node pool.
2. NVIDIA driver on those nodes.
3. NVIDIA Container Toolkit: lets a container see the GPU.
4. NVIDIA Device Plugin: tells Kubernetes “this node has 2 GPUs” so the scheduler can count them.
5. DCGM exporter: GPU metrics into Prometheus.

The NVIDIA GPU Operator installs items 2 to 5 in one step. That is why it appears in every Private AI reference architecture, and why my stack builder warns you if you pick NVIDIA GPUs and Kubernetes without it.

Rules for every GPU workload

1. Ask for the GPU explicitly. No request, no GPU, even on a GPU node.
2. GPUs are whole units. One or two, never half. One GPU belongs to one pod at a time. (Sharing exists: time-slicing, MIG on A100/H100, or fractional scheduling with NVIDIA Run:ai. It is an add-on, not the default. Think vGPU profiles versus passthrough.)
3. Keep ordinary pods off GPU nodes. Put a taint on them; only pods that tolerate it may land. Otherwise Postgres will happily occupy your ₹10-lakh node while vLLM waits. This is a “must run on hosts in group” DRS rule.
4. Match the card. Label nodes by GPU type and use a nodeSelector; a quantized 70B model must land on the 48 GB card, not the 24 GB one.
5. Model weights need persistent storage. vLLM downloads 35 to 40 GB on first start. Put it on a PersistentVolume or the pod re-downloads after every restart.
6. One model server, not many replicas. With one GPU you run one vLLM pod; “scale to three” needs three GPUs. Scaling the Streamlit pod is fine; it is CPU-only.
7. Expose it as an internal Service, never to the internet. vLLM has no authentication by default. It becomes http://vllm:8000/v1 inside the cluster; the app talks to it, the world talks to the app.
8. Give it time to start. Loading 40 GB takes minutes; without a long startup grace period Kubernetes decides it is dead and restarts it in a loop.

Rules 1, 3, 4, 5, 6 and 8 in one place:

apiVersion: apps/v1
kind: Deployment
metadata: { name: vllm }
spec:
  replicas: 1                          # rule 6: one GPU, one server
  selector: { matchLabels: { app: vllm } }
  template:
    metadata: { labels: { app: vllm } }
    spec:
      nodeSelector: { gpu: l40s }      # rule 4: land on the 48 GB card
      tolerations:                     # rule 3: allowed onto the tainted GPU node
        - key: nvidia.com/gpu
          operator: Exists
          effect: NoSchedule
      containers:
        - name: vllm
          image: vllm/vllm-openai:latest
          args: ["--model", "<a 4-bit checkpoint of Llama-3.3-70B>", "--max-model-len", "16384"]
          resources:
            limits: { nvidia.com/gpu: 1 }   # rule 1: ask for the GPU
          volumeMounts:
            - { name: weights, mountPath: /root/.cache/huggingface }
          startupProbe:                # rule 8: minutes, not seconds
            httpGet: { path: /health, port: 8000 }
            failureThreshold: 60
            periodSeconds: 10
      volumes:
        - name: weights
          persistentVolumeClaim: { claimName: model-weights }   # rule 5

And the one line that changes in the application: the base URL that used to point at Groq now points at http://vllm:8000/v1. vLLM and Ollama both speak the same OpenAI-style API, so the rest of the code does not know the difference.

Question 8 · SizeWhat does a solution architect need to size and price this?

Everything above is the why. This section is the worksheet: the calculations I would put in front of a customer before a single purchase order. I run one worked example all the way through so the arithmetic is visible; change the inputs and the method stays the same.

The chain to keep in your head

Every sizing conversation about an LLM platform, whether the vendor is NVIDIA, Nutanix, VMware or nobody, walks the same chain left to right. If you remember one line from this post, remember this one:

model → precision → memory → concurrency & context → GPU type and count → Kubernetes scheduling → serving engine (vLLM / NIM / NAI) → endpoint → application

Each arrow is a decision that constrains the next. Pick the model and precision and the memory is arithmetic. Add concurrency and context and the KV cache is arithmetic. Memory plus measured throughput gives the GPU count. The GPU count decides the Kubernetes node pool and scheduling rules. The serving engine turns the GPUs into a URL. The application only ever sees the URL.

Worked example: a 50-person training organisation. Peak 20 concurrent users, 400 queries per hour at peak, 2,000 queries a day. Each query is about 3,000 input tokens (system prompt + 6 retrieved chunks + question) and about 500 output tokens. Must occasionally handle a 16,000-token context. Answers within 30 seconds. Llama-3.3-70B, 4-bit. Corpus: 1,000 PDFs of about 300 pages each. Documents may not leave the premises.

Step 1 · Tokens per day (drives the API comparison)

tokens/day = queries/day × (input + output)
2,000 × (3,000 + 500) = 7 million tokens/day ≈ 154 million/month (22 working days)

At about ₹60 per million tokens on a paid API that is ≈ ₹9,000 to 12,000 per month, ≈ ₹1.1 to 1.5 lakh a year. Keep that number; it is the yardstick everything else is measured against.

Step 2 · GPU memory: weights

weights GB = parameters × bytes per parameter × 1.1 (runtime overhead)

These are the Question 3 figures with the 1.1 runtime factor applied, which is why 140 GB shows as 154 GB here. Use this table for sizing and the earlier one for explaining.

Step 3 · GPU memory: the KV cache (the part everyone forgets)

Every conversation being processed keeps a running memory of its tokens on the GPU, the KV cache. Its size per token depends on the model’s architecture:

KV bytes per token = 2 × layers × KV heads × head dimension × bytes per value

For the worked example, 20 concurrent conversations averaging 3,500 tokens each:

20 × 3,500 × 320 KB ≈ 22 GB of KV cache

Step 4 · Add them up, pick the card

VRAM needed = weights + KV cache + ~2 GB headroom
40 + 22 + 2 = 64 GB

That does not fit on one 48 GB card. Options, in order of cost:

Decision for the example: 2× L40S. Note how the KV cache, not the weights, made the decision. This is the single most common sizing mistake I see: people size for the model and forget the conversations.

Step 5 · Throughput and latency (measure, don’t guess)

required aggregate output = peak concurrent × output tokens ÷ target seconds
20 × 500 ÷ 30 s ≈ 333 tokens/second across all users

A GPU’s real throughput depends on model, precision, batch size and context length, and the numbers change every quarter. Do not size from a vendor slide. Rent the exact card for an hour, load the exact model, and run vLLM’s own benchmark (vllm bench serve) with your token profile. If the measured aggregate tokens per second at your concurrency is above the required figure with 30% margin, the card count from Step 4 stands; if not, add GPUs or reduce the model. One hour of rental replaces a year of regret.

Step 6 · Storage

The lesson: RAG storage is trivial. A million chunks is a few gigabytes. The only storage that matters for performance is the NVMe the weights load from.

Step 7 · CPU, RAM, network, power

Step 8 · Availability

One server means downtime for every kernel patch, driver update and model swap. If that is not acceptable, the BOM doubles (a second server, N+1) and Kubernetes starts to earn its keep: it drains one node while the other serves. For a training team a maintenance window is usually fine. For a customer-facing product, budget for two.

Step 9 · Software: what is licensed by what unit

For the example I take the open-source route everywhere and add NVIDIA AI Enterprise as an option line. It is what a customer buys when they want a phone number to call at 2 a.m., and it is what the VMware / Nutanix / HPE / Dell bundles include.

Step 10 · The bill of materials

Step 11 · Three-year TCO versus the API

Step 12 · The cheap variant, for contrast

Same organisation, but the team accepts an 8B model for document Q&A after a two-week trial on the API. Weights 5 GB + KV 9 GB = 14 GB, so one 24 GB card (an RTX 6000-class workstation GPU, or one L40S with room for a second model). Server ≈ ₹6 to 9 lakh all-in. Three-year TCO ≈ ₹18 lakh. The model choice moved the bill by ₹27 lakh; nothing else in the design changed. Run the quality trial before the sizing exercise, not after.

The Enterprise AI Stack Builder produces the product-level bill of materials (the “what”) for any design you click together; this section is the sizing behind it (the “how many”). Together they are the two halves of a solution document.

Question 9 · Run and EvolveSo what would I actually do, and tell a client?

Keep the API for the live tool. It works, it is free at my volume, and the model quality is the ceiling for that model regardless of where it runs.

Do not buy an A100 or H100. If you want the hands-on experience, and if you are in infrastructure you should, there are two cheap routes:

1. Ollama on your laptop with an 8B model. Free, thirty minutes, shows you the whole self-hosting flow. Weaker answers, fine for learning.

   ollama run llama3.1:8b

2. Rent a GPU by the hour and run vLLM with the quantized 70B: the same model as Groq, on hardware you control, for the price of a coffee. Switch it off when done. In India, E2E Networks, Yotta, NxtGen, or AWS Mumbai g6e instances keep the data in the country; RunPod and Lambda are the US-centric options.

   docker run --gpus all -p 8000:8000 
     -v ~/.cache/huggingface:/root/.cache/huggingface 
     -e HUGGING_FACE_HUB_TOKEN=$HF_TOKEN 
     vllm/vllm-openai:latest 
     --model <a 4-bit AWQ/GPTQ checkpoint of Llama-3.3-70B> --max-model-len 16384

Self-host only for privacy, quota or offline reasons. When you do, the change to the application is a base URL. The change to your life is everything else in this post: drivers, operators, storage, backup, monitoring, patching. That is not a complaint; that is the job, and it is the job infrastructure people already know how to do.

Day 2 and beyond: what “run” and “evolve” look like

What I would tell a client is the table from Question 1 and the TCO from Question 8, side by side. The vendor “Private AI” bundles are this stack with the sharp edges filed off and a support contract attached. Whether you buy the bundle or assemble it yourself, you should be able to point at every row and say what runs there, why, and what it costs over three years.

Glossary, with the everyday equivalent

Token

A piece of text about three-quarters of a word long. Models read, write, and get billed in tokens. A million tokens is roughly ten novels.

Context window

How many tokens the model can hold in mind at once: your question, the retrieved paragraphs and the answer combined. Like the size of the desk you are allowed to spread papers on.

Inference

Using a trained model to produce answers. Everything in this post. Running the VM, not building the golden image.

Parameter / weight

One learned number inside the model. “70B” means 70 billion of them. One dial on a 70-billion-dial mixing desk.

Precision (FP32 / FP16 / FP8 / INT8 / INT4)

How many bytes are spent storing each parameter: 4, 2, 1, 1, 0.5. Fewer bytes means a smaller model in memory and slightly less accuracy.

VRAM

Memory on the GPU card. The whole model must fit in it, like a VM’s reservation must fit in host RAM.

Quantization

Storing model weights in fewer bits (4 instead of 16) so the same model fits in less VRAM, at a small cost in precision. JPEG instead of RAW.

KV cache

The GPU memory a conversation occupies while it is being processed, about 320 KB per token for a 70B model. Grows with concurrent users and context length; the sticky notes on the desk. The usual reason a card that “fits the model” still runs out of memory.

Tensor parallel

Splitting one model across two or more GPUs in the same server so their VRAM adds up. RAID 0 for models.

RAG

Retrieval-augmented generation: find the relevant text first, then let the model write from it. An open-book exam.

Embedding

A list of numbers that represents the meaning of a piece of text, so “similar” can be computed. A library index card with a 768-digit shelf number.

Vector database

A database that stores embeddings and finds the nearest ones fast: pgvector, Milvus, Qdrant. The vCenter inventory search for paragraphs.

Pod / Node

Kubernetes’ VM and ESXi host, respectively.

Taint / toleration / nodeSelector

Kubernetes’ VM-Host affinity rules: keep the wrong workloads off the GPU hosts and put the right ones on them.

CNI / CSI

The plug-in standards for Kubernetes networking and storage. Think NSX and vSAN drivers, but vendor-neutral.

GPU Operator

NVIDIA’s installer that puts drivers, container toolkit, device plugin and metrics on every GPU node of a cluster. The vGPU Manager bundle, for Kubernetes.

Open weights

A model whose weights you can download and run, but whose licence carries conditions (Llama). Distinct from OSI open source (Mistral, Qwen, Granite, gpt-oss).

N+1

One more server than you need, so one can be down. HA admission control.

TCO

Total cost of ownership: hardware, power, licences, support and people over the life of the system, not just the purchase price. The car over three years, not the showroom sticker.

Disclosure: DrJhaGPT Pro is my own tool. Groq’s free tier, pricing and limits are as observed at the time of writing and change frequently; treat every rupee figure above as an order of magnitude, not a quote. The KV-cache formula uses Llama-3’s published architecture and differs for other model families.