Say you want to build a chatbot that answers from your own documents, a RAG chatbot. The confusing part is not the idea, it is the shopping. Every cloud has a dozen AI products, and it is hard to see which one does what. So here is the map. A RAG chatbot is the same handful of steps no matter who you buy from, and this post shows you exactly which product plays each role in open source, Microsoft Azure, AWS, IBM watsonx, and a VMware on-prem setup.
A RAG chatbot is always the same pipeline
Before comparing vendors, it helps to see that they are all building the same thing. Your documents get prepared, turned into vectors, stored, searched, and finally an AI model writes the answer using what was found. That is it. The parts have fancy names, but the flow does not change.
The stages, in plain terms
- Ingest & parse: pull text out of PDFs, docs, and web pages.
- Chunk: cut that text into small, searchable passages.
- Embed: turn each passage into a vector (a list of numbers that captures meaning).
- Vector database: store those vectors so you can search them fast.
- Retrieve & re-rank: find the passages closest to the question, then sort the best to the top.
- LLM: the language model writes the answer using the retrieved passages.
- Around all of it: orchestration (the glue), hosting (where it runs), guardrails (safety), and monitoring (is it any good).
The same chatbot, five ways
Here is the whole thing side by side. Read a row to see your options for one stage, or read a column to see what an all-in stack looks like. You do not have to stay in one column, and most real builds do not.
| Stage | Open source (free) | Microsoft Azure | AWS | IBM watsonx | VMware (on-prem) |
|---|---|---|---|---|---|
| Ingest & parse | Unstructured, Apache Tika, LangChain loaders | AI Document Intelligence | Textract, Bedrock Knowledge Bases | Watson Discovery, Data Prep Kit | Same OSS tools, run on Tanzu |
| Chunk & prep | LangChain / LlamaIndex splitters | AI Search integrated vectorization | Bedrock Knowledge Bases chunking | watsonx pipelines, Data Prep Kit | OSS splitters on your VMs |
| Embeddings | BGE, E5, MiniLM (sentence-transformers) | Azure OpenAI text-embedding-3 | Titan Embeddings, Cohere (Bedrock) | watsonx Granite (slate) embeddings | Open models on GPU, NVIDIA NIM |
| Vector database | pgvector, Qdrant, Weaviate, Milvus, Chroma | AI Search, Cosmos DB, PostgreSQL | OpenSearch, Aurora pgvector, Kendra | watsonx.data (Milvus), Db2, Elasticsearch | pgvector / Milvus on Tanzu |
| Retrieve & re-rank | BM25 + bge-reranker, ColBERT | AI Search hybrid + semantic ranker | Kendra, OpenSearch + Cohere Rerank | Watson Discovery, watsonx re-rank | OSS hybrid + reranker, on-prem |
| LLM (generation) | Llama, Mistral, Qwen via Ollama / vLLM; Groq free tier | Azure OpenAI (GPT-4o), Foundry catalog | Bedrock (Claude, Nova, Llama, Mistral) | watsonx.ai (Granite, Llama, Mistral) | NVIDIA NIM + open models on vGPU |
| Orchestration | LangChain, LlamaIndex, Haystack | AI Foundry / Prompt Flow, Semantic Kernel | Bedrock Agents & Knowledge Bases | watsonx Orchestrate, watsonx Assistant | OSS frameworks on Tanzu |
| App hosting | Docker + FastAPI / Streamlit | Container Apps, App Service, AKS | Lambda, Fargate, EKS, App Runner | IBM Cloud, Red Hat OpenShift | vSphere VMs, Tanzu Kubernetes |
| Guardrails & safety | NeMo Guardrails, Llama Guard, Guardrails AI | AI Content Safety | Bedrock Guardrails | watsonx.governance (HAP / PII) | OSS guardrails, on-prem |
| Monitoring & eval | Ragas, Langfuse, Phoenix, TruLens | Foundry evaluations, Azure Monitor | CloudWatch, Bedrock evaluations | watsonx.governance | Aria Operations + OSS |
These are examples of the kind of service that fits each box, not an exhaustive list, and cloud product names change quickly. The point is the pattern: every ecosystem has an answer for every stage.
The real trade-off
Once you see the grid, the choice stops being about features, because they all cover the same stages. It becomes about two things: how much you want handed to you, and where your data is allowed to live.
Managed cloud services (Azure, AWS, IBM) get you live fastest and handle scaling and patching, but you rent the stack and your data flows through their systems. Open source costs nothing in licences and never locks you in, but you operate every piece yourself. On-prem with VMware sits at the far end: the most control and the strongest data-residency story, for teams that cannot let documents leave the building.
So which stack should you choose?
Open source (free)
Tight budget, full control, no lock-in, and you have the skills to run it. Ideal for learning, startups, and side projects. The catch: you maintain everything.
Microsoft Azure
You already live in Microsoft 365 and Entra, want GPT models with enterprise controls, and like that Azure AI Search covers most of retrieval in one product.
AWS
You are already on AWS. Bedrock gives the widest model choice (including Claude) plus serverless scale, and Knowledge Bases wires up RAG for you.
IBM watsonx
A regulated enterprise that wants governance first, IBM Granite models, and an easy hybrid or on-prem path through Red Hat OpenShift.
VMware on-prem
Your data cannot leave the building (sovereignty, air-gapped, compliance). You already run vSphere and can add GPUs. VMware Private AI with NVIDIA packages it.
Most real builds are a hybrid
You do not have to pick one column and commit. The smart move is usually to take the cheapest or simplest option at each stage. My own assistant, DrJhaGPT, is exactly that: open-source embeddings that run locally and cost no tokens, a managed vector database on a free tier, and a hosted LLM for the writing. Open source where it saves money, managed where it saves time. That mix-and-match is the norm, not a compromise.
Bottom line
A RAG chatbot is not five different things, it is one pipeline with five sets of price tags. Learn the seven stages once and every vendor’s catalog suddenly makes sense, because each product is just their version of a box you already understand. Start with the pipeline, then choose each box by budget, speed, and where your data is allowed to live.
Building one yourself?
I write about the practical side of AI and cloud infrastructure, the choices that decide whether a system is cheap, fast, and easy to run. If that is your world, there is more here for you.
Explore the guides & tools →
Ask DrJhaGPT →
Which stack are you building on? Tell me in the comments, and subscribe for more field notes on RAG and AI infrastructure.
About The Author
DrJha