The first grounded chatbot I shipped on Google Cloud returned a confident answer that did not exist in any document we owned. The model had filled a gap. The fix was not a better prompt. It was retrieval: give the model the right passages from your own data before it writes a word, and make it cite them. On Google Cloud that job splits across two products that look similar and are not, and a grounding layer that ties them to Gemini.
Vertex AI Search is a managed retrieval engine: point it at your documents, get Google-quality search and grounding with almost no plumbing. Vertex AI RAG Engine is a managed orchestration layer where you choose the chunking, the embedding model, and the vector backend. Grounding with Agent Search attaches citations from your data to a Gemini answer, and can run alongside Grounding with Google Search. Default to Vertex AI Search for document-heavy retrieval; reach for RAG Engine when you need control.
Vertex AI Search and RAG Engine, side by side
Both put your data in front of a model. They sit at different heights. Vertex AI Search is the higher-level product. You create a data store, connect a source such as a Cloud Storage bucket, a website, or a BigQuery table, and Google indexes it with the same retrieval stack that runs Search. You get semantic ranking, snippets, and citations without picking an embedding model or writing a chunker. It is the fastest way to a working answer.
RAG Engine is lower. It is a managed runtime for the retrieval pipeline, and it hands you the knobs Vertex AI Search hides. You decide how documents are chunked, which embedding model turns chunks into vectors, and which vector database stores them. A vector is a list of numbers that captures meaning, so two passages about the same thing land near each other. RAG Engine runs the import, the embedding, and the query for you, but the design choices are yours.
One naming note, because it will trip you up in the console. Through 2026 Google folded these retrieval products, plus Agent Builder, under the Gemini Enterprise Agent Platform. The docs moved, the URLs changed, and some pages still say Vertex AI Search while others say Agent Search. The capabilities are the same. I use the product names Google uses in the SDK, which still read vertexai and Vertex AI Search.
flowchart LR D[Documents] --> IM[Import] IM --> CE[Chunk and embed] CE --> VB[(Vector backend)] VB --> RT[Retrieval] RT --> G[Gemini grounded answer]
How does RAG Engine actually retrieve?
You create a corpus, which is RAG Engine name for one retrieval index. You import files into it. RAG Engine splits each file into chunks, runs every chunk through an embedding model to get a vector, and writes those vectors to the backend you chose. At query time your text becomes a vector too, and the backend returns the nearest chunks by distance. Those chunks go into the Gemini prompt as context.
The search algorithm is a real decision, not a detail. KNN, exact nearest neighbor, checks the query vector against every stored vector. Recall is perfect and latency climbs with the corpus. Google recommends KNN for corpora under about 10,000 files. ANN, approximate nearest neighbor, trades a sliver of recall for far lower latency on large corpora and is in preview on RAG Engine. If your index is small, keep KNN and move on. If it is large and slow, ANN is the lever.
Pick a vector backend for the corpus
RAG Engine gives you several places to store the vectors. The default is RagManagedDb, a fully managed Spanner instance that RAG Engine runs for you. It comes in a Basic tier fixed at 0.1 nodes, which is cheap and fine for a demo or a small internal tool, and a Scaled tier that autoscales from 1 to 10 nodes for production traffic. If you want the newest retrieval features and direct visibility into the index, RagManagedVertexVectorSearch puts your data in Vector Search 2.0 while RAG Engine still manages the collections. There is also a Vertex AI Feature Store option, and you can point RAG Engine at Vertex AI Search itself as the retrieval backend when you already run search apps there.
My rule is short. Start on RagManagedDb Basic while you prove the retrieval quality is there. Move to the Scaled tier the day real users arrive. Switch to Vector Search 2.0 only when you need its features or the scale, because it is more to reason about, not less.
| Backend | Who runs it | Scale note | Reach for it when |
|---|---|---|---|
| RagManagedDb Basic | RAG Engine (Spanner) | Fixed 0.1 nodes | Prototypes, small internal tools |
| RagManagedDb Scaled | RAG Engine (Spanner) | Autoscale 1 to 10 nodes | Production traffic, default choice |
| RagManagedVertexVectorSearch | RAG Engine on Vector Search 2.0 | Large corpora, ANN option | Newest features, index visibility |
| Vertex AI Feature Store | You, via RAG corpus link | Reuses existing store | You already run Feature Store |
| Vertex AI Search | Google, fully managed | Google-scale retrieval | You want turnkey, not knobs |
Table 1. Vector backends for a RAG Engine corpus. Node figures are from the RagManagedDb tiers.
Grounding with Agent Search and Google Search
Retrieval fetches passages. Grounding is what makes Gemini answer from them and cite them, instead of answering from memory. On Google Cloud you attach grounding to a Gemini call in two flavors. Grounding with Agent Search points the model at your Vertex AI Search data sources, up to ten of them per request. Grounding with Google Search points it at the public web for freshness. You can combine them, so an answer can lean on your policy documents and the open web in the same turn.
The distinction people miss is that grounding does not replace RAG Engine, it consumes what retrieval produced. If your data lives in a Vertex AI Search data store, Grounding with Agent Search is the whole story and you may never touch RAG Engine. If your data lives in a RAG Engine corpus, you wire that corpus in as a retrieval tool on the model call instead. Same outcome, a cited answer, different plumbing underneath.
Worked example
Say you have 40,000 support PDFs to make answerable. At an average of six chunks per document you get about 240,000 vectors. That is well past the KNN guidance limit of roughly 10,000 files, so exact scan latency starts to bite. In my rough tests the same query ran near 420 ms on KNN and near 90 ms on ANN at that size. So the design is: RagManagedVertexVectorSearch backend, ANN search, Scaled tier, and grounding through the corpus retrieval tool. Under a few thousand files I would have left every default alone.
Latency figures are illustrative from a small test, not a published benchmark.
When is plain Vertex AI Search enough?
More often than engineers expect. If your data is documents and web pages, and you want good retrieval without owning a pipeline, a Vertex AI Search data store plus Grounding with Agent Search will carry a real product. You skip chunk tuning, embedding choice, and vector database sizing. That is a lot of work you do not do.
You outgrow it when you need control the managed path does not give. Custom chunking for oddly structured data. A specific embedding model your evaluations proved better. A vector backend you already operate. Metadata filters and hybrid retrieval logic that the turnkey ranker will not express. When you hit two or more of those, RAG Engine earns its place. Until then, the simpler product is the right call, and I have watched teams spend a quarter rebuilding what a data store gave them on day one.
| You want | Use | Why |
|---|---|---|
| Answers from private docs, fast | Vertex AI Search plus Agent Search grounding | No pipeline to build or size |
| Control over chunking and embeddings | RAG Engine corpus | The knobs are yours |
| Fresh public-web facts | Grounding with Google Search | Reaches beyond your data |
| Private docs and web in one answer | Both grounding modes together | They combine per request |
Table 2. Match the job to the product before you write code.
Build a corpus and query it
Here is the shortest useful RAG Engine flow: create a corpus, import a folder from Cloud Storage, then retrieve. The SDK surface is still the vertexai package. Google is moving generative calls to the google-genai SDK, so treat the exact import path as something to check against the current docs before you ship.
import vertexai
from vertexai import rag
vertexai.init(project=PROJECT_ID, location=LOCATION)
# 1. Create a corpus, choosing the embedding model once
embed = rag.RagEmbeddingModelConfig(
vertex_prediction_endpoint=rag.VertexPredictionEndpoint(
publisher_model=PUBLISHER_EMBED_MODEL))
corpus = rag.create_corpus(
display_name=support-kb,
backend_config=rag.RagVectorDbConfig(rag_embedding_model_config=embed))
print(corpus.name) # projects/.../ragCorpora/1234567890
# 2. Import a folder of files from Cloud Storage
rag.import_files(
corpus.name,
paths=[gs://my-bucket/support-pdfs/],
transformation_config=rag.TransformationConfig(
chunking_config=rag.ChunkingConfig(chunk_size=512, chunk_overlap=100)))
# 3. Retrieve the nearest passages for a question
res = rag.retrieval_query(
rag_resources=[rag.RagResource(rag_corpus=corpus.name)],
text=How do I reset a device to factory defaults,
rag_retrieval_config=rag.RagRetrievalConfig(top_k=5))
for ctx in res.contexts.contexts:
print(round(ctx.score, 3), ctx.source_uri)Expected output: the corpus resource name, then five lines of distance score and source URI, closest passage first.
Failure mode: call retrieval_query right after import_files and you may get empty contexts, because embedding a large folder is asynchronous. Poll the import operation to completion before you query, or you will think retrieval is broken when it is only still indexing. Method names above are marked because the SDK is in flux.
Retrieval quality is decided by chunking and the source data, not by the vector database. Teams obsess over ANN versus KNN and pick the wrong chunk size, then wonder why answers are thin. Fix the input first. A clean corpus on KNN beats a messy corpus on any backend.
Measure retrieval before you trust it
A grounded answer is only as good as the passages that fed it, and you cannot eyeball that at scale. Build a small set of real questions with the passage you know should answer each one. Then check two things. Recall at k asks whether the right passage showed up in the top k results the retriever returned; if it never surfaces, no amount of prompt work saves the answer. Groundedness asks whether the final text actually rests on the retrieved passages or drifted back to the model memory. Vertex AI ships a Gen AI evaluation service that scores both, and I run it as a gate, not a one-time check.
The reason this matters here is that every knob in this post moves one of those two numbers. Chunk size and the embedding model move recall. Grounding mode and top_k move groundedness. If you tune blind you will trade one for the other and not notice. Fifty labeled questions caught more bad retrieval for me than any dashboard. Part 23 goes deep on the evaluation service; treat this as the reason to wire it in from the start.
Where I start: Vertex AI Search first, RAG Engine when you need the knobs
Open a Vertex AI Search data store, point it at your documents, and put Grounding with Agent Search on the Gemini call. Ship that. It answers from your data with citations and you built no pipeline. Move to RAG Engine only when a real requirement forces control the managed path will not give: custom chunking, a specific embedding model, or a vector backend you already run. When you do, start on RagManagedDb, keep KNN until a latency trace tells you otherwise, and remember the embedding model is a one-time decision. This is how you get from a question to a grounded answer on Google Cloud without paying for machinery you do not need. Part 13 picks the story up where retrieval becomes a tool an agent decides to call: build on the Vertex AI foundation from Part 2 and read how the same job looks on Amazon Bedrock Knowledge Bases. Try the corpus flow in a scratch project this week and measure your own retrieval before you tune a thing.
References
- Introducing Vertex AI RAG Engine, Google Cloud blog
- Use Vector Search 2.0 with RAG Engine
- Grounding with Agent Search
About The Author
DrJha