Your answer quality is decided before the model runs. A grounded app that retrieves the wrong three chunks will write a confident, well-formed, wrong answer, and no prompt tweak saves it. I have watched a team spend a week rewriting a system prompt when the real fault sat upstream: the user question was embedded with the same setting as the stored documents, so the nearest vectors were paraphrases of the question rather than answers to it. The retriever was working perfectly. It had just been asked the wrong way.

That one setting is task_type, and it is the cheapest fix in retrieval. This part is the data side of the Google Cloud stack: how text becomes vectors, which embedding model to pick, how to split documents into chunks, and how those vectors land in an index you can query in milliseconds. Get this right and the model has good material to reason over. Get it wrong and everything downstream inherits the mistake.

What data prep and embeddings cover

Two jobs sit between your raw documents and a grounded answer. Data prep is the work of turning messy source files into clean, right-sized pieces of text with useful labels attached. Embedding is the step that converts each piece into a vector, a fixed-length list of numbers that places the text at a point in a high-dimensional space where similar meanings sit close together. A retriever then answers a question by embedding the question the same way and pulling back the nearest stored vectors. Everything in a retrieval-augmented generation, or RAG, system rides on those two jobs being done well.

People skip straight to the model because the model is the visible part. The data pipeline is where the wins actually are. A better chunking rule or the correct task_type moves answer quality more than swapping Gemini Flash for Pro, and it costs a fraction as much. This is unglamorous plumbing, and it is where I spend most of my time on a RAG build.

How an embedding turns text into coordinates

An embedding model reads a span of text and emits a vector, say 768 or 3072 numbers long. Those numbers are coordinates. Two texts about the same thing land near each other; two unrelated texts land far apart. Distance is usually measured as cosine similarity, the angle between the two vectors, which ignores length and cares only about direction. When you search, you embed the query, then ask the index for the stored vectors with the smallest angle to it. That is the whole trick, and it is why a good embedding matters more than a clever prompt: if the right chunk is not near the query in vector space, the model never sees it.

The number of dimensions is a knob, not a fixed property. Higher dimensions capture more nuance and cost more to store and search. gemini-embedding-001 emits 3072 numbers by default but was trained with Matryoshka Representation Learning, a technique that packs the most important information into the leading dimensions so you can truncate the vector to 1536 or 768 and keep most of the quality. You choose the length with the output_dimensionality parameter. Shorter vectors mean a smaller, cheaper, faster index, and the storage difference is not small.

Choosing an embedding model in 2026

Three Google models cover almost every case. gemini-embedding-001 is the quality leader, generally available in both the Gemini API and Vertex AI, and it has held a top spot on the multilingual MTEB benchmark since its experimental launch. It handles more than 100 languages and takes up to 2048 input tokens per request. text-embedding-005 is the English and code specialist, cheaper and lighter, with a default of 768 dimensions and a longer 3072 token input. text-multilingual-embedding-002 covers 100-plus languages at 768 dimensions when you want broad language support without paying for the top model.

One rule saves pain later: never mix embedding models inside a single index. The vectors from two models live in different spaces, so a query from one cannot find documents embedded by the other. If you change models, you re-embed the whole corpus and rebuild the index. Pin the model and its dimension count in your config, and treat a change as a migration, not a tweak.

How to chunk a document

Chunking is splitting a long document into pieces small enough to embed cleanly and specific enough to retrieve. It is the setting people most often get wrong. Chunk too large and each vector blurs several ideas together, so the search matches loosely and the model wades through irrelevant text. Chunk too small and a single fact gets cut across two pieces, so neither vector carries the whole answer. The sweet spot for prose is around 300 to 500 tokens per chunk. A token is roughly three-quarters of a word, so that is a few paragraphs.

Add a small overlap between neighbouring chunks, usually 50 to 100 tokens, so a sentence sitting on a boundary is not orphaned. A common production preset is 512 tokens with 100 overlap; a lighter development preset is 300 with 50. Split on natural boundaries where you can, headings, paragraphs, list items, rather than a blind character count, because a chunk that respects structure reads as a coherent unit and embeds better. If your documents carry structure like Markdown or HTML, use it.

Attach metadata to every chunk: the source document, a section title, a timestamp, an access label. Metadata lets you filter before the vector search runs, so a query scoped to one product or one tenant never even compares against the rest. It also gives you a citation to show the user and an audit trail when an answer is questioned. Chunks without metadata work in a demo and hurt in production.

Task types are not optional

This is the setting from the opener, and it is short because the rule is simple. Vertex AI embedding models accept a task_type that tells the model what the text is for, and it changes the vector it produces. Embed your stored chunks with RETRIEVAL_DOCUMENT. Embed the user question with RETRIEVAL_QUERY. The model learned that documents and questions are asymmetric, so tagging each side correctly pulls answers toward their questions in vector space. Use one type for both and you flatten that asymmetry, and recall drops for no visible reason.

Here is the embedding step in the Vertex AI Python SDK, with both sides of the search tagged correctly and truncated to 768 dimensions.

from vertexai.language_models import TextEmbeddingModel, TextEmbeddingInput

model = TextEmbeddingModel.from_pretrained('gemini-embedding-001')

def embed(text, task):
    inp = TextEmbeddingInput(text=text, task_type=task)
    # gemini-embedding-001 takes a single input text per request
    return model.get_embeddings([inp], output_dimensionality=768)[0].values

doc_vec = embed('Vertex AI Vector Search uses the ScaNN algorithm.', 'RETRIEVAL_DOCUMENT')
qry_vec = embed('What algorithm does Vector Search use?', 'RETRIEVAL_QUERY')
print(len(doc_vec), len(qry_vec))

Expected output: 768 768, two vectors of the length you asked for. Failure mode: embed the query with RETRIEVAL_DOCUMENT instead of RETRIEVAL_QUERY and cosine similarity pulls the question toward near-duplicate questions rather than answers, so the retrieved chunks look on-topic but do not contain the fact. The call still succeeds, which is exactly why the bug hides.

Loading vectors into Vector Search

Once every chunk is a vector, it needs a home that can find nearest neighbours fast. Vertex AI Vector Search, the engine formerly called Matching Engine, is Google’s managed answer. It runs ScaNN, the approximate nearest neighbour algorithm from Google Research that trades a sliver of exactness for a large speed gain, so a query against millions of vectors returns in milliseconds instead of scanning every one. Vector Search 2.0 is generally available in 2026 and adds a storage-optimized tier for very large corpora and an auto-tuning option that manages the index configuration for you.

flowchart LR
  A[Source docs] --> B[Clean and chunk]
  B --> C[Embed with task_type]
  C --> D[Vector Search index]
  D --> E[Deploy to endpoint]
  F[User question] --> G[Embed as query]
  G --> H[Nearest neighbour search]
  E --> H
  H --> I[Top chunks to Gemini]

You load an index one of two ways. Batch updates read a file of vectors from Cloud Storage and build or refresh the index in bulk, which suits a corpus that changes on a schedule. Streaming updates insert, update, or delete individual vectors through the API and become searchable after a short delay, which suits data that changes through the day. Batch is cheaper per vector and simpler to reason about; streaming keeps the index fresh. Most builds start batch and add streaming only for the slice of data that has to be current.

# 1. Create an index from embeddings staged in Cloud Storage
gcloud ai indexes create 
  --display-name=rag-index 
  --metadata-file=index_metadata.json 
  --region=us-central1

# 2. Deploy the index to an endpoint so it can serve queries
gcloud ai index-endpoints deploy-index INDEX_ENDPOINT_ID 
  --index=INDEX_ID 
  --deployed-index-id=rag_v1 
  --region=us-central1

Expected: step 1 returns a long-running operation and an index resource name; step 2 attaches the index to an endpoint after several minutes and reports the deployment. Common failure: querying between the two steps returns a deployed-index-not-found error, because an index only serves once deployment completes. Watch the operation to done before you send traffic.

Sizing and costing an index

Vector Search pricing is infrastructure-based, not per-query. You pay for the compute nodes that hold and serve the deployed index, billed per node-hour, and the bill scales with index size and the number of replicas you run for availability and throughput. Storage and index-build steps carry their own charges on top. The two levers you control are how many dimensions each vector carries, which sets the storage, and how many replicas you deploy, which sets the serving floor.

Embed at scale, or let a managed layer do it

When the corpus is large and already sitting in your warehouse, do not pull it through a Python loop to embed it. BigQuery can call the embedding model in place with a SQL function, so millions of rows become vectors as a query, close to the data, with no export. That is the right tool when your text lives in tables and you want embeddings as another column.

If you would rather not build the chunk-embed-index pipeline by hand at all, the Vertex AI RAG Engine from Part 12 wraps chunking, embedding, and retrieval behind a managed API and can sit on top of Vector Search. The trade is the usual one: less control over each step for far less code to own. Build the pipeline yourself when you need to tune chunking or filtering precisely, and lean on the RAG Engine when a sensible default retrieval is enough. The same open-versus-managed split shows up on the Azure side in the Azure data prep and indexing part, if you run both clouds.

Where embedding pipelines go wrong

Beyond the task_type trap, a few failures repeat across projects. The first is a stale index. Your source documents change, but nobody wired a re-embed, so the retriever confidently serves last quarter’s policy. Decide the refresh cadence up front and pick batch or streaming to match it. The second is silent truncation: a chunk longer than the model’s input limit gets cut, and the tail of the text never makes it into the vector. Enforce your chunk size below the token limit rather than trusting the source to behave.

The third is skipping metadata and then wishing you had it. Without a source and a section on each chunk, you cannot filter, cite, or debug, and retrofitting metadata means re-ingesting everything. The fourth is treating recall as a feeling instead of a number. Build a small evaluation set of real questions with their correct chunks and measure how often the right chunk lands in the top three or five. That number, not a hunch, tells you whether a chunking change or a dimension change helped. When you are ready to grade the whole answer rather than just retrieval, the evaluation service in the next part covers it.

Fix retrieval before you tune the model

Here is the order I work in, and it holds up. Set the task_type correctly on day one, because it is free and it is the most common silent fault. Pick gemini-embedding-001 at 768 dimensions and pin it, so your index stays consistent and cheap. Chunk to 300 to 500 tokens on natural boundaries with a little overlap, and attach source and section metadata to every piece. Load into Vector Search with batch updates, add streaming only for data that must be current, and size replicas from your real traffic rather than a default. Then, and only then, look at the prompt and the model.

Build the retrieval test before the pipeline goes live: a dozen questions, their correct chunks, and an assertion that the right chunk lands in the top three. It runs in seconds and it catches the exact class of bug that costs a week. Next in the series we grade the whole system with the Gen AI evaluation service, which measures whether the answer, not just the retrieval, is any good. Write that retrieval test today, before you touch another prompt.

References

• Choose an embeddings task type, Vertex AI documentation
• Vector Search overview, Vertex AI documentation
• Gemini Embedding generally available, Google Developers Blog
• Gemini and open text embedding models in BigQuery, Google Cloud Blog

About The Author

Dr. Pranay Jha

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

See author's posts