Almost every generative AI system I have built on Google Cloud collapses into one of four shapes. The names on the whiteboard change, the vendor decks get glossier, but the boxes and arrows keep landing in the same four places. Knowing which shape you are in tells you what to build, what it costs, and what will page you at 2am.

Four shapes cover almost everything

A reference architecture is not a diagram you frame on the wall. It is a starting point you argue with. The value is that it fixes the boring decisions so you can spend your judgement on the two or three that actually matter for your workload. On Google Cloud the four shapes below map cleanly onto managed services, which means most of the infrastructure is a configuration choice, not a build.

The order matters. Each shape adds one capability to the one before it, and each addition adds cost, latency, and failure surface. Start at the top and only move down when the workload forces you to. I have watched teams reach for an agent when a direct call would have shipped in a week, and I have watched a nightly report get wired as a live endpoint that nobody ever hit in real time. Both cost real money.

Direct calls, and why they rarely stay simple

The simplest shape is a frontend that takes a request, calls a Gemini model, and returns the text. On Google Cloud that frontend is almost always a Cloud Run service, because it scales to zero, bills per request, and speaks HTTP without you running a cluster. The model call goes to Vertex AI using a model ID like gemini-2.5-flash for cheap high-volume work or gemini-2.5-pro when the task needs deeper reasoning. That is the whole architecture. Two boxes and a line.

It rarely stays two boxes. The first production incident is usually a prompt injection or an off-topic answer, so you add safety filters and a system instruction. Then someone asks why the bot does not know last quarter numbers, and you are one step from RAG. Then legal wants an audit trail, and you add logging. None of that changes the core, but it is worth being honest that the direct call is where you start, not where you finish. Use it when the model already knows enough to answer, the request is short-lived, and a wrong answer is cheap. A marketing copy generator fits. A support agent quoting your refund policy does not.

RAG when the answer lives in your documents

RAG is the pattern you reach for when the model has to answer from your content rather than its training data. The managed path splits into two subsystems that run on different schedules. Ingestion prepares your data and builds an index. Serving answers user queries against that index. Keeping them separate is the single most useful design decision here, because your documents change on their own clock and your traffic spikes on another.

On Google Cloud the reference version wires ingestion as an event-driven flow. A document lands in a Cloud Storage bucket, which publishes a message to Pub/Sub, which triggers a Cloud Run function. The function parses and chunks the file, calls the embeddings API to turn each chunk into a vector, and writes those vectors into a Vertex AI Vector Search index. On the serving side, a Cloud Run frontend passes the query to a Cloud Run backend, which embeds the query with the same model, runs a vector-similarity search, applies Responsible AI safety filters, and sends the retrieved chunks plus the question to Gemini. If you prefer a database you already run, the same shape works with AlloyDB for PostgreSQL or Cloud SQL holding the vectors instead of Vector Search.

flowchart LR
  subgraph Ingestion
    A[Cloud Storage] --> B[Pub/Sub]
    B --> C[Cloud Run function]
    C --> D[Embeddings API]
    D --> E[Vector Search index]
  end
  subgraph Serving
    U[User query] --> F[Cloud Run frontend]
    F --> G[Cloud Run backend]
    G --> E
    G --> H[Gemini on Vertex AI]
    H --> I[Safety filters]
    I --> F
  end

One rule saves you a class of bad bugs: the query must be embedded with the exact same model and parameters as the documents were. Mismatched embedding models silently return garbage neighbours, and the symptom looks like a bad prompt, so people waste days rewriting instructions. If you want the fully managed shortcut instead of assembling these boxes yourself, Vertex AI Search covers ingestion, chunking, and retrieval as one service, which I unpacked in Part 12.

Agentic when the model has to act

An agent is the shape you need when answering the question requires doing something: calling an API, querying a database, running a search, then deciding what to do with the result. The difference from RAG is the loop. RAG retrieves once and generates once. An agent reasons, calls a tool, reads the result, and reasons again until it is done. That loop is where the capability comes from, and also where the cost and the tail latency come from.

The Google reference build uses the Agent Development Kit, the ADK, to define the agent and its tools, and deploys it as a Cloud Run service. A frontend Cloud Run service handles the chat interface, the agent runs the reasoning loop against a Gemini model, and tools reach the outside world through the Model Context Protocol, MCP. For database access the MCP Toolbox for Databases handles connection pooling and auth so the agent does not hold credentials directly. If you would rather not manage the runtime, the same ADK agent deploys to Vertex AI Agent Engine, the managed agent runtime, or to GKE for teams already standardized on Kubernetes. I compared those runtime choices in Part 21.

Worth flagging: Google has been folding Vertex AI Agent Builder into a broader offering it now calls the Gemini Enterprise Agent Platform, and some current docs use that name for what earlier parts of this series called Agent Builder. The services underneath, the ADK, Agent Engine, Vector Search, are the same. If a console label or a docs URL reads differently from what you remember, that rebrand is usually why.

Batch when nobody is waiting

Batch is not a fifth pattern so much as a delivery mode you can bolt onto the others. If no human is sitting there waiting for the answer, you should not be paying online prices for it. Vertex AI batch prediction reads inputs from BigQuery or Cloud Storage, runs them asynchronously against a Gemini model within a service window of up to twenty-four hours, and writes results to a BigQuery table or a JSONL file in Cloud Storage. In exchange for giving up the low-latency response, you pay half the token price.

This is the pattern that quietly saves the most money, and the one teams forget exists. Classifying a backlog of support tickets, tagging a product catalogue, generating embeddings for a million documents, summarizing yesterday log of transcripts: all of these are batch jobs pretending to be online workloads. Anything that runs on a schedule or fills a queue belongs here. The fifty percent discount is not a promotion, it is the standing rate for asynchronous work.

What does each pattern cost to run

The token bill dominates most of these architectures, and the batch discount is the biggest single lever you control without changing the model. Take a concrete workload: a nightly classification job over 100,000 documents, each about 2,000 input tokens and 300 output tokens, on gemini-2.5-flash. At the online rate of 0.30 dollars per million input tokens and 2.50 dollars per million output tokens, that is 60 dollars of input and 75 dollars of output per day, 135 dollars daily. Run the same job as a batch and the rate halves to 0.15 and 1.25, giving 67.50 dollars a day. Over a thirty day month that is 4,050 dollars online against 2,025 dollars batch.

The gap widens as volume grows, because token cost is linear in requests. The chart below tracks daily cost for the same per-document numbers across four traffic levels. At 200,000 documents a day the online path costs 270 dollars daily while batch holds at 135. Nothing about the model changed. You chose a delivery mode. For the full pricing picture, including provisioned throughput and context caching, see Part 6.

The table sets the four patterns side by side on the decisions that actually differ.

Submitting a batch job

Here is the batch pattern as runnable code, using the Google Gen AI SDK against Vertex AI. Input rows sit in a BigQuery table, each row carrying a request, and the results land in a second BigQuery table. This is the whole job. There is no endpoint to keep warm and no autoscaler to tune.

from google import genai
from google.genai.types import CreateBatchJobConfig

client = genai.Client(vertexai=True, project='my-project', location='us-central1')

job = client.batches.create(
    model='gemini-2.5-flash',
    src='bq://my-project.genai.batch_input',
    config=CreateBatchJobConfig(dest='bq://my-project.genai.batch_output'),
)

print(job.name)
print(job.state)

Expected output: a job resource name like projects/…/batchPredictionJobs/12345 and a state of JOB_STATE_PENDING, moving to JOB_STATE_RUNNING then JOB_STATE_SUCCEEDED. Failure mode: if the input table is missing the request column the SDK expects, the job flips to JOB_STATE_FAILED with an invalid input error, and you have paid nothing because no tokens were processed. Check the schema before you scale up.

Which failures bite in production

Each pattern fails in its own way, and knowing the shape of the failure ahead of time is most of the value of picking a pattern deliberately. Direct calls fail on quota and on safety blocks. When your traffic spikes past your Vertex AI quota, requests get throttled with a 429, so cross-region inference and a retry with backoff are not optional at scale. RAG fails on retrieval quality long before it fails on the model. If the answer is wrong, ninety percent of the time the right chunks were never retrieved, so log the retrieved chunk IDs on every request or you are debugging blind.

Agents fail on loops and on tool errors. A model that cannot make progress will keep calling tools until it burns your token budget, so cap the number of steps and set a hard timeout. When a tool returns an error, the agent has to be told how to handle it, or it will confidently hallucinate around the gap. Batch fails quietly, which is the most dangerous kind. A malformed row does not page anyone at 3am, it just sits in the failed column of your output table until someone reads the results a day later. Validate the input schema and check the success count against the input count every run. Cloud Trace and Cloud Logging give you the spans and the errors across all four patterns, which I covered in the observability part earlier in this series.

Two patterns worth standardizing on

If I could only support two of these across a team, I would keep RAG and batch, and I would build the serving path first. RAG is the shape most enterprise questions actually need, and it degrades gracefully into a direct call when retrieval is not required. Batch is the shape that pays for itself, and it reuses the same models and prompts you already wrote for serving. Agents earn their place on specific problems, but they are the pattern most often reached for too early, and they carry the most operational weight per request.

So the order I recommend is concrete. Ship the RAG serving path so users get grounded answers. Wire batch for everything offline, and watch the token bill fall by half on that work. Add an agent only when you can name the branch that a fixed sequence cannot handle. The next and final part in this series steps back from the boxes and gives the verdict on where Google Cloud stands against AWS, Azure, and the rest. Before you read it, pick one workload you run today and decide which of these four shapes it really is. Most of the time the honest answer is simpler than what you have built.

Cross-series: the AWS take on the same question is in Bedrock Reference Architectures.

References

• Generative AI with RAG, Cloud Architecture Center
• Single-agent AI system using ADK and Cloud Run
• Batch predictions, Generative AI on Vertex AI

About The Author

Dr. Pranay Jha

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

See author's posts