Your service worked in the demo and threw this in production: 429 RESOURCE_EXHAUSTED. Nothing changed in the code. What changed is that real traffic arrived in one region at the same minute as everyone else on the same model, and the shared capacity ran dry. This part is about the three levers that decide whether that happens to you: which location you send a request to, how quota is granted on that location, and when a guaranteed floor is worth paying for.

Vertex AI is Google Cloud’s managed platform for building with generative models, and Gemini is the family of models you call on it. Every call you make lands in a location. Getting that location wrong shows up as slow responses for users on the far side of the world, as compliance findings when data leaves a jurisdiction it was never allowed to leave, and as those 429 errors under load. Getting it right is mostly a set of decisions you make once, up front, and rarely revisit.

What a location means on Vertex AI

A location is the geography where your request is processed and where the endpoint you call lives. Vertex AI gives you three kinds. A region is a single place, such as us-central1 in Iowa or europe-west4 in the Netherlands, and you reach it through a hostname that carries the region in it, like us-central1-aiplatform.googleapis.com. A multi-region is a broader jurisdiction, the US or the EU, used mostly for where data sits at rest. The global endpoint is the odd one out: a single hostname, aiplatform.googleapis.com, that does not pin your request to any one place and instead sends it wherever the model has room.

The word endpoint just means the address you send the request to. Each region has its own address, and the global endpoint has one address that fans out behind the scenes. When you set up a client, the location string you pass is what selects the endpoint. Pass us-central1 and you talk to Iowa. Pass global and you hand routing to Google. That one string carries more consequence than its length suggests, because it decides latency, residency, and which pool of capacity you draw from all at once.

Not every model lives in every region. A new Gemini version often lands in a handful of regions first, and some models are offered only on the global endpoint, or only in a specific region, and never move. So the real first question is not which region you prefer. It is which locations actually serve the exact model you want, and that list is the constraint you build around.

Regional endpoints and where your data is processed

Data residency is the rule that your data has to stay inside a named geography. Many teams carry it as a hard requirement, written into a contract or a regulation, not a preference. Regional endpoints are how you meet it on Vertex AI. When you call a regional endpoint, Google processes that request inside the geography tied to the region, and for Gemini the content of the request is not stored after the response comes back. Pick europe-west4 and the model runs in the EU. Pick us-central1 and it runs in the US. The location string is your residency control.

Two guarantees hide behind that. One is data at rest, meaning anything Vertex AI stores for you, which the US and EU multi-regions keep inside those bounds. The other is machine learning processing, meaning where the model actually computes the answer, which a regional endpoint keeps inside that region. Both matter, and they are set separately, so it is worth checking each against your obligation rather than assuming one covers the other.

Regional also means full feature coverage. Tuning a model, running batch prediction over a large file, and context caching, where you pay to keep a big shared prompt warm between calls, all run on regional endpoints. That completeness is the quiet reason regional is the sane default. You lose nothing on features, you gain a residency guarantee, and the only thing you give up is the extra availability that comes from spreading across the planet.

When should you reach for the global endpoint?

Reach for it when availability is the thing you care about most and residency is not a constraint. The global endpoint answers a request from whichever region has free capacity at that instant. When one region is saturated and would return a 429, another region takes the call. In practice that means far fewer resource exhausted errors and a higher effective ceiling on throughput, because you are drawing from a much larger shared pool than any single region holds.

The price of that is control. You cannot know or choose which region processes a given call, so the global endpoint gives no residency guarantee. It also carries a separate quota from the regional endpoints, so capacity you see there is not the same pool you see in us-central1. And as of this writing it does not support tuning, batch prediction, or context caching, so anything that leans on those has to run on a region regardless.

So the rule is simple. If your data can go anywhere and you want the fewest 429s and the steadiest uptime, the global endpoint is the right call, and for a stateless chat or completion workload it is often the better default than any single region. If you have a residency line to hold, or you need tuning, batch, or caching, stay regional. Plenty of production systems run regional as the primary path and keep the global endpoint as a fallback for the minute a region runs hot.

How Vertex AI quotas actually work

A quota is the ceiling on how much you can call in a window of time. For years that meant a fixed number of requests per minute per region, a number you watched and filed a request to raise. Newer Gemini models changed the model underneath. They run on Dynamic Shared Quota, or DSQ, where Google pools all the pay as you go capacity for a given model in a given region and hands it out in real time across every customer asking for it. There is no preset number for you to raise, because there is no fixed per customer limit. You draw from the pool, and when the pool is tight, you feel it as a 429.

That is a real shift in how you operate. Under the old fixed quota you could plan against a known number and ask for more ahead of a launch. Under DSQ there is nothing to pre approve, so your defense is different: retry with backoff when you get a 429, spread load across more than one region, and lean on the global endpoint to pull from a wider pool. The pool is usually generous, but it is shared, and a spike that hits everyone at once, on a popular model in a popular region, is exactly when it bites.

Not everything is on DSQ. Some models and some services still expose the older rate quotas you can view and request increases on in the Google Cloud console, and certain enterprise controls let you cap or reserve usage. The practical takeaway holds either way. For pay as you go traffic on current Gemini models, treat capacity as elastic but not guaranteed, and design the client to survive a 429 rather than assuming a number protects you.

Provisioned Throughput when a shared pool is not enough

DSQ is elastic and cheap, but best effort. When you have a workload that cannot tolerate a 429 at peak, you buy a floor. Provisioned Throughput reserves capacity for a specific model in a specific region, measured in a unit Google calls a generative AI scale unit, or GSU. Your guaranteed throughput is the number of GSUs you buy times the throughput each GSU delivers for that model. Inside that reservation you are not competing with the shared pool, so the peak that would 429 on DSQ is served.

The enforcement is smarter than a hard gate. Vertex AI applies a flexible window when it meters Provisioned Throughput for Gemini, so a short burst above your steady rate can still pass instead of being chopped the instant you cross the line. That matters for real traffic, which is spiky by nature. It does not mean the floor is infinite. Size the reservation for your sustained peak, and let DSQ or the global endpoint absorb the rare spike beyond it.

Provisioned Throughput is not a discount. It usually costs more per unit of work than pay as you go, and you pay for the reservation whether you fill it or not. What you buy is predictability, not a lower bill. So the honest question is whether a 429 at peak actually costs your business something. For a background job, no, ride DSQ. For a checkout assistant or anything a paying user is staring at, a floor is cheap insurance.

Picking a region for a Frankfurt and Tokyo audience

Put the levers together on a real decision. Say you run a chat assistant with users split between Frankfurt and Tokyo, and legal tells you EU user data must stay in the EU. Residency settles the European half at once: those calls go to a regional endpoint in the EU, europe-west4 for example, and nowhere else. The global endpoint is off the table for that traffic because it cannot promise the EU. That is the constraint doing your first cut for you.

The numbers above are the reason you never pick a single global default and walk away. A region choice worth about 90 ms for your EU users, made once, beats any client side tuning you will do later. And the residency requirement did not cost you anything extra, because the low latency EU region was also the compliant one. That is the happy case, and it is common: the fast choice and the legal choice are often the same region.

Here is a small client that puts the region decision in code and survives a 429. It uses the Google Gen AI SDK, picks a location per request, and retries with exponential backoff before giving up. Read the location argument as the one lever this whole part is about.

import time
from google import genai
from google.genai.errors import ServerError, ClientError

PROJECT = 'my-project'

def make_client(location):
    # location is the one lever: a region like 'europe-west4'
    # for EU residency, or 'global' for the widest capacity pool.
    return genai.Client(vertexai=True, project=PROJECT, location=location)

def ask(location, prompt, retries=4):
    client = make_client(location)
    delay = 1.0
    for attempt in range(retries):
        try:
            r = client.models.generate_content(
                model='gemini-2.5-flash', contents=prompt)
            return r.text
        except (ServerError, ClientError) as e:
            # 429 RESOURCE_EXHAUSTED lands here under a shared-pool spike
            if attempt == retries - 1:
                raise
            time.sleep(delay)
            delay *= 2   # 1s, 2s, 4s, then fail loudly
    return None

# EU users stay in region for residency, Tokyo can ride global
print(ask('europe-west4', 'Summarize our refund policy in one line.'))
print(ask('global', 'Draft a greeting for a new user.'))

Expected output: two lines of generated text, one processed in the EU region and one from whichever region the global endpoint chose. Under load, you will see a pause of one, two, or four seconds where a retry absorbed a 429 before a response came back.

Failure mode: if every retry hits a 429, the call raises after the last attempt, which is correct: better a loud failure your caller can handle than a silent hang. Blind retries without backoff make a busy pool worse for everyone, so keep the doubling delay. And do not route EU residency traffic to the global location to dodge a 429, because that trades a latency blip for a compliance breach.

Default to regional, add global as your overflow valve

My recommendation is plain. Start on a regional endpoint chosen from two inputs in this order: where your data is allowed to live, then where your users sit. Regional gives you residency and every feature, and for most workloads a well chosen region is all you ever need. Do not open with the global endpoint just because it sounds resilient, because you cannot walk residency back once data has left the jurisdiction.

Then add the global endpoint as an overflow valve for the traffic that is allowed to use it. Keep residency bound traffic pinned to its region, and let everything stateless and unconstrained fall back to global when a region runs hot. On quota, assume Dynamic Shared Quota and build a client that retries a 429 with backoff rather than trusting a number. Buy Provisioned Throughput only where a peak time 429 would cost you a user or a contract, and size it for your sustained peak, not your worst minute ever.

One thing to do this week: open your client code and find the location string. If it is a single hardcoded region serving users on two continents, or a global default sitting under data that has a residency rule, you already found your first fix. Part 9 goes private from here, moving from public endpoints to Private Service Connect and VPC Service Controls, so the region you just chose is reachable only from inside your own network.

The same three levers exist on other clouds under different names. The AWS build of this decision is Amazon Bedrock regions, quotas, and cross region inference, where cross region inference plays the role the global endpoint plays here, and the quota model is closer to the older fixed style.

References

• Vertex AI generative AI deployments and endpoints
• Vertex AI generative AI data residency
• Generative AI on Vertex AI quotas and system limits
• Use Provisioned Throughput on Vertex AI

About The Author

Dr. Pranay Jha

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

See author's posts