The security review came back with one line highlighted in red: your Gemini traffic leaves the building over the public internet. The model call worked, the region was right, the residency box was ticked. None of that mattered to the reviewer, because the request still travelled to a public Google address, and a public address is a public address whether or not anyone abused it.
This part closes that gap. Two Google Cloud features do the work, and teams mix them up constantly. Private Service Connect, or PSC, gives your traffic a private door into Vertex AI over an internal IP address, so nothing rides the public internet. VPC Service Controls, or VPC-SC, draws a wall around your project so data cannot be pulled out to somewhere it was never meant to go. One is a path. The other is a boundary. You almost always want both, and turning one on without the other is where the outages and the false confidence come from.
Key takeaways
Private Service Connect gives Vertex AI traffic a private internal IP path so requests never touch the public internet, but on its own it does nothing to stop data leaving your project. VPC Service Controls is the boundary that stops exfiltration: put the Vertex AI API inside a service perimeter and public internet access to it is blocked, while ingress and egress rules decide exactly who and what may cross. There are two PSC shapes to know, a private path to the Google APIs and a private path to specific producer services like Vector Search, and you pick by what you are calling. Turn the perimeter on in dry run first, read what it would have denied, add ingress rules for the legitimate calls, then enforce. Skip that and you take down your own pipelines the day you flip the switch.
What Private Service Connect gives you
Private Service Connect is a way to reach a Google service using an internal IP address from inside your own VPC network. A VPC network, or Virtual Private Cloud, is the private network your resources live in. Normally a call to Vertex AI resolves to a public address on googleapis.com and leaves your network to get there. With PSC you create a small piece of plumbing called a forwarding rule, which is just a rule that says traffic to this internal IP goes to that Google service, and you point it at an internal IP you own, such as 10.10.0.5. Your clients now talk to 10.10.0.5, PSC does the address translation behind the scenes, and the request reaches Vertex AI without ever using a public IP.
The direction matters. PSC here is one way: your workload reaches into the Google-managed service, and the service does not reach back out into your network. That is the property a security team wants. On the producer side, the side Google runs, there is a VPC network created only for you that holds the service resources, and it connects to your endpoint through a service attachment, the published handle that a consumer endpoint attaches to. You do not manage that side. You create the endpoint, choose the internal IP, and Google does the rest.
Be clear about what PSC is not. It is a private route, nothing more. It does not decide who is allowed to call, and it does not stop a request that has left your intended boundary. A client with valid credentials and a network path can still send your data somewhere it should not go, and PSC will happily carry it over an internal IP the whole way. Private does not mean safe from exfiltration. That job belongs to the wall, which is the next feature.
Two private paths, and which one you actually need
PSC shows up in two shapes on Vertex AI, and picking the wrong one wastes an afternoon. The first is PSC for Google APIs. This gives you one private endpoint that stands in for the public API surface, so a call to aiplatform.googleapis.com resolves to your internal IP instead of a public one. You reserve a global internal address, attach a forwarding rule to a bundle of Google APIs, and add a private DNS record so googleapis.com points at the endpoint. This is the path most teams want first, because it covers the everyday control plane calls: submit a prompt, start a tuning job, list your models.
The second shape is a PSC endpoint to a specific producer service. Some Vertex AI services do not sit behind the shared API surface and instead publish their own service attachment that you connect a consumer endpoint to. Vector Search index queries work this way, and so do dedicated private prediction endpoints and Agent Engine. Here the endpoint IP has to sit in the same region as the service attachment, because the producer service is regional and the plumbing is pinned to that region. If your workload is high volume vector lookups or a private online prediction endpoint, this is the shape you build.
One caveat that trips people. Google does not support PSC backends, a different construct used for load-balanced access, with Vertex AI online prediction endpoints. Reach for the documented PSC endpoint or dedicated private endpoint pattern instead of trying to bolt a backend onto a prediction endpoint. It is a small distinction with a large error message if you get it wrong.
| Access method | What it reaches | Scope | Reach for it when |
|---|---|---|---|
| PSC for Google APIs | aiplatform.googleapis.com control plane | Global endpoint, one internal IP | Everyday model and job calls, private |
| PSC endpoint to a service | Vector Search, dedicated prediction, Agent Engine | Regional, same region as the attachment | High volume data plane, private online inference |
| Private Google Access | Google APIs from a subnet with no external IP | Subnet level, still public IP ranges | Simple private egress, no dedicated endpoint |
Here is the PSC for Google APIs path in gcloud, the one most teams build first. It reserves an internal IP, points it at the VPC-SC-safe bundle of APIs, and wires DNS so googleapis.com resolves to the endpoint. Read the bundle flag closely, because it is the one line that decides whether this endpoint helps your perimeter or quietly widens the hole.
gcloud compute addresses create psc-vertex-ip
--global
--purpose=PRIVATE_SERVICE_CONNECT
--addresses=10.10.0.5
--network=prod-vpc
# point the endpoint at APIs that VPC-SC protects, not every API
gcloud compute forwarding-rules create psc-vertex-apis
--global
--network=prod-vpc
--address=psc-vertex-ip
--target-google-apis-bundle=vpc-sc
# make googleapis.com resolve to the endpoint inside the VPC
gcloud dns managed-zones create psc-googleapis
--dns-name=googleapis.com.
--visibility=private
--networks=prod-vpc
--description='route Google API traffic to the PSC endpoint'
gcloud dns record-sets create '*.googleapis.com.'
--zone=psc-googleapis --type=A --ttl=300
--rrdatas=10.10.0.5Expected output: three resources created, then from a VM in prod-vpc a lookup of aiplatform.googleapis.com resolves to 10.10.0.5, and a Vertex AI call succeeds while staying on the internal IP. A describe of the forwarding rule shows the vpc-sc bundle and your address.
Failure mode: pick all-apis instead of vpc-sc and the endpoint also reaches APIs the perimeter does not protect, so you widen the exfil path the wall was meant to close. Skip the DNS record and clients still resolve the public googleapis.com address, so traffic leaves over the internet even though the endpoint exists. The endpoint without the DNS is a door no one walks through.
A VPC Service Controls perimeter, and what it blocks
VPC Service Controls draws a boundary, called a service perimeter, around a set of projects and the Google services inside them. Put the Vertex AI API into the perimeter as a protected service and one thing happens immediately: public internet access to that API is blocked. A request from outside the boundary, with no rule permitting it, is denied before it reaches your data. The perimeter also stops the reverse move, a request from inside trying to send data to a Vertex resource in some other project outside the wall. That second direction is the whole point, because most data loss is not an outsider breaking in, it is a credential inside quietly copying data out.
Crossing the wall is not all or nothing. Two rule types govern the edge. An ingress rule permits specific traffic to come in, matched on attributes like the calling identity, the source project, or a source network. An egress rule permits specific traffic to go out to a named service or project. You start closed and open only the paths you can name. A service account in a partner project that has to read your feature store gets an ingress rule that names exactly that identity and that service, and nothing else changes.
The Google-managed environment that runs your Vertex AI workloads also loses its default internet access once the project is in a perimeter. That is deliberate, because an open path out is an open path for data to leave. It is also the single most common reason a first perimeter rollout breaks something, and I will come back to it, because a training job that used to pull a package from the public internet will now fail, and it will fail quietly if you are not watching for it.
Why PSC alone will not stop exfiltration
Here is the confusion I see most often. A team stands up PSC, watches their Vertex traffic move to an internal IP, and reports back that the platform is now secure. It is more private. It is not more contained. PSC changed the road the request travels, not the rule about where the request is allowed to end up. A compromised service account inside the project can still call Vertex AI and stream your data to a bucket in an attacker project, and because that call goes over your internal path, it looks like normal internal traffic the whole way.
VPC-SC is the piece that closes that. The perimeter denies the call to the outside project regardless of how private the network path is, because it checks the destination against the boundary, not the wire. So the two features answer different questions. PSC answers how does my traffic get there without the public internet. VPC-SC answers where is my traffic allowed to end up. A real setup wants both: PSC so nothing rides a public IP, VPC-SC so a valid credential cannot carry your data past the wall.
| Question | Private Service Connect | VPC Service Controls |
|---|---|---|
| What it is | A private internal IP path to a service | A boundary around projects and services |
| Keeps traffic off the public internet | Yes | Not its job |
| Stops data leaving to another project | No | Yes |
| Controls who may call | No, that is IAM | Partly, via ingress and egress rules |
| Can break a job on day one | Rarely | Yes, plan the rollout |
Rolling out a perimeter without breaking your pipelines
The day you enforce a perimeter is the day things that quietly reached out over the internet stop working. Vertex AI Pipelines that pull a package from a public index fail, because the managed environment lost its default internet access. A component that fetches a dataset from a public URL fails for the same reason. Request and response logging is not available with VPC Service Controls, so if you leaned on that for debugging, it goes dark. None of these are bugs. They are the wall doing its job, and every one of them can be planned for before it bites.
The tool that saves you is dry run mode. A perimeter can run in a mode where it logs what it would have denied without actually denying it. You leave it in dry run for a day or two of real traffic, pull the audit logs, and read the list of would-be blocks. Every legitimate call in that list needs either an ingress or egress rule before you enforce. The illegitimate ones are the exfil attempts you just found. Only when the legitimate list is covered do you flip from dry run to enforced.
Order also matters for some resources. A few Vertex AI endpoints have to be created only after the project is already inside the perimeter, because the plumbing is built to respect the boundary from the start. Create them first and you get to tear them down and rebuild them, which is a bad afternoon during a migration. Read the current list of these before you sequence the work, because it changes as services move to the model.
Worked example
A team runs the perimeter in dry run for 48 hours across five projects. The audit logs show 1,240 requests that would have been denied. Reading them by category: 1,180 are legitimate cross-project calls, mostly two partner service accounts reading a feature store and a CI runner deploying models. 45 are pipeline components fetching public packages, which need an egress path, not an open door. 15 are calls from an identity no one recognises, going to a project outside the org.
The plan writes itself. Add ingress rules that name the two partner service accounts and the CI identity, covering the 1,180. Route the 45 package fetches through an approved egress path so builds keep working. Investigate the 15, which are the reason you did this at all. After the rules land, a second dry run should show close to zero legitimate denials before you enforce.
The lesson: 1,180 of 1,240 would-be denials were your own traffic. Enforce on day one without this pass and you take down 95 percent of legitimate calls to catch the 15 that matter.
Private path plus perimeter, and the order I turn them on
My recommendation is to build both and to build them in this order. Start with the perimeter in dry run, because it is the piece that actually stops data loss and the piece most likely to break something, so you want its findings before you change anything else. Read the dry run, write ingress and egress rules for the legitimate traffic, and only then enforce. Do not lead with PSC and call it done, because a private path with no boundary is a faster road to the same exfiltration you started with.
Then add PSC so the traffic that the perimeter now allows travels on an internal IP instead of a public one. Use the vpc-sc bundle for the Google APIs path so the private door matches what the wall protects, and use a regional PSC endpoint for Vector Search or a dedicated prediction endpoint where the data plane needs it. When both are in place, a valid credential cannot carry your data past the wall, and nothing you send rides a public address to get where it is going. That is the pair the security reviewer was actually asking for.
One thing to do this week: turn on a dry run perimeter around one non-production project that calls Vertex AI, let it watch a day of traffic, and read the would-be-denied list. That single list tells you more about your real exposure than any architecture diagram. Part 10 picks up the identity half of this, moving from the network boundary to IAM, customer-managed encryption keys, and data governance, so who may call is controlled as tightly as where the call may go.
The same private access problem has an AWS shape, where the private path is PrivateLink and the boundary is drawn with VPC endpoints and policies. The mirror of this part is Amazon Bedrock private access with PrivateLink and VPC endpoints, which solves the same two problems with different plumbing.
References
- VPC Service Controls with Vertex AI
- Accessing Vertex AI through Private Service Connect endpoints
- Access Google APIs through Private Service Connect endpoints
- Vertex AI network access and security overview
About The Author
DrJha