Most Vertex AI data problems I have been called in to clean up did not start with a clever attacker or a broken model. They started with a single line, run once, months before anyone noticed it mattered:
gcloud projects add-iam-policy-binding my-project
--member='group:engineering@example.com'
--role='roles/aiplatform.admin'One command. The whole engineering group. Full administrative control over every model, endpoint, tuning job, and stored dataset in the project. It felt reasonable on a Friday afternoon. It is also the reason a service account nobody remembered could still read a tuned model six months later, long after the person who created it had left. This part is about the three controls that keep that line from becoming an incident: who may call the models, who holds the key that encrypts what they produce, and what Google and you can see and keep after the request is done. Vertex AI is Google Cloud’s managed platform for building with generative models, and these three controls are the identity and data layer that sits inside the network boundary you drew in Part 9.
Key takeaways
Identity and Access Management, or IAM, answers one question: who may call. The predefined roles roles/aiplatform.user and roles/aiplatform.admin are wider than most callers need, so scope a custom role to the single permission a client actually uses, which for a prediction client is aiplatform.endpoints.predict. Customer-managed encryption keys, or CMEK, put a key you control in Cloud KMS in front of the data Vertex AI stores at rest, but it only works when you grant the Vertex AI service agent the CryptoKey Encrypter/Decrypter role and keep the key in the same region as the resource. Google does not use your prompts, responses, or tuning data to train its foundation models, though abuse monitoring can log prompts for up to 90 days unless your workload qualifies for zero data retention. Admin Activity audit logs are always on, but Data Access logs, the ones that record who read what, are off by default, so turn them on before you need them, not after an incident when they would have helped.
Three controls, three questions
The network boundary from Part 9 decided where a request may travel. It said nothing about who is allowed to send it, whether the data it leaves behind is encrypted under a key you hold, or whether anyone can prove after the fact who read what. Those are three separate questions, and Google Cloud answers them with three separate controls. Mixing them up is where teams get a false sense of coverage, because each one is silent about the other two.
IAM answers who may call. It grants a principal, which is any identity such as a user, a group, or a service account, a role, which is a bundle of permissions, on a resource. CMEK answers who holds the key. By default Google encrypts your data at rest with keys it manages, and CMEK swaps in a key you own in Cloud KMS so you can rotate it, or destroy it, on your terms. Governance and audit answer what was seen and kept. Cloud Audit Logs record the calls, Access Transparency records the rare times Google staff touch your content, and Google’s data commitments define what is retained and what is never used to train a model.
You want all three, and you want them in a deliberate order. Identity first, because a call that should never have been allowed is the cheapest thing to stop. Encryption next, because it decides who can read the data even if they reach it. Audit last, because it is how you learn what actually happened. The rest of this part takes them one at a time.
IAM roles for Vertex AI, and why the predefined ones are too wide
Vertex AI ships three predefined roles that cover most of what people reach for. Vertex AI Viewer, roles/aiplatform.viewer, is read only: it can list and describe resources but change nothing. Vertex AI User, roles/aiplatform.user, is the working role: it can submit predictions, run jobs, and use the resources in a project. Vertex AI Administrator, roles/aiplatform.admin, is full control, including creating and deleting endpoints and persistent resources. These exist to make setup quick, and quick is exactly the trap. A prediction client that only ever calls a model gets handed roles/aiplatform.user because it is right there, and now that same identity can also start a tuning job and delete an endpoint it should never touch.
Google’s own guidance is to prefer custom roles for anything that faces production, because you build them and you decide exactly which permissions go in. A permission is a single verb on a single resource type, written like aiplatform.endpoints.predict, which is the one a client needs to send a prompt to a deployed endpoint. Put that permission, and only that permission, in a custom role and bind it to your service account. The client keeps working. The blast radius of a leaked credential shrinks from administrator to caller.
There is a fourth identity that surprises people: the service agent. When you enable Vertex AI, Google creates a managed service account named service-PROJECT_NUMBER@gcp-sa-aiplatform.iam.gserviceaccount.com that acts on your behalf to run jobs and, as you will see next, to use your encryption key. You do not create it and you rarely grant it much, but when CMEK or a cross-service job fails with a permission error, this agent is usually the identity that was missing a grant. Know its name before you need it.
| Role | What it can do | Reach for it when | Watch out for |
|---|---|---|---|
| roles/aiplatform.viewer | List and describe, no changes | Dashboards, read-only audit | Can still read resource metadata |
| roles/aiplatform.user | Predict, run jobs, use resources | A human doing hands-on work | Too wide for a predict-only client |
| roles/aiplatform.admin | Full control, create and delete | Platform owners only | A leaked key is a leaked admin |
| custom, predict only | Just aiplatform.endpoints.predict | Any production caller | You maintain the permission list |
Here is the least-privilege version in gcloud. It defines a custom role holding only what a prediction client uses, then binds that role to a service account. Read the permissions flag as the whole point: two verbs, not two hundred.
gcloud iam roles create vertexPredictOnly
--project=my-project
--title='Vertex Predict Only'
--description='call a deployed endpoint, nothing else'
--permissions=aiplatform.endpoints.predict,aiplatform.endpoints.get
--stage=GA
# bind the narrow role to the app service account
gcloud projects add-iam-policy-binding my-project
--member='serviceAccount:app@my-project.iam.gserviceaccount.com'
--role='projects/my-project/roles/vertexPredictOnly'Expected output: a custom role created with two permissions, then a policy binding added. The service account can now call generate_content against a deployed endpoint, but an attempt to create or delete an endpoint returns PERMISSION_DENIED, which is the point.
Failure mode: grant roles/aiplatform.user instead and the same account can submit tuning jobs, deploy models, and delete endpoints, none of which a prediction client needs. A leaked key for that account is now a leaked operator, not a leaked caller. The convenience of the broad role is paid back in full the day the credential ends up somewhere it should not be.
How CMEK actually changes the picture
Everything Vertex AI stores for you is already encrypted at rest, by default with keys Google generates and manages. For many teams that is enough. CMEK is for the teams whose contract or regulator says the encryption key must be one the customer controls, rotates, and can revoke. With CMEK you create a key in Cloud KMS, Google’s managed key service, inside a key ring, which is just a regional container for keys. Vertex AI then uses your key to wrap the data-encryption keys that protect the resource, so nothing is readable at rest without your key being available.
Two rules decide whether this works on the first try. First, the key has to live in the same region as the resource it protects. A key in us-east1 cannot encrypt a Vertex AI resource in us-central1, and the create call fails outright rather than falling back. Second, the Vertex AI service agent from the last section needs the CryptoKey Encrypter/Decrypter role, roles/cloudkms.cryptoKeyEncrypterDecrypter, granted on that key. Without it the agent cannot wrap or unwrap data, and the resource never finishes creating. Both errors read like something is broken. Neither is. They are the alignment CMEK requires, and skipping either is the most common reason a first CMEK rollout stalls.
Be clear about what CMEK does and does not buy you. It gives you a kill switch: disable the key and every resource that depends on it goes dark, which is drastic and dangerous in equal measure. It does not decide who may call the model, that is still IAM, and it does not by itself hide data from Google personnel during a support action, that visibility is what Access Transparency reports. CMEK is control over the key, nothing more and nothing less. That is worth real money to a regulated team and close to nothing to a team without that obligation, so match it to the requirement rather than turning it on for the feeling of safety.
gcloud kms keyrings create vertex-ring
--location=us-central1
gcloud kms keys create vertex-key
--location=us-central1 --keyring=vertex-ring
--purpose=encryption
--rotation-period=90d
--next-rotation-time=2026-10-01T00:00:00Z
# let the Vertex AI service agent use the key
gcloud kms keys add-iam-policy-binding vertex-key
--location=us-central1 --keyring=vertex-ring
--member='serviceAccount:service-PROJECT_NUMBER@gcp-sa-aiplatform.iam.gserviceaccount.com'
--role='roles/cloudkms.cryptoKeyEncrypterDecrypter'Expected output: a key ring and a key in us-central1 with 90 day rotation, then a policy binding that lets the aiplatform service agent wrap and unwrap data with it. A Vertex AI resource created with the kms-key-name flag pointing at vertex-key now encrypts its data at rest under your key.
Failure mode: put the key in us-east1 while the Vertex resource lives in us-central1 and the create call fails, because the key and the resource must share a region. Skip the service-agent binding and the resource cannot use the key at all, so it hangs and never creates. Disable or destroy the key later and every resource that depends on it stops serving, which is the kill switch working as designed, not an outage to chase.
What Google can see, and what it keeps
The question I get from every legal team is the same: what happens to our prompts. The baseline commitment is clear and worth stating plainly. Google does not use your prompts, your model responses, or your tuning data to train its foundation models. When you tune a Gemini model, the private data and outputs stay yours and never enter the training corpus. That is a contractual position, not a setting you toggle, and it is the reason a regulated workload can run on Vertex AI at all.
There is one nuance to know, because it catches people. Abuse monitoring runs automated safety classifiers over traffic, and if a request trips a classifier, Google may log the prompt to check whether it breached the acceptable use policy. That logged content is stored securely for up to 90 days, in the same region or multi-region you selected, and it is not used to train or tune any model. For most teams that is an acceptable safety trade. For a bank or a hospital it may not be, which is why zero data retention exists: eligible customers on eligible models can turn off that retention so flagged content is not stored at all. Confirm your model and account qualify before you promise a regulator a hard zero.
So the honest summary is layered. Prompt content is never training data. It may be retained briefly for abuse review unless you hold zero data retention. Your own logging, on the other hand, is entirely your responsibility, and the same warning from Part 8 applies: Vertex AI can keep zero prompt content while your application quietly writes every prompt to a log sink that keeps it for a year. Trace where the data goes after the model call, not just what Google does with it.
| Data category | Used to train foundation models | Default retention |
|---|---|---|
| Prompts and responses | No | Not retained after the response, 0 days |
| Tuning data and tuned model | No, stays private to you | Kept until you delete it |
| Abuse-monitoring logs, flagged only | No | Up to 90 days, 0 with zero data retention |
| Data Access audit logs | No | Off by default, 30 days once on |
| Admin Activity audit logs | No | Always on, 400 days, metadata only |
Worked example
A bank runs a Gemini workload under a rule that no customer prompt may persist beyond the transaction. Out of the box, three things can retain data: context caching if it is enabled, abuse-monitoring logs for up to 90 days on flagged prompts, and Data Access audit logs at the Cloud Logging default of 30 days once you switch them on. Admin Activity logs hold metadata, who called and when, not the prompt text, for 400 days.
The bank closes each content path in turn. It applies for zero data retention on its model, which drops the abuse-monitoring retention from 90 days to 0. It disables context caching. Prompt content now persists for 0 days. What remains is audit metadata: 30 days of Data Access records showing which service account called, and 400 days of Admin Activity records, neither of which contains a customer prompt.
The compliance line was met not by one master switch but by naming every retention path and closing the ones that held content, while keeping the ones that only hold metadata. 0, 90, 30, and 400 are four different numbers for four different things, and treating them as one is how teams either over-retain or lose their audit trail.
Audit logs and Access Transparency, the part teams skip
Cloud Audit Logs are the record of who did what on aiplatform.googleapis.com, the service name Vertex AI logs under. They come in two kinds that behave very differently. Admin Activity logs capture changes to configuration, creating an endpoint, editing a role binding, and they are always on and cannot be turned off. Data Access logs capture reads of your data and the calls that touched it, and they are off by default because they can be high volume. That default is the trap. The log that would tell you which identity read a tuned model last Tuesday is the one nobody enabled until the week they needed it and found nothing there.
Access Transparency is a different lens, pointed at Google rather than at your own users. In the rare cases a Google employee accesses your content, during a support ticket you opened, for example, Access Transparency writes a near real time log of what they touched, which operation they ran, and the justification for it. It arrives in Cloud Logging alongside your audit logs. For a regulated buyer this is often the control that closes the deal, because it turns Google’s side of the boundary from a promise into a log line you can read. Enabling it is an organisation-level step, not a per-project one, so it usually belongs to the platform team.
Least privilege by default, CMEK by exception
Here is how I would set this up on a new project. Start with identity, because it is the cheapest control and the one most often left wide. Give humans a group with a scoped role, give each workload a service account with a custom role holding only the permissions it uses, and keep roles/aiplatform.admin for the two or three people who actually own the platform. Do not reach for the predefined user role just because it is one click. A predict-only custom role is barely more work and it is the difference between a leaked caller and a leaked operator.
Treat CMEK as an exception, not a default. Turn it on where a contract or a regulator names customer-controlled keys, and size the operational cost honestly: key rotation, the availability risk of a kill switch, and the same-region constraint that shapes where resources can live. Where there is no such requirement, Google-managed encryption is already protecting your data at rest and CMEK adds work without adding coverage. On governance, enable Data Access audit logs immediately, apply for zero data retention if a regulator needs a hard zero on prompt content, and confirm the current retention terms on the live docs because they move. Least privilege is the rule for everyone. CMEK is the rule for the teams who can name why.
One thing to do this week: run gcloud projects get-iam-policy on your main Vertex AI project and read every binding that grants roles/aiplatform.admin or roles/aiplatform.user. For each one, ask which single task needs it. The bindings that cannot answer are your first cleanup, and they are almost always where a leaked credential would do the most damage. Part 11 moves from securing the platform to calling it well, with the Gemini API, streaming responses, and function calling, so the identity you just tightened is the one making real requests.
The same identity and key problem has an AWS shape, where roles are IAM policies and the key sits in KMS rather than Cloud KMS. The mirror of this part is Amazon Bedrock data residency, KMS, and security, which solves who may call and who holds the key with different plumbing and the same logic.
References
- Vertex AI access control with IAM
- Vertex AI customer-managed encryption keys
- Generative AI and data governance on Vertex AI
- Access Transparency in Vertex AI
About The Author
DrJha