Identity path from vCenter SSO to Pinniped Concierge
Authentication in a VKS estate runs through two Pinniped components and one decision you made months earlier. Pinniped is the open source authentication service Broadcom ships inside vSphere Supervisor. It splits into a Supervisor half that speaks OIDC to your identity provider, and a Concierge half that exchanges the resulting token for a short lived client certificate the Kubernetes API server will accept. Both halves run as pods on the Supervisor. Only the Concierge half runs on each VKS cluster, which is vSphere Kubernetes Service, the product Broadcom used to call TKG Service and before that TKGS.
That split explains most identity failures you will meet. With no external identity provider registered, VCF CLI forwards your login to vCenter Single Sign On, obtains a token, and writes the vSphere Namespaces you can reach into your kubeconfig. Register an OIDC provider and that path changes for every VKS cluster on that Supervisor at once, whether or not the team running cluster three asked for it. Resource names, API groups and documentation URLs still carry tkg throughout, so do not be surprised when a secret you are hunting is named after a Tanzu Kubernetes cluster. Nothing about the rename is finished on disk.
Read the bottom of that diagram twice, because it is where candidates lose marks. Access to a vSphere Namespace, which is the resource boundary the Supervisor projects into Kubernetes, is granted in the vSphere Client under the namespace Permissions card, where you pick an identity source and a role of Viewer, Editor or Owner. Access inside a VKS cluster is granted with ordinary Kubernetes RBAC, meaning Role Based Access Control, using Role, ClusterRole, RoleBinding and ClusterRoleBinding objects. One federated user can hold Editor on a vSphere Namespace and still be Forbidden inside the workload cluster because nobody wrote the RoleBinding. Those two surfaces share an identity and share nothing else. General Kubernetes RBAC mechanics are covered in the VKS Series and this Part does not re-teach them.
Preflight and version record
Three things must be true before you open the Identity Providers screen. Your Supervisor must be Ready, because registration writes to Pinniped custom resources and a Supervisor mid reconcile will accept the form and quietly fail to roll it out. Your identity provider must be OIDC compliant, meaning it publishes a discovery document at the issuer URL. And DNS used by both the Supervisor and the VKS clusters must resolve whatever hostname you put in the callback URL, because the browser redirect and the token validation happen from different places on the network.
Four Pinniped pods on a three zone Supervisor is what a healthy estate looks like. If pinniped-supervisor is missing entirely and only Concierge is present, you are looking at a Supervisor that has never been given an external provider, which is fine, or one that lost the component during an upgrade, which is not. Broadcom has a knowledge base article for the latter and it is worth knowing that the symptom is a Kubernetes status warning rather than a login failure.
Registration procedure in four steps
Step 1, collect the callback URL and three values from the IdP
Registration is a two sided handshake and the Supervisor side comes first. In the vSphere Client go to Supervisor Management, then Supervisors, then Configure, then Identity Providers, and copy the Identity Provider Callback URL shown there. It takes the form https://SUPERVISOR-VIP/wcp/pinniped/callback. Paste that value into your provider as the sign in redirect URI and, if the provider asks for one, the sign out redirect URI. Broadcom documents Okta, Workspace ONE Access, Dex, GitLab and Google OAuth 2 as tested providers, and any OIDC compliant provider will work. Set the grant type to Authorization Code and add Refresh Token, because without refresh the session dies at the first token expiry and your operators will blame the cluster.
Come back with three values: Issuer URL, Client ID and Client Secret. Before you leave the provider, configure the groups claim on the provider side as well. In Okta that lives under the Sign On tab, OpenID Connect ID Token section, where you set a Groups claim type filter. Matching the claim name groups against a regex of asterisk sends every group. That is fine for a lab and too broad for production, where you should filter to the prefix your platform groups use.
Step 2, fill the provider fields that decide authorisation
Click the plus sign on the Identity Providers screen and work through provider configuration, OAuth 2.0 client details, and additional settings. Broadcom marks four of those fields Required and the rest Optional, and that labelling is honest about registration and misleading about outcomes. Two Optional fields decide whether anyone can do anything. Keep the table below next to you while you fill the form; it is the artifact from this Part worth saving, an IdP field to failure lookup.
| Field | Marked | Get it wrong and this is the symptom |
|---|---|---|
| Provider Name | Required | Cosmetic. Appears in the vSphere Client and in the Pinniped resources. |
| Issuer URL | Required | OIDC discovery fails, registration never completes, no VKS cluster is touched. |
| Username Claim | Optional | Username becomes the issuer URL concatenated with the sub claim, so it reads as https://ISSUER?sub=UUID and every RoleBinding on a named user misses. |
| Groups Claim | Optional | No groups are carried from the provider at all. Every group RoleBinding misses silently and login still succeeds. |
| Client ID | Required | Browser stops at the provider with invalid_client before any consent screen. |
| Client Secret | Required | Consent succeeds, token exchange fails, callback returns an error page. |
| Additional Scopes | Optional | Claim names are right and the claims are absent, because the provider was never asked for that scope. |
| Certificate Authority Data | Optional | x509 certificate signed by unknown authority when the provider uses a private CA. |
| Additional Authorize Parameters | Optional | Provider specific. Usually a missing prompt or audience parameter the IdP demands. |
Step 3, bind an IdP group inside the VKS cluster
Namespace permission on the Supervisor is a vSphere Client action, so grant your federated group Editor on the team-payments vSphere Namespace there first. Inside the workload cluster you write RBAC yourself, and this is the object that proves whether the groups claim survived the journey.
Step 4, log in as a federated user and read the claims back
Never type a password on a command line and never bake one into a script. Export the username and let the browser flow carry the secret. Verification is not that login succeeded; login succeeds even when everything downstream is broken. Verification is what kubectl auth whoami prints, which needs a kubectl client of 1.28 or later.
Green looks exactly like that. A readable username and a named group. Anything else and you have work to do, no matter what the vSphere Client says about the provider being registered.
Empty groups claim, diagnosis and rollback
Here is the failure I promised, taken from a customer Supervisor where I had left both Optional claim fields blank because the form said Optional and I was in a hurry. Login worked. Everything after login did not.
Both blanks are visible in six lines of output. Username is an opaque issuer plus sub string rather than an email, which is what Broadcom documents will happen when Username Claim is empty. Groups contains only system:authenticated, which every authenticated caller gets for free, so the k8s-platform-admins RoleBinding never matched. No error was raised at registration time and no warning appeared in the vSphere Client. That cost me ninety minutes and two people staring at a perfectly correct RoleBinding.
Fixing it is an edit, not a rebuild. Return to Supervisor Management, Supervisors, Configure, Identity Providers, edit the registered provider, set Username Claim to email and Groups Claim to groups, and save. Pinniped rewrites its resources and the Concierge pods restart on the Supervisor and on every VKS cluster. Users must log out and log in again, because the old ID token in their cache still carries no groups. Repeat the kubectl auth whoami check and expect the readable username and named group from Step 4.
Backing out an IdP registration safely
Removing the provider from that same screen returns the Supervisor to vCenter Single Sign On for VCF CLI logins. Deletion is as wide as registration was, so every VKS cluster reverts together and every federated session is invalidated. Do two things before you remove anything. Confirm at least one vCenter Single Sign On account still holds Owner on the vSphere Namespaces you care about, otherwise you will hand back a namespace nobody can administer. And pull the kubeconfig secret for each cluster to a safe location first, so that a botched revert leaves you a way in. That is the whole point of the next section.
Break glass access with cluster secrets
Every VKS cluster carries a small set of secrets in its vSphere Namespace on the Supervisor, and objective 2.4 names Kubernetes admin credentials explicitly, so know these by name and by purpose. Broadcom is direct about the intent of the kubeconfig one: it exists so you can reach a cluster when vCenter Single Sign On authentication is not available.
| Secret name | What it holds | Rotation |
|---|---|---|
| CLUSTER-kubeconfig | A kubeconfig for the kubernetes-admin user on the cluster control plane | Managed by the cluster controllers. Treat as break glass, not a daily driver. |
| CLUSTER-ca | Root CA certificate for the cluster control plane used by kubectl | Rotates with cluster certificate lifecycle. |
| CLUSTER-ssh | SSH private key for the vmware-system-user account on any node | Used for node level troubleshooting only. |
| CLUSTER-ssh-password | Password for vmware-system-user on any node | Used for node level troubleshooting only. |
| CLUSTER-ccm-token-RANDOM | Service account token for the paravirtual cloud controller manager | Delete the secret to trigger rotation. |
| CLUSTER-pvcsi-token-RANDOM | Service account token for the paravirtual CSI plug in | Delete the secret to trigger rotation. |
Notice what that identity is. Username kubernetes-admin, group system:masters, which is unconditional cluster-admin that bypasses every RoleBinding you just wrote and leaves no useful attribution in an audit log. A great deal of published guidance treats this secret as the convenient way to give platform engineers access when SSO is inconvenient. Reject that advice. It converts your carefully federated estate into a shared root password, and on this exam it is also the wrong answer to any question that includes the phrase least privilege. Extract it during an incident, use it, then destroy the local copy.
Failures to remediation
| What you see | Cause | Fix |
|---|---|---|
| Groups shows only system:authenticated | Groups Claim blank, or the provider is not emitting a groups claim | Set Groups Claim to groups, add the groups claim filter on the provider, log out and back in |
| Username reads as an issuer URL with a sub query string | Username Claim blank | Set Username Claim to email or preferred_username, then rewrite any RoleBinding that named the old value |
| Browser stops at the provider with invalid_client | Client ID mismatch between provider and Supervisor | Recopy Client ID, confirm you are looking at the same application integration |
| Callback returns an error page after consent | Redirect URI on the provider does not match the Supervisor callback URL exactly | Repaste https://SUPERVISOR-VIP/wcp/pinniped/callback, check for a trailing slash and for FQDN versus VIP drift |
| x509 certificate signed by unknown authority during login | Provider fronted by a private CA and Certificate Authority Data left empty | Paste the base64 encoded CA chain into Certificate Authority Data and save |
| Login works on Supervisor, Forbidden on one VKS cluster only | Concierge on that cluster has not finished reconciling, or nobody wrote cluster RBAC | Check Pinniped pods on that cluster, then apply the RoleBinding for the group |
| Sessions expire after a short period and users must re-authenticate constantly | Refresh Token grant not enabled on the provider application | Add Refresh Token to the grant types alongside Authorization Code |
Exam focus for objective 2.4
Objective checkpoint
| Original practice question | Answer and reasoning |
|---|---|
| An administrator registers an OIDC provider on a Supervisor that hosts four VKS clusters. Which clusters use the new provider? | All four. Registration is a Supervisor level object and the system reconfigures Pinniped Concierge on every VKS cluster on that Supervisor. |
| A federated user logs in successfully but every group RoleBinding is ignored. Which single setting is the most likely cause? | Groups Claim left empty on the provider registration. When it is blank no groups are carried from the provider, so only system:authenticated is present and authentication still succeeds. |
| vCenter Single Sign On is unavailable and a VKS cluster must be reached to collect diagnostics. Which secret provides administrator access, and what identity does it carry? | CLUSTER-kubeconfig, in the cluster vSphere Namespace. It authenticates as kubernetes-admin in the system:masters group, which is why it is a break glass credential rather than a standing grant. |
Identity plan for the reference estate
My recommendation for this lab, and for most estates that look like it, is one external provider registered at the Supervisor with Username Claim set to email and Groups Claim set to groups from day one, a small number of platform groups rather than per team groups, and vCenter Single Sign On retained for vSphere administrators who manage namespaces in the vSphere Client. Avoid the pattern where operators keep an extracted kubeconfig on their laptop because federated login is thirty seconds slower on the first attempt. That trade is not worth it: you lose attribution, you lose revocation, and on exam day it is the answer that looks convenient and scores zero.
Tonight, in your own lab, run kubectl auth whoami against your Supervisor context and against one VKS cluster. If Groups shows anything other than a named group you recognise, you have the bug from this Part sitting in production right now, and you can fix it in about four minutes plus a rollout. Part 9 picks the lab up at Kubernetes releases and content libraries, including the air gapped case, which is where the next objective lands.
References
- Register an External Identity Provider with Supervisor, VMware Cloud Foundation 9.0, Broadcom TechDocs
- Configure an External Identity Provider for VKS Clusters, VMware Cloud Foundation 9.0, Broadcom TechDocs
- Get VKS Cluster Secrets Using Kubectl, vSphere Supervisor 9.0 Services and Standalone Components, Broadcom TechDocs
- VMware Cloud Foundation VKS Administrator Exam Guide 3V0-24.25, Broadcom
About The Author
DrJha