, ,

Identity, RBAC and Secrets Assessment for TKGI to VKS (TKGI to VKS Series, Part 9)

Identity does not ride along with a Velero backup. Assess TKGI UAA, LDAP and RBAC against vCenter SSO, Pinniped and VKS namespaces, and reseal secrets before you cut over.

TKGI to VKS Series · Part 9 of 26

Who can delete a namespace the morning after we cut over? A security lead asked me exactly that in a migration review, and I did not have a clean answer, because the honest one is that nobody can until you rebuild the access model by hand. Identity is the piece of this migration that refuses to ride along with a backup. RoleBindings back up fine and restore fine, and then they grant access to no one, because the names they point at stopped existing the moment you left UAA behind. Last part we mapped every persistent volume to a storage policy and a migration method. This part maps who can do what, and it finds every binding that a restore will quietly break before the restore breaks it.

Who this is for: A platform engineer, cluster admin or security-minded operator holding a TKGI 1.18 estate on NSX-T and planning a VKS target on VCF 9. Terms on first use: UAA is User Account and Authentication, the OAuth2 server TKGI uses to issue tokens; LDAP is Lightweight Directory Access Protocol, the directory UAA delegates to; OIDC is OpenID Connect, the identity layer a Kubernetes API trusts; RBAC is Role Based Access Control, the Kubernetes model of Roles and bindings; vCenter SSO is vCenter Single Sign On, the identity service for vSphere; Pinniped is the authentication service VKS uses to federate an external OIDC provider into a guest cluster; a vSphere Namespace is the Supervisor object that carves out resources and access for a team; a ServiceAccount is a non-human identity a workload uses to call the API.
Key takeaways: Identity does not back up and restore, so inventory it as its own workstream. TKGI authenticates through UAA delegating to LDAP, VKS authenticates through vCenter SSO for the platform and Pinniped for external OIDC into guest clusters, and the subject names differ on every binding. Kubernetes Roles and ClusterRoles are portable, but their subjects are not, so a blind apply of your RoleBindings grants access to nobody. Each Supervisor supports exactly one external identity provider, which forces a real choice between SSO federation and a separate OIDC plane. Rebuild ServiceAccount tokens and reseal every secret on the target, never carry a base64 secret across as if it were data.

Identity inventory you cannot skip

A storage inventory keys on the disk. An identity inventory keys on the subject, the name a binding grants power to, because that name is the thing that will or will not survive the move. Two kinds of subject live in a TKGI cluster. Human subjects arrive with an oidc: prefix, the marker UAA stamps on identities it issued after chaining to LDAP. Machine subjects arrive as system:serviceaccount references, tokens minted inside the cluster. Both need to be enumerated, and neither carries across untouched, so pull the full list from every cluster before you design a single thing on the target.

# tested against TKGI 1.18 with UAA and LDAP, target VCF 9.0 VKS, kubectl 1.30, kubectl vSphere plugin 8.x kubectl get clusterrolebinding,rolebinding -A -o custom-columns="KIND:.kind,NAME:.metadata.name,ROLE:.roleRef.name,SUBJECTS:.subjects[*].name" # expected output, prod cluster, trimmed KIND NAME ROLE SUBJECTS ClusterRoleBinding platform-admins cluster-admin oidc:platform-admins ClusterRoleBinding ci-deployer edit system:serviceaccount:ci:deployer RoleBinding app-team-a-edit edit oidc:app-team-a RoleBinding app-team-b-view view oidc:app-team-b

Across the reference estate, three clusters covering dev, staging and prod, that pass returned about 40 RoleBindings, 8 ClusterRoleBindings, 12 distinct LDAP groups behind the oidc: subjects, and 63 ServiceAccounts once you exclude the default one per namespace. Small enough to read in an afternoon, large enough that nobody remembers all of it, which is the whole reason you write it down. Note which bindings grant cluster-admin, because those are the ones a mistake during cutover turns into either a lockout or an over-grant, and both are incidents.

One layer of TKGI access does not show up in kubectl at all. Cluster lifecycle rights live in UAA as scopes, not as Kubernetes RBAC, and they govern who may create, resize or delete a cluster through the TKGI API. Read them from UAA so they are on the sheet too, because they map to a completely different place on VKS.

# UAA scopes grant TKGI cluster management, separate from in-cluster RBAC, read from env, never inline a secret uaac target https://uaa.tkgi.example:8443 uaac token client get admin -s "$UAA_ADMIN_SECRET" uaac groups | grep pks.clusters # expected pks.clusters.admin pks.clusters.manage

Two scopes, two destinations. pks.clusters.admin grants full management of every cluster and becomes vCenter role plus Supervisor access on the target. pks.clusters.manage grants management of clusters a user created and becomes a per-namespace permission on VKS rather than a global grant. Writing both onto the sheet now is what stops a platform admin from discovering on cutover day that they can log in but cannot create a cluster.

How TKGI and VKS authenticate users

On TKGI you ran the identity server. UAA was yours, it issued the tokens, and it delegated to LDAP or Active Directory through a chained flow where UAA checks its own store first and then the external directory. A user ran tkgi login, got a kubeconfig carrying a UAA OIDC token, and the cluster API validated that token and matched its groups against oidc: subjects in your RoleBindings. Every cluster trusted the same UAA, which is why group names were consistent and why lift and shift feels like it should work.

VKS splits that job in two. vCenter SSO authenticates at the platform layer, and it federates Active Directory as an identity source natively, so the directory you already trust plugs straight in. For teams that must authenticate against a non-AD provider, the Supervisor runs Pinniped and federates one external OIDC identity provider such as Okta or Entra ID into the guest clusters. A user runs kubectl vsphere login, gets a token from SSO or from Pinniped, and the guest cluster API matches its identity against sso: subjects in RBAC. Supervisor runs the Pinniped Supervisor and Concierge components, while each VKS guest cluster runs only the Concierge, which is the detail that makes external OIDC work end to end.

flowchart TB
  subgraph s1 [TKGI auth path]
    A1[User runs tkgi login] --> A2[UAA server]
    A2 --> A3[LDAP or AD directory]
    A2 --> A4[kubeconfig with OIDC token]
    A4 --> A5[Cluster API checks RBAC on oidc groups]
  end
  subgraph s2 [VKS auth path]
    B1[User runs kubectl vsphere login] --> B2[vCenter SSO]
    B2 --> B3[AD identity source or external OIDC via Pinniped]
    B2 --> B4[kubeconfig with SSO or Pinniped token]
    B4 --> B5[Guest API checks RBAC on sso subjects]
  end
Authentication moves from a UAA server you run to vCenter SSO with Pinniped for external OIDC.

One line on that diagram carries a design constraint that catches teams late. A Supervisor supports exactly one external OIDC identity provider. If half your teams live in Okta and half in a second OIDC tenant, you cannot wire both through Pinniped on the same Supervisor, and you plan around it now rather than discovering it during a pilot.

Identity and RBAC mapping sheet

Here is the artifact to keep from this part, the identity mapping sheet. One row per TKGI construct, resolved to its VKS equivalent, what actually changes, and the action you take. Build it once against your own estate and it drives every access step later in the series. The reference estate fills in like this.

TKGI constructWhat it doesVKS equivalentWhat changesMigration action
UAA serverissues OIDC tokensvCenter SSO plus Pinnipedidentity provider replacedreconfigure, do not migrate
UAA to LDAP delegationchains to ADvCenter SSO AD identity sourcenative SSO federationadd AD as SSO identity source
pks.clusters.admin scopefull TKGI cluster adminvCenter role plus Supervisor accessadmin lands at vSphere layergrant vCenter role, Supervisor edit
pks.clusters.manage scopemanage own clustersvSphere Namespace Owner or Editper namespace, not globalassign namespace permissions
ClusterRoleBinding to oidc:groupin-cluster adminClusterRoleBinding to sso:groupsubject prefix and namerewrite subjects, reapply
Roles and ClusterRolespermission definitionsidentical objectsnothing, portableapply as is
ServiceAccount and tokenworkload identityServiceAccount, new tokentokens regeneratedrecreate, re-mint tokens
external Vault via sidecarsecret injectionSecret Store Service, vault-injectoradd-on model, VKS 3.6 plusre-integrate, do not copy

Two rows carry most of the risk. The oidc:group to sso:group rewrite is where a lift and shift of RBAC fails, because the subject string changes and a binding that points at a name the API never issues is a binding that grants nothing. And the two UAA scope rows split apart on VKS, one to a vCenter role and one to a per-namespace permission, so a single TKGI admin identity often becomes two distinct grants at two layers. Miss that and your platform team can authenticate but cannot act.

Field note: On the staging cutover our CI deploy job started returning 403 Forbidden the next morning. We had restored the RoleBindings from a Velero backup and they read correct, subjects and all. The subject on the deploy binding was system:serviceaccount:ci:deployer, the ServiceAccount existed, but the token baked into the pipeline kubeconfig was the old TKGI cluster token, which the new VKS API had never signed. Ninety minutes of a blocked release train later, we re-minted a token on the VKS cluster, updated the pipeline secret and the job went green. RoleBindings restore. Trust does not.

Secrets assessment and what to reseal

Secrets tempt you to treat them as data, because they look like data in etcd. They are not. Supervisor and each VKS cluster encrypt secrets in etcd with a key handed down from vCenter at boot, and a TKGI cluster sealed its secrets under a different trust domain entirely. A secret exported from the old platform is ciphertext with no valid key on the new one, so the assessment here is not how to copy secrets, it is which secrets you re-create and from what source of truth. Start by listing them, excluding the auto-generated ServiceAccount tokens that the target will mint fresh.

kubectl get secrets -A –field-selector type!=kubernetes.io/service-account-token -o custom-columns="NS:.metadata.namespace,NAME:.metadata.name,TYPE:.type" # expected output, trimmed NS NAME TYPE default web-tls kubernetes.io/tls default harbor-pull kubernetes.io/dockerconfigjson app postgres-credentials Opaque app sh.helm.release.v1.web.v1 helm.sh/release.v1

Classify each type by where its source of truth lives, because that decides the action. A registry pull secret comes from your Harbor credentials, a TLS secret comes from a certificate authority you control, a database password comes from whatever vault or password store holds it, and a Helm release secret is just state that a redeploy regenerates. The reference estate sorts cleanly.

Secret typeExample in the estateMove it?Action
ServiceAccount tokenauto-generated tokensnotarget mints fresh
dockerconfigjsonharbor-pullrecreatere-create against VKS Harbor
TLS cert and keyweb-tls ingress certreissuereissue for new endpoints
Opaque app secretpostgres-credentialsresealreseal from source of truth
Helm release secretsh.helm.release.v1noredeploy the chart
external Vault referencevault path annotationre-integratewire Secret Store Service
Gotcha: Anything you cannot re-create from a source of truth is a secret you did not actually have under management. If a database password exists only inside a running TKGI cluster and nowhere else, the migration is the moment you find that out, and the fix is to record it in your vault before you touch the cluster, not to try to extract ciphertext that will not decrypt on the far side.

RBAC portability and where the mapping breaks

Common advice says RBAC is just YAML, so export it and kubectl apply it on the target. Half of that is true. Roles and ClusterRoles are pure permission definitions with no reference to any identity provider, and they apply on VKS unchanged. Bindings are the other half, and they break in three specific ways. First, the subject prefix changes from oidc: to the sso: form the guest cluster expects. Second, the group name itself can differ, because the string LDAP returned to UAA is not always the string vCenter SSO surfaces from the same Active Directory. Third, ServiceAccount tokens are regenerated, so any binding whose consumer carries a cached token fails until that token is re-minted, which is the failure the field note described. Count the objects by type and the shape of the work is obvious.

Identity objects to remap, reference estateThree clusters, dev staging prod, counts by object type0357040RoleBindings63ServiceAccounts12LDAP groups8ClusterRoleBindings
ServiceAccounts dominate the count, and every one of them needs a fresh token on VKS.

That chart also settles the harder design question, whether to lead with vCenter SSO or with Pinniped external OIDC. My verdict is to make vCenter SSO with an Active Directory identity source the primary path, because the reference estate already authenticates against AD and SSO federates it natively without standing up a second identity plane. Reserve Pinniped external OIDC for the case where a team genuinely must authenticate against a non-AD provider, and remember the one-provider limit means you cannot hedge by wiring both at once. The path to avoid is the tempting one, pointing Pinniped at your LDAP so it feels like UAA did, which recreates the old topology on new plumbing and buys you nothing while adding a component to operate.

For the component mechanics of how VKS presents identity, namespaces and SSO, the vSphere Kubernetes Service Complete Guide covers the target platform in depth, and this series links to it rather than re-teaching it. For where this identity work sits in the overall plan, the migration guide holds the full sequence.

What to do with your identity inventory on Monday

Pull the binding and ServiceAccount inventory from all three clusters first, because you cannot design an access model you have not enumerated. Then read the two UAA scopes and write down, per admin, which becomes a vCenter role and which becomes a namespace permission. Fill in the identity mapping sheet, one row per construct, and mark every oidc: subject that needs rewriting to sso: form so the rewrite is mechanical when you get to it. Run the secrets list and classify each entry to reseal, reissue, re-create or redeploy, and for any Opaque secret you cannot trace to a source of truth, record it in your vault today, before the cluster it lives in becomes a stopped machine. That sheet, built now against your own estate, is what turns the access cutover later in this series from a scramble into a checklist. Next part takes this further into multi-tenancy, mapping TKGI plans and their self-service model onto vSphere Namespaces, so the teams you just inventoried land in a structure they can actually operate. Open a blank sheet and run the first kubectl inventory against your prod cluster before you close the laptop.

TKGI to VKS Series · Part 9 of 26
« Previous: Part 8  |  Guide  |  Next: Part 10 »

References

About The Author


Discover more from Journal of Intelligent Infrastructure

Subscribe to get the latest posts sent to your email.

Leave a Reply

Your email address will not be published. Required fields are marked *

Architect’s Toolkit

About the Author

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

Discover more from Journal of Intelligent Infrastructure

Subscribe now to keep reading and get access to the full archive.

Continue reading