Two passes. That is how many times a CA certificate has to go through base64 before a VKS cluster will accept it, and skipping the second pass is the most common reason a trust fix looks applied and changes nothing at all. Objective 5.3 lives almost entirely in the gap between applied and effective.

Part 31 left the estate with clusters that build. VM classes are bound, the cluster plan renders, the content library serves releases. This Part starts one layer above that, with a cluster that is completely healthy and a deployment that will not come up, because its image sits in a Harbor project the nodes have never been told to trust. Nothing in the cluster status will tell you that. VKS is the product formerly named TKG Service, and the naming still shows: the secret these controllers look for is named after the cluster with a user-trusted-ca-secret suffix, and plenty of the surrounding resources still carry tkg. That is current VCF 9.0 output, not legacy output.

Three trust surfaces and where a pull actually breaks

An image pull on a VKS cluster crosses three surfaces, and each one is owned by a different object in a different cluster. Kubelet asks containerd, the container runtime on the node, to fetch the image. Containerd opens a TLS session to the registry and validates the server certificate against the node operating system trust store. If validation passes, containerd presents credentials taken from a Kubernetes secret of type dockerconfigjson that lives in the workload cluster, in the same namespace as the pod. Only after both of those does the registry decide whether that account may pull from the project.

Trust and authentication fail with different error text and are repaired in different clusters. That sentence is most of the objective. A trust failure names x509 and never reaches the registry logic at all. An authentication failure produces an HTTP status, usually 401 or 403, which means the TLS session already succeeded and your CA work is done. Reading the first line of the kubelet event tells you which cluster to open, and reading it wrong sends you to the wrong kubeconfig for the next hour.

On VKS the node trust store is not something you edit. It is populated during node bootstrap from a Kubernetes secret held in the vSphere Namespace on the Supervisor, and the Cluster object only consumes that secret if its topology names the data key through the trust variable. Two objects, one on each side, and both have to agree on a string you chose yourself.

Numbered repair procedure for a failing pull

Preflight is short. You need the registry CA certificate in PEM form, cluster edit rights in the vSphere Namespace on the Supervisor, and a maintenance window, because the correct fix rolls nodes. If you cannot roll nodes today, stop at step 4 and schedule the rest, because a half applied trust change is indistinguishable from no change.

Step 1, separate a trust failure from an authentication failure

Read the kubelet event before anything else. Two outputs, two different repairs, and they are visible in the first clause of the message.

An x509 line means go to the Supervisor. A 401 means stay in the workload cluster. If openssl reports an issuer you do not recognise, someone reissued the Harbor certificate and the nodes are still carrying the old root, which presents as x509 on a cluster that worked yesterday.

Step 2, build the double encoded secret in the vSphere Namespace

Kubernetes already base64 encodes the value of a data field. VKS expects the value of that field to itself be a base64 encoding of the PEM. Hence two passes. A single pass produces a secret that applies without complaint and decodes on the node into garbage.

Three strings matter and all three are yours to choose badly. Namespace must be the vSphere Namespace where the cluster lives. Secret name is free text but the controllers generate a default of cluster name plus user-trusted-ca-secret, and matching that convention saves you from the retirement fault described later. Data key, harbor-ca here, is the string the Cluster object will name in step 3.

Step 3, name the data key in the cluster trust variable

Applying a secret changes nothing on its own. Cluster topology has to reference it, and it references the data key, not the secret name. That indirection catches people who correctly assume the opposite.

Step 4, force the value onto the nodes

Here is the real failure, and it is the reason this Part exists. On VKS 3.3.0 through 3.3.2, updating certain Cluster variables or the Harbor Supervisor Service certificate does not trigger an automatic rolling update, and a related defect in the same window leaves containerd unrestarted after CA certificates are installed. Broadcom fixed both in VKS 3.3.3. Until you are on 3.3.3, the cluster reports a clean patch and the nodes never learn anything.

Step 5, repair authentication inside the workload cluster

A 401 needs nothing from the Supervisor. Create the credential where the pod runs and attach it to the identity the pod uses. Read the robot account password from an environment variable so it never lands in shell history or a manifest.

Verification, back out and a pull error lookup

Green looks like a pod that goes Running without a restart, on a node whose age is younger than your trust change. Delete the stuck pods rather than waiting out the backoff timer, which grows to five minutes and will make a working fix look broken.

That last block is worth memorising. Broadcom documents the same DataSecretGenerationFailed condition as a known issue in the 3.3.0 through 3.3.2 window, where additional trusted CAs configured against a TanzuKubernetesCluster are lost once that object is retired and replaced by a Cluster resource. Bootstrap then fails on every new machine, so the symptom appears at the next scale up or upgrade rather than at the moment of breakage. If you inherit an estate with retired TanzuKubernetesCluster objects, check for the secret before you scale anything.

flowchart TD
  A[Pod stuck on ImagePullBackOff] --> B{Message names x509 or a 401}
  B -->|401| C[Create dockerconfigjson secret in the workload cluster]
  C --> D[Attach it to the pod service account]
  B -->|x509| E{CA secret exists in the vSphere Namespace}
  E -->|No| F[Encode the PEM twice and apply on the Supervisor]
  E -->|Yes| G{Trust variable names the data key}
  G -->|No| H[Patch topology variables and plan a rollout]
  G -->|Yes| I{Machine age newer than the trust change}
  I -->|No| J[Force a MachineDeployment rollout, 3.3.2 will not do it]
  I -->|Yes| K[Fault is on the registry, check project and robot role]

Keep the table below. It is the artifact from this Part worth returning to, because every row pairs a literal string you will actually see with the surface that owns the repair.

Now for the advice this Part contradicts. Search almost any forum for the x509 string and the top answer will tell you to connect to the node, drop the certificate into the operating system trust directory, refresh the trust store and restart containerd. That works. It works for about as long as the node lives, and it is the wrong answer here for two reasons. Every replacement machine comes up untrusted, so the fault reappears at the next upgrade, autoscale event or repair, usually to somebody else. Worse, the manual fix hides which of the two defects you were hitting, so you never learn whether the estate needs the VKS upgrade. Declarative repair through the trust variable is slower on the day and it is the only one that survives a node rollout, which is exactly the property the exam is testing for. Reserve node level intervention for proving a hypothesis, never for fixing production.

Exam focus, objective 5.3

Objective checkpoint

Trust repair order I use before escalating

My recommendation is narrow and I hold it firmly. Read the error, pick the cluster, then work in one direction only: secret, variable, rollout, verify on a new node. Never repair a node by hand, and never verify a trust fix by looking at the Cluster object, because the Cluster object will happily report a change that never left the Supervisor. If your estate is anywhere in the VKS 3.3.0 to 3.3.2 window, schedule the move to 3.3.3 before you spend another afternoon like the one I described, because two separate defects in that window both present as a fix that did not work.

A clean end state on this objective is small enough to check in a minute: pods Running with zero restarts, a machine age younger than the trust change, the trust variable readable from jsonpath, the CA secret still present in the vSphere Namespace, and an image pull secret attached to the service account only where the registry demands credentials. If all five are true you are done, and if any one is false you already know which surface to open.

Tonight, in your own lab, break it deliberately. Apply the CA secret with a single base64 pass instead of two, watch the pull fail with exactly the same x509 line as a missing secret, and note how little the error told you. That is the discrimination objective 5.3 asks for. Work through the rest of the series from the guide, and if you arrived here from a TKGI estate, the migration path is covered separately in the TKGI to VKS guide.

References

VMware Cloud Foundation VKS Administrator exam guide, 3V0-24.25
VMware vSphere Kubernetes Service 3.3.3 and earlier release notes, fixed and known issues
Configuring a VKS cluster with a self signed container registry, William Lam

About The Author

Dr. Pranay Jha

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

See author's posts