A candidate on a coaching call last month asked me why his Prometheus package kept failing with an authentication error while his application pods were pulling images from that same Harbor without a complaint. He had created one secret, put it in the namespace where his application ran, and assumed that covered the cluster. It covered exactly one of four separate authentication and trust paths, and objective 4.8 expects you to keep all four apart in your head.

Two package mechanisms living in one release

Start with the thing that will waste your study time if nobody warns you. Broadcom now documents two ways to get a standard package onto a VKS cluster, and only one of them exists on the estate the exam is built against. VKS 3.5 introduced an add-on management layer with AddonRepository and AddonRepositoryInstall objects, an embedded repository shipped on every cluster, and an override annotation for pointing at your own copy. VCF 9.0 shipped VKS 3.3, which has none of those CRDs. If you search for how to install a package today you will land on the 3.5 pages first, follow them, and get a CRD not found error on a perfectly healthy cluster.

Popular study advice says read the newest documentation because it is the most correct. For this objective that advice is wrong, and it is wrong in a way that costs you an item. Exam content is based on VCF 9.0, so the mechanism you need in your fingers is the Carvel one: a PackageRepository object in tkg-system, PackageInstall objects reconciled by kapp-controller, and secrets managed by the secretgen controller. Learn the 3.5 add-on model afterwards, as an operator, not as a candidate.

That second preflight is worth thirty seconds on any cluster you are handed. It answers the question of which documentation set applies before you type anything destructive, and it is the single fastest way to tell a 3.3 estate from a 3.5 one without asking anybody.

Standard package repository install and package discovery

Namespace choice is the whole procedure. Adding a PackageRepository to tkg-system makes every Package and PackageMetadata object it carries visible from every namespace on the cluster, without copies. Add the same repository to a per application namespace and those definitions are scoped to that namespace alone, so a second team creating a PackageInstall elsewhere sees nothing available and files a ticket. I have watched that happen twice, both times because somebody followed a blog that used a namespace called packages.

That last check is the one I would run in front of an examiner. Fourteen package metadata rows plus a header, read from a namespace where nothing was ever installed, is the observable proof that tkg-system was the right home. Run the same command after adding the repository to a private namespace and you get one line, the header, and no explanation.

Registry secrets and their export boundary

Here is where the candidate on that call went wrong, and where most experienced admins go wrong, because Kubernetes has trained us that an image pull secret is a namespace scoped thing you attach to a pod. That is true for your workloads. It is not true for packages. kapp-controller does the fetching for a PackageInstall, it runs in tkg-system, and it reads credentials through the secretgen controller. A secret sitting in your application namespace is invisible to it.

Two objects are needed, not one. A dockerconfigjson secret in tkg-system, and a SecretExport in tkg-system whose name matches the secret exactly and whose toNamespaces list covers where the package will land. Miss the SecretExport and the secret exists, kubectl shows it happily, and the fetch still fails. That gap between a resource existing and a resource being usable is exactly the shape of trap this objective likes.

Read that error line carefully, because it is the fork in the road. UNAUTHORIZED means credentials, so you are in secret and SecretExport territory. An x509 error mentioning an unknown authority means certificate trust, which no secret in tkg-system will ever fix. Candidates who conflate those two spend twenty minutes recreating credentials for a problem that lives on the node.

Private registry trust baked into a cluster spec

Node level trust is provisioning time work. Every worker and control plane node in a VKS cluster runs containerd, and containerd will refuse a registry whose certificate chain it does not recognise, no matter what credentials you hand it. On the Cluster v1beta1 API the mechanism is the trust variable, populated with the name of a secret that carries the registry certificate authority. Change it on a running cluster and the machines roll, which is a rolling update in the Part 21 sense, with all the disruption budget considerations that came with it.

One exemption is worth memorising. When Harbor runs as a Supervisor Service, as it does on this lab estate from Part 19, the platform already arranges trust for it, so the additional certificate step is only needed for a registry that is not the platform registry, or for a certificate that no public authority signed. Broadcom states that plainly in the add-on repository procedure, and a scenario item that mentions the Harbor Supervisor Service is very likely testing whether you know to skip a step rather than add one.

Name discipline saves you here. Secret name follows the cluster name, the key inside the secret is what the variable references, and the two are not the same string. I now write the key first, paste it into the variable, and only then create the secret, purely because I got this backwards once and spent a machine roll finding out.

flowchart TD
  A[Image pull or package fetch fails] --> B{Which component is pulling}
  B -->|Application pod| C[Check imagePullSecrets on pod or service account]
  B -->|kapp controller| D[Check secret in tkg system plus SecretExport]
  B -->|Node containerd| E[Check trust variable and trusted CA secret]
  B -->|vSphere Pod on Supervisor| F[Check Supervisor container registry entry]
  C --> G{Error text}
  D --> G
  E --> G
  F --> G
  G -->|UNAUTHORIZED| H[Credential problem, rebuild secret and export]
  G -->|x509 unknown authority| I[Trust problem, fix CA and roll the cluster]

Registry secret placement lookup

This table is the artifact from this Part worth pinning above your desk. Four consumers, four homes, four different failures. Nothing else in objective 4.8 generates as many wrong answers as putting the right object in the wrong place.

Air-gapped relocation with imgpkg

Part 9 built the local content library for the air-gapped Kubernetes release exercise. Packages need their own relocation, because a content library moves node images and has nothing to say about Carvel bundles. imgpkg copy handles both halves, a pull to a tarball on a machine that has internet access, and a push into your registry from a machine that does not.

Budget the time honestly. On a 1 Gb link into the lab, the four stages below took 63 minutes end to end, and the tarball landed at 11 GB for 138 images. Nobody in a change window expects a package repository move to take an hour, so tell them before you start rather than after.

Failures I hit, with their error lines

Exam focus for objective 4.8

Repository and registry choices I would ship

War story, and it cost me ninety minutes I did not have. A customer cluster could not install cert-manager from their own Harbor. Credentials were right, I could docker login by hand from the jump host with the same account, and the CA was already trusted because Harbor was the platform registry. Every check passed and the fetch kept failing. What I had missed was that the Harbor project holding the relocated bundle had been created as private and the service account we were using had pull rights on a different project entirely. My login worked because my human account was a project admin somewhere else. Ninety minutes of Kubernetes debugging for a permission set two clicks deep in a registry UI. Since then the first thing I do on a package failure is pull the exact same reference with imgpkg, from a shell, using the service account credentials rather than mine.

Verdict for this estate. Put the standard package repository in tkg-system and nowhere else, and treat any suggestion of a per team repository namespace as a mistake to be argued out of the design. Create the registry secret and the SecretExport together in one manifest so the pair cannot drift, and export to all namespaces rather than a list, because a list is a future outage every time somebody adds a namespace. Relocate the bundle into Harbor even when you are not air-gapped, because the pull is faster, the versions stop moving under you, and you get one place to audit what your clusters are actually running. Avoid the pattern of trusting a private registry by editing nodes after provisioning; if the trust is not in the cluster spec, it does not survive a machine roll.

Tonight, in your own lab, do this. Create a registry secret in tkg-system without a SecretExport, add a repository pointing at a private Harbor project, and read the usefulErrorMessage field. Then add the export and watch a fetch secret appear. That single loop teaches the boundary better than any diagram. Objective 4.8 also leans on the Harbor work from the Harbor series and on the cluster mechanics in the VKS series, neither of which this Part repeats. Part 24 moves to snapshots inside VKS clusters. Full path is indexed on the VCAP-VKS exam guide pillar.

References

• Using Private Registries with VKS Clusters, VCF 9.0, Broadcom TechDocs
• Update an Add-on Repository Install, VCF 9.0, Broadcom TechDocs
• Installing Standard Packages on VKS Clusters, VCF 9.0, Broadcom TechDocs
• VMware Cloud Foundation VKS Administrator Exam Guide, 3V0-24.25, Broadcom

About The Author

Dr. Pranay Jha

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

See author's posts