, ,

Harbor Supervisor Service and external-dns Add-on Lifecycle (VCAP-VKS Exam Series, Part 19)

Objective 4.4 is a lifecycle objective, not an install objective. Here is the Harbor Supervisor Service procedure end to end on VCF 9.0, the external-dns package that does not install the same way, and the deactivate then uninstall then delete order that vCenter enforces.

VCAP-VKS Exam Series · Part 19 of 34

A Harbor install sat in Configuring for forty one minutes on a Supervisor I had activated the week before, and vCenter offered no reason at all. Three underscores in a storage class name, pasted straight out of a vSphere Client field into a data values file, were the entire story. That is the shape of objective 4.4 in practice: almost nothing goes wrong in the install wizard, and almost everything goes wrong in the Supervisor objects the wizard quietly depends on.

Who this is for: A candidate working through Section 4 who has already activated a Supervisor and carved namespaces, and now needs to add platform services on top. This Part covers Objective 4.4, published wording Install, uninstall, and manage Supervisor add-on services (e.g., Harbor, external-dns). Terms defined on first use here: a Supervisor Service is a vSphere certified Kubernetes operator that a vSphere administrator registers with vCenter and installs onto a Supervisor; a core Supervisor Service has its operator pre-installed during Supervisor activation and cannot be removed; a vSphere Pod is a container that runs directly on ESX with its own tiny virtual machine boundary; Contour is an Envoy based ingress controller shipped as its own Supervisor Service; external-dns is a controller that writes DNS records for Services and Ingresses; Trivy is the image scanner bundled with Harbor. VKS is the product formerly named TKG Service or TKGS, so expect tkg to keep appearing in paths, custom resource names and API groups throughout this Part.
Key takeaways: Objective 4.4 rewards lifecycle order, not wizard clicks. A Supervisor Service may have many versions registered with vCenter but exactly one installed per Supervisor at a time, and removing a service is a four step ordered sequence, deactivate, uninstall, delete version, delete service, that vCenter will block if you attempt it out of order. Harbor deploys as vSphere Pods into an automatically created service namespace, so it inherits the NSX requirement from whatever networking mode you picked at Supervisor activation and will never land on a vSphere Distributed Switch Supervisor. Both example services in the objective wording install by completely different mechanisms, and external-dns is not a Supervisor Service on VCF 9.0 at all. Headline preflight command is kubectl get storageclass before you paste a single line of the Harbor data values file.
# Versions this Part was tested against VCF 9.0, vCenter 9.0 (record your own appliance build from the VAMI banner) vSphere Supervisor 9.0, NSX VPC networking enabled VKS 3.3.1 supplying Kubernetes releases Harbor Supervisor Service 2.11.2 or later (load balancer exposure requires 2.11.2+) Contour Supervisor Service, installed before Harbor on the same Supervisor kubectl v1.32.x with the kubectl-vsphere plugin shipped by this vCenter

Add-on delivery models on a Supervisor

Read the objective wording again and notice that it groups Harbor and external-dns as if they were siblings. On VCF 9.0 they are not. Harbor is a Supervisor Service, downloaded as a pair of YAML files, registered against vCenter and installed onto a named Supervisor by a vSphere administrator holding the Manage Supervisor Services privilege. External-dns ships in the VKS standard package catalogue and is installed by a cluster owner, inside a workload cluster, against that cluster API. Nothing about the second workflow touches Supervisor Management in the vSphere Client. Candidates who study only the Harbor path answer half of this objective.

A third category matters as much and gets studied least. Core Supervisor Services have their operators installed automatically when you activate the Supervisor, and Broadcom names VKS itself and the Velero vSphere Operator as examples. You can add and upgrade their versions without touching the Supervisor version, which is why a VKS release bump does not require a Supervisor upgrade, but you cannot remove their operators from the platform. Any exam item that asks you to uninstall VKS from a Supervisor is asking a question with no correct answer, so read those stems carefully for what they really mean, which is deactivating or uninstalling a service version.

CriterionSupervisor ServiceCore Supervisor ServiceVKS standard package
Blueprint exampleHarbor, Contour, Argo CDVKS, Velero vSphere Operatorexternal-dns, Contour in cluster, Istio
Who installs itvSphere admin, Supervisor ManagementvSphere admin, operator already presentCluster owner, against the VKS cluster API
Where the workload landsvSphere Pods in an auto created service namespaceSupervisor control planeRegular pods inside the workload cluster
Supervisor networking requiredNSX or NSX VPC, because vSphere Pods are involvedWhatever the Supervisor was activated withAny mode, vDS included
Versions installable at onceOne per SupervisorOne per SupervisorOne per package namespace
Fully removableYes, four ordered stepsNo, the operator stays on the platformYes, delete the package resource

Preflight for the Harbor Supervisor Service

Last Part we deployed workloads two ways on this estate, as vSphere Pods and through VM Service. Harbor is the same vSphere Pod machinery wearing a product name, which is exactly why its prerequisites are Supervisor prerequisites rather than Harbor prerequisites. Four checks decide whether the install can succeed, and all four are visible from a kubectl session against the Supervisor before you open the wizard. I run them in this order every time because each one has caught me at least once.

$ kubectl vsphere login –server=sup01.lab.local –insecure-skip-tls-verify –vsphere-username administrator@vsphere.local # password read from the KUBECTL_VSPHERE_PASSWORD environment variable, never typed inline $ kubectl get storageclass NAME PROVISIONER RECLAIMPOLICY ALLOWVOLUMEEXPANSION AGE vks-default csi.vsphere.vmware.com Delete true 21d vks-high-performance csi.vsphere.vmware.com Delete true 21d $ kubectl get svc -n projectcontour NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE envoy LoadBalancer 10.96.144.11 10.20.44.51 80:31427/TCP,443:32188/TCP 6d $ kubectl get nodes -o wide | head -4 NAME STATUS ROLES AGE VERSION 4230a1c8f9b2… Ready control-plane 21d v1.32.3+vmware.wcp.1 esx-a-01.lab.local Ready agent 21d v1.32.3-sph-9f2d4a1

Three signals matter in that output. Storage class names are already lower case and hyphenated, which is how Kubernetes renders a storage policy called VKS High Performance; you must write them that way in the data values file, not the way vCenter displays them. Envoy holds a real external address, so Contour is live and Harbor can hang an HTTPProxy off it. And ESX hosts appear as agent nodes, which only happens on an NSX backed Supervisor, so vSphere Pods can schedule. On a Supervisor activated with vSphere Distributed Switch that last line is simply absent, and the Harbor install will create its namespace, report Configuring, and never finish. Nothing in the vSphere Client will tell you why, which is the Supervisor decision from Part 6 reaching forward to bite you weeks later.

Harbor installation procedure

Step one, download two files. From the vSphere Supervisor Services page on the Broadcom support portal you get harbor-service-x.xx.x.yml, which is the service definition telling the Supervisor how the service is structured and which versions it is compatible with, and harbor-data-values-x.xx.x.yml, which carries your system level parameters. Broadcom also publishes the wider catalogue of service YAML files at the vSphere TMM Supervisor Services site, which is worth bookmarking because it is where new services appear first.

Step two, register the service. In Supervisor Management, Services, choose Add New Service and upload the service definition file. Registration is a vCenter level action and installs nothing; the service lands in Active state and now appears as a card. Step three, edit the data values file. Step four, on the Harbor card choose Actions, Manage Service, pick the version and the target Supervisor, paste the edited data values into YAML Service Config and finish. Only one version can be installed per Supervisor, so choosing a version here is also choosing to replace whatever is there.

# harbor-data-values-x.xx.x.yml, the fields that actually decide success hostname: harbor.lab.local https: 443 # leave these labels exactly as shipped or VKS integration silently stops working tlsCertificate: tlsSecretLabels: {"managed-by": "vmware-vRegistry"} harborAdminPassword: ${HARBOR_ADMIN_PASSWORD} # exported in the shell, not committed secretKey: ${HARBOR_SECRET_KEY} # must be exactly 16 characters database: password: ${HARBOR_DB_PASSWORD} core: secret: ${HARBOR_CORE_SECRET} xsrfKey: ${HARBOR_XSRF_KEY} # must be exactly 32 characters jobservice: replicas: 1 secret: ${HARBOR_JOB_SECRET} persistence: persistentVolumeClaim: registry: storageClass: "vks-high-performance" accessMode: ReadWriteOnce size: 10Gi jobservice: storageClass: "vks-default" size: 1Gi database: storageClass: "vks-high-performance" size: 1Gi redis: storageClass: "vks-default" size: 1Gi trivy: storageClass: "vks-default" size: 5Gi network: ipFamilies: ["IPv4"] # IPv6 is not supported, do not change enableContourHttpProxy: true # flip to false if you use an NGINX load balancer enableNginxLoadBalancer: false # and flip this one to true at the same time

Five fields in that file cause most of the pain. Storage class names must be transformed from the policy name by lower casing every character and replacing underscores and spaces with hyphens, and nothing validates this until a PersistentVolumeClaim goes Pending. SecretKey is exactly sixteen characters and xsrfKey is exactly thirty two, and a length mismatch fails the pods rather than the wizard. IpFamilies stays IPv4 because IPv6 is unsupported. TlsSecretLabels must stay as shipped, because that label is how the vRegistry integration finds Harbor and wires trust into VKS clusters; teams that replace it with something tidier get a Harbor that works perfectly from a laptop and cannot be pulled from by any cluster. And the two exposure switches are a pair, not one setting, so an admin who sets enableNginxLoadBalancer true without setting enableContourHttpProxy false gets both paths half configured. Total baseline storage across those five claims is eighteen gibibytes before a single image is pushed, which is worth knowing when a namespace quota is tight.

flowchart TD
  A[Download service and data values YAML] --> B[Add service to vCenter]
  B --> C[Version state Active]
  C --> D[Install one version on a Supervisor]
  D --> E[Service vSphere Namespace created automatically]
  E --> F[Upgrade by adding a newer version then installing it]
  F --> D
  C --> G[Deactivate version]
  D --> H[Uninstall version from every Supervisor]
  G --> H
  H --> I[Delete version from vCenter]
  I --> J[Delete entire service]
Supervisor Service lifecycle. Registration and installation are separate states, and removal only runs right to left through deactivate, uninstall, delete version, delete service.

External-dns inside a VKS cluster

Popular study advice says to treat the two named services in objective 4.4 as one workflow. That advice is wrong, and the exam knows it. On VCF 9.0 external-dns is documented in the VKS standard package reference, installed into a workload cluster, and validated against three provider families: AWS Route 53, Azure DNS, and RFC2136 compliant servers such as BIND. Our lab estate uses the RFC2136 path against an internal BIND instance, which is also the only one of the three you can practise without a public cloud account.

# run against the VKS cluster, not the Supervisor $ kubectl config use-context vks-app-01 $ kubectl create namespace external-dns-ns namespace/external-dns-ns created # generate the defaults rather than hand writing them, then edit $ tanzu package available get external-dns.kubernetes.vmware.com/<VERSION> –default-values-file-output external-dns-data-values.yaml # minimal RFC2136 values for the lab BIND server $ cat external-dns-data-values.yaml deployment: args: – –source=service – –source=ingress – –domain-filter=apps.lab.local – –provider=rfc2136 – –rfc2136-host=10.20.10.53 – –rfc2136-port=53 – –rfc2136-zone=apps.lab.local – –rfc2136-tsig-secret-alg=hmac-sha256 – –rfc2136-tsig-keyname=externaldns-key – –txt-owner-id=vks-app-01 $ tanzu package install external-dns -p external-dns.kubernetes.vmware.com –version <VERSION> –namespace external-dns-ns –values-file external-dns-data-values.yaml

Pin the version from your own package catalogue rather than copying one, because standard package versions track the Kubernetes release the cluster is running and a stale pin is the fastest way to a reconcile failure. Two operational notes are worth carrying into production. TXT owner identity must be unique per cluster or two clusters will fight over the same records and flap them, and the TSIG key is a secret that belongs in a Kubernetes Secret rather than in the values file you keep in git. For the wider DNS and ingress design that sits above this, Part 13 already covered the ingress and egress choices this controller writes records for.

Verification, uninstall order and failure signatures

Green looks like a service namespace you did not create, full of running vSphere Pods, plus a resolvable FQDN. Installation creates that namespace automatically, one per Supervisor Service, and you manage the service resources from it.

$ kubectl get ns | grep harbor svc-harbor-domain-c8 Active 14m $ kubectl get pods -n svc-harbor-domain-c8 NAME READY STATUS RESTARTS AGE harbor-core-7c9f4b8d6-x2rlw 1/1 Running 0 12m harbor-database-0 1/1 Running 0 13m harbor-jobservice-0 1/1 Running 0 12m harbor-portal-6d8b9c5f4-9tqvz 1/1 Running 0 12m harbor-redis-0 1/1 Running 0 13m harbor-registry-0 2/2 Running 0 12m harbor-trivy-0 1/1 Running 0 12m $ kubectl get pvc -n svc-harbor-domain-c8 NAME STATUS CAPACITY STORAGECLASS AGE harbor-database Bound 1Gi vks-high-performance 13m harbor-jobservice Bound 1Gi vks-default 13m harbor-redis Bound 1Gi vks-default 13m harbor-registry Bound 10Gi vks-high-performance 13m harbor-trivy Bound 5Gi vks-default 13m $ curl -sI https://harbor.lab.local/api/v2.0/systeminfo | head -1 HTTP/2 200

Here is what my forty one minute failure actually looked like. Pods sat Pending, no events surfaced in the vSphere Client, and only a describe against the claim named the cause.

$ kubectl describe pvc harbor-registry -n svc-harbor-domain-c8 | tail -5 Events: Type Reason Age From Message —- —— —- —- ——- Warning ProvisioningFailed 9s persistentvolume-controller storageclass.storage.k8s.io "VKS_High_Performance" not found # fix: normalise the name, then reinstall the service version $ kubectl get storageclass -o custom-columns=NAME:.metadata.name –no-headers vks-default vks-high-performance

Backing out is where candidates lose marks, because uninstall and delete are different operations with different blast radii and vCenter enforces an order between them. Uninstalling a version from a Supervisor removes every service resource including that service namespace, yet application instances already running in Kubernetes workloads keep running. Upgrading is not additive either: any resource specified by the old version and absent from the new one is deleted, so a pod A in version one becomes a pod B in version two and pod A goes away. Keep this table next to you, because it is the artifact this Part exists to produce.

OrderOperationWhat it changesWhat survives
1Add service to vCenterRegisters metadata and compatibility, state becomes ActiveNo Supervisor is touched
2Add a new versionNew version registered and ActiveInstalled versions keep running
3Install on a SupervisorApplies the YAML, creates a service vSphere Namespace and its vSphere PodsOnly one version per Supervisor at a time
4UpgradeApplies the newer YAML, deletes resources the new version does not declareRunning workloads are not impacted
5Deactivate version or serviceBlocks new installs of that version anywhereExisting installs keep running
6Uninstall version from a SupervisorRemoves all service resources including the service namespaceApplication instances in workloads keep running
7Delete version, then delete serviceRemoves registration from vCenter entirelyBlocked unless deactivated and uninstalled first
Harbor lifecycle operations, wall clock minutes Measured on the reference estate, VCF 9.0 with NSX VPC, three zone Supervisor, vSAN backed storage policies. Register service YAML 2 Install, bad storage class 41 Install, corrected values 14 Upgrade to next version 9 Uninstall from Supervisor 6 A failed install costs roughly three times a good one, because nothing times out and you decide when to stop waiting.
SymptomError you actually seeCause and fix
Service stuck in Configuring, pods Pendingstorageclass.storage.k8s.io "VKS_High_Performance" not foundPolicy name pasted verbatim. Lower case it and replace underscores and spaces with hyphens, then reinstall the version.
Namespace created, zero pods, no eventsSilence, and no agent nodes in kubectl get nodesSupervisor was activated on vSphere Distributed Switch, so vSphere Pods cannot run. Rebuild the Supervisor on NSX or host the registry elsewhere.
Add New Service greyed outPermission to perform this operation was deniedAccount lacks the Manage Supervisor Services privilege on this vCenter. Grant it at the vCenter root object.
Harbor UI unreachable after a clean installcurl: (6) Could not resolve host: harbor.lab.localDNS record never created. Map the Harbor FQDN to the Envoy ingress address, or to the NGINX external IP if you flipped both exposure switches.
VKS clusters cannot pull, laptop canx509: certificate signed by unknown authoritytlsSecretLabels was edited, so the vRegistry integration never published trust. Restore the shipped labels and add the CA per Part 10.
Pods CrashLoopBackOff right after installinvalid secret key lengthsecretKey is not 16 characters or xsrfKey is not 32. Correct both and reinstall the version.
Delete Service Version refusedVersion is installed on one or more SupervisorsDeactivate the version, uninstall it from every Supervisor, then delete it. Order is enforced.

Exam focus for objective 4.4

Exam focus, Objective 4.4: Published wording is Install, uninstall, and manage Supervisor add-on services (e.g., Harbor, external-dns), and the operative verbs are install, uninstall and manage in equal weight. Expect this objective to surface as build list items asking you to sequence a lifecycle operation, matching items pairing a state such as Deactivated with what it permits, point and click items on the Supervisor Management Services view, and multiple selection items on prerequisites. Practise saying out loud what each of the seven operations destroys and what it leaves running, because that distinction is what the item writers can test unambiguously. Trap that catches experienced admins: assuming registration and installation are one action, and assuming a service can be deleted while it is still installed. A second trap sits in the objective wording itself, where two examples are named that install by entirely different mechanisms, so an item can legitimately ask which one never appears under Supervisor Management at all.

Objective checkpoint

Question 1. You must remove a Supervisor Service completely from a vCenter where it is installed on two Supervisors. Put the operations in order.
Answer: deactivate every version, uninstall each version from both Supervisors, delete each service version, then delete the service. Reasoning: vCenter refuses a delete while a version is Active or still installed, so the order is enforced rather than advisory.

Question 2. A Harbor Supervisor Service install creates its vSphere Namespace but starts no pods, and the Supervisor reports healthy. What single Supervisor level decision explains this?
Answer: the Supervisor was activated with vSphere Distributed Switch networking rather than NSX. Reasoning: Harbor runs as vSphere Pods, and vSphere Pods require an NSX backed Supervisor, so the constraint was fixed at activation time.

Question 3. Of the two services named in objective 4.4, which is not installed through Supervisor Management on VCF 9.0, and where does it install instead?
Answer: external-dns, which installs as a VKS standard package inside a workload cluster. Reasoning: Broadcom documents it in the standard package catalogue for VKS clusters, not in the Supervisor Services catalogue.

Add-on service call for this estate

My recommendation is unfashionable: install Contour first and accept the default Contour ingress path for Harbor, even on an estate that already runs NSX Advanced Load Balancer for everything else. I did the opposite on my first build, reasoning that one load balancer for the whole platform was tidier, and spent an afternoon discovering that the NGINX path needs two data values flipped together rather than one, that the resulting external address needs its own DNS record anyway, and that the Harbor documentation assumes Contour in every subsequent troubleshooting step. Reverting to Contour took eleven minutes. Tidiness cost me roughly four hours and bought nothing, because Harbor is the only consumer of that ingress path.

A clean result on this estate looks like five things. One service namespace per installed Supervisor Service that you never created by hand. Seven Harbor vSphere Pods Running and five claims Bound against normalised storage class names. An FQDN that resolves to the Envoy address and returns HTTP 200 on the system info endpoint. Shipped tlsSecretLabels untouched, proven by a VKS cluster pulling an image without an insecure flag. And external-dns reconciled inside the workload cluster with a unique TXT owner identity, writing records into the zone you delegated to it.

Tonight, do one thing in your own lab: install a Supervisor Service, then try to delete it in the wrong order and read the exact refusal vCenter gives you. That refusal message is worth more than any flashcard on this objective. If Harbor itself is new ground, the Harbor series covers projects, robot accounts and replication properly, and the VKS series walks the product underneath this exam without the exam framing. Part 20 moves from platform services to the clusters themselves, provisioning and scaling VKS clusters with kubectl and the VCF CLI.

VCAP-VKS Exam Series · Part 19 of 34
« Previous: Part 18  |  Guide  |  Next: Part 20 »

References

About The Author


Discover more from Journal of Intelligent Infrastructure

Subscribe to get the latest posts sent to your email.

Leave a Reply

Your email address will not be published. Required fields are marked *

Architect’s Toolkit

About the Author

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

Discover more from Journal of Intelligent Infrastructure

Subscribe now to keep reading and get access to the full archive.

Continue reading