, ,

VKS Workload Deployment Models, Ingress Controllers and Registries (VCAP-VKS Exam Series, Part 27)

Objective 4.12 asks you to build workload deployment models across pods, VMs, ingress controllers and private registries on a VKS cluster. Here is the runbook, the Avi VIP arithmetic that decides the answer, and why the official Contour tutorial fails on a current cluster.

VCAP-VKS Exam Series · Part 27 of 34
Key takeaways: Objective 4.12 is not a manifest typing exercise. It asks you to pick an exposure model and defend it, and every one of those models spends a resource that was fixed when the Supervisor was enabled. A Service of type LoadBalancer consumes one virtual IP from the Avi pool per service; an ingress controller consumes one virtual IP for the whole cluster and then routes by host and path behind it. On a lab pool of 30 addresses that difference is the whole exam question. Headline commands: kubectl get svc -n projectcontour envoy to see the shared virtual IP, and kubectl describe pod to read the real reason an image did not pull. Broadcom still publishes a Contour tutorial built on Contour release 1.9, PodSecurityPolicy and the networking.k8s.io/v1beta1 Ingress API, and applying it verbatim on a current VKS cluster fails on all three counts.

A candidate on a coaching call last month asked me a question I have now heard four times: if a Service of type LoadBalancer already gives an application a public address, why would an exam objective bother mentioning ingress controllers at all. He had built every demo he owned on LoadBalancer services and had never run out of anything. That is the honest reason the question keeps coming up. In a lab with four applications, nothing breaks. In an estate with forty, the platform runs out of addresses, and the objective is written by people who have watched that happen.

Who this is for: A candidate who deployed Supervisor side workloads in Part 18 and wired storage in Part 26, and now has to put an application in front of users from inside a VKS workload cluster. This Part covers Objective 4.12, published wording Create workload deployment models including pods, VMs, ingress controllers and private registries [VERIFY the exact printed clause against the guide PDF linked in References; the numbering and scope are firm, my transcription of the sentence is from a secondary listing]. Terms defined on first use: VKS is vSphere Kubernetes Service, the product formerly named TKG Service or TKGS, and on VCF 9.0 that rename is unfinished, so tkg still appears in namespaces, package names and documentation URLs; a Service of type LoadBalancer is a Kubernetes object that asks the platform for an external address; an Ingress is a Kubernetes object describing HTTP or HTTPS routing rules; an ingress controller is the component that actually implements those rules, and Kubernetes ships none by default; Contour is the ingress controller Broadcom documents for VKS, with Envoy as its data plane proxy; Avi is NSX Advanced Load Balancer, the component that hands out virtual IPs, abbreviated VIP; VM Service is the Supervisor capability that lets a namespace user create a virtual machine through the Kubernetes API; imagePullSecrets is the pod field naming credentials for a private registry.

Four deployment models on one VKS estate

Objective 4.12 names four things in one sentence, and candidates read that as four unrelated topics. They are not. Pods, virtual machines, ingress controllers and private registries are the four decisions you make about a single application: where the process runs, how traffic reaches it, and where its image comes from. Broadcom documents deploying workloads to a VKS cluster using pods, services, persistent volumes and higher level resources such as deployments and replica sets, which is ordinary Kubernetes. What is not ordinary is the second half of every one of those sentences, because on VKS the platform answers are supplied by the Supervisor.

Trace it once and it stays traced. A Service of type LoadBalancer inside a workload cluster does not create anything by itself. It asks the cloud provider running in that cluster, which asks the Supervisor, which asks whichever load balancer you selected when the Supervisor was enabled. If that was Avi, an address comes out of a VIP range configured at Supervisor enablement, and no amount of editing the service manifest changes how many addresses are in that range. Same shape for images: a pod in a workload cluster pulls through containerd on a worker node, and whether that pull succeeds depends on trust material baked into the cluster spec, not on anything you can patch into the deployment afterwards. Both of those chains are covered in Part 13 and Part 23, so I will not restate the mechanics, only the consequence.

Virtual machines belong in this objective for a reason that catches container people rather than vSphere people. A VM Service machine lives on the Supervisor, in a vSphere Namespace, not inside the workload cluster. So an application composed of a containerised web tier in a VKS cluster and a licensed database on a VM Service machine has two different exposure stories, two different address sources, and no shared Kubernetes Service object to join them. Candidates lose points by drawing one ingress in front of both.

flowchart TD
  A[Workload needs to be reachable] --> B{Protocol is HTTP or HTTPS}
  B -->|no| C[Service type LoadBalancer, one Avi VIP per service]
  B -->|yes| D{More than two routes expected in this cluster}
  D -->|no| C
  D -->|yes| E[Ingress controller, one shared Avi VIP for the cluster]
  E --> F{Callers need a stable name}
  F -->|yes| G[Publish records with external dns on the Supervisor]
  F -->|no| H[Hand out the Envoy VIP directly]
  C --> I{Workload is a VM Service machine}
  I -->|yes| J[Expose on the Supervisor, not in the cluster]
Exposure model chooser for objective 4.12, ending at the resource each path consumes

Preflight before a single manifest lands

Last Part left the estate with a PostgreSQL StatefulSet holding expanded volumes and a stateless web deployment sitting on cluster internal addresses only. This Part puts that web deployment in front of a browser and makes it pull from Harbor rather than a public registry. Four checks before anything is applied, and each one prevents a failure I have actually watched.

# Versions this Part was run against # VCF 9.0, vSphere Supervisor 9.0, VKS 3.3.1, workload cluster on Kubernetes 1.32, # NSX Advanced Load Balancer in the Supervisor load balancing path, # kubectl v1.32.3 with the kubectl-vsphere plugin shipped by the Supervisor. kubectl config current-context vks-app-prod kubectl get nodes –no-headers | wc -l 5 # 1. Confirm the cluster can already obtain an external address at all kubectl get svc –all-namespaces –field-selector spec.type=LoadBalancer NAMESPACE NAME TYPE EXTERNAL-IP PORT(S) kube-system vks-metrics-lb LoadBalancer 10.60.14.22 443:31842/TCP # 2. Confirm how much room is left in the Avi pool the Supervisor was given # (run against the Supervisor context, not the workload cluster) kubectl –context supervisor get virtualmachineservice -A –no-headers | wc -l 11 # 3. Confirm the private registry is reachable and trusted from this cluster kubectl get tkc app-prod -n ns-app -o jsonpath="{.spec.settings.network.trust}" {"additionalTrustedCAs":[{"name":"harbor-ca"}]} # 4. Confirm no ingress controller is already installed and fighting for the class kubectl get ingressclass No resources found

Check two is the one candidates skip and the one that decides everything below. If your Supervisor was enabled with a VIP range of 30 addresses and eleven are already spoken for, you have nineteen LoadBalancer services left in the entire estate, across every workload cluster, forever, until somebody extends the range at the Supervisor. Check three returning an empty result is a guaranteed image pull failure later, and it is far cheaper to see it now than at step five.

Numbered run, deployment to service to ingress to private image

Step 1. Deploy the workload and expose it the naive way, so you have a measured baseline to argue against. Three replicas, one Service of type LoadBalancer, and a stopwatch on how long the address takes to appear.

kubectl create namespace shop kubectl create deployment web –image=harbor.lab.local/shop/web:1.4 –replicas=3 -n shop kubectl expose deployment web –type=LoadBalancer –port=80 –target-port=8080 -n shop kubectl get svc web -n shop –watch NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE web LoadBalancer 198.51.100.41 <pending> 80:31207/TCP 3s web LoadBalancer 198.51.100.41 10.60.14.31 80:31207/TCP 41s # 41 seconds, and one address permanently removed from the Supervisor pool.

Step 2. Watch the pods fail to start, because the image came from a private registry and nothing told the cluster how to authenticate. This is the failure the objective expects you to recognise from its error text rather than by guessing.

kubectl get pods -n shop NAME READY STATUS RESTARTS AGE web-6c4f9d7b58-2xk4l 0/1 ImagePullBackOff 0 52s web-6c4f9d7b58-9wqpr 0/1 ImagePullBackOff 0 52s web-6c4f9d7b58-hd8mn 0/1 ErrImagePull 0 52s kubectl describe pod -n shop -l app=web | grep -A3 Failed Warning Failed 18s (x3 over 51s) kubelet Failed to pull image "harbor.lab.local/shop/web:1.4": failed to resolve reference "harbor.lab.local/shop/web:1.4": failed to authorize: failed to fetch oauth token: unexpected status from GET request to https://harbor.lab.local/service/token: 401 Unauthorized # Fix: reference the registry secret created in Part 23. Never inline the password. kubectl create secret docker-registry harbor-pull -n shop –docker-server=harbor.lab.local –docker-username="$HARBOR_USER" –docker-password="$HARBOR_PASS" kubectl patch deployment web -n shop –type=strategic -p='{"spec":{"template":{"spec":{"imagePullSecrets":[{"name":"harbor-pull"}]}}}}’ kubectl get pods -n shop web-7f9b4c6d94-4nzsv 1/1 Running 0 14s

Read that error carefully, because two different registry failures produce visually similar output and the exam distinguishes them. A 401 Unauthorized means the cluster reached Harbor, completed a TLS handshake, and was refused credentials, so the answer is a pull secret. An x509 message naming an unknown authority means the handshake never completed, so the answer is trust material in the cluster spec, and a pull secret will not help. HARBOR_USER and HARBOR_PASS are environment variables in my shell; a credential typed on a command line lands in shell history and, on a shared jump host, in somebody else audit log.

Step 3. Install an ingress controller, and here is where the published tutorial actively hurts you. Broadcom documentation for VCF 9.0 still walks through Contour release 1.9, a ClusterRoleBinding against psp:vmware-system-privileged, and an Ingress manifest on networking.k8s.io/v1beta1. PodSecurityPolicy was removed from Kubernetes in 1.25 and that beta Ingress API stopped being served in 1.22. On a VKS 3.3.1 cluster running Kubernetes 1.32 all three fail.

# What happens if you follow the published tutorial literally kubectl create clusterrolebinding default-tkg-admin-privileged-binding –clusterrole=psp:vmware-system-privileged –group=system:authenticated Error from server (NotFound): clusterroles.rbac.authorization.k8s.io "psp:vmware-system-privileged" not found kubectl apply -f contour.yaml unable to recognize "contour.yaml": no matches for kind "Ingress" in version "networking.k8s.io/v1beta1" # What to do instead: take Contour from the standard package repository you # synced in Part 23, so the version is matched to the Kubernetes release. kubectl get packages -n tkg-system –field-selector spec.refName=contour.tanzu.vmware.com -o custom-columns=NAME:.spec.refName,VERSION:.spec.version NAME VERSION contour.tanzu.vmware.com 1.30.1+vmware.1 export CONTOUR_VERSION=$(kubectl get packages -n tkg-system –field-selector spec.refName=contour.tanzu.vmware.com -o jsonpath="{.items[-1:].spec.version}") kubectl create namespace projectcontour kubectl create clusterrolebinding contour-admin –clusterrole=cluster-admin –serviceaccount=projectcontour:contour-sa

The version string above is what my repository carried on the day I ran this; yours comes out of that same command and will differ. Pinning the package version by reading it from the repository rather than typing it is the habit that survives an upgrade, and it is exactly the pattern the package objective in Part 23 sets up.

Step 4. Convert the application from its own virtual IP to a route behind the shared one. Note the two fields that did not exist in the published tutorial and are mandatory now: ingressClassName replaces the old annotation, and pathType is required on every path.

# Give back the dedicated VIP first kubectl patch svc web -n shop -p='{"spec":{"type":"ClusterIP"}}’ cat <<EOF | kubectl apply -f – apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: web namespace: shop spec: ingressClassName: contour rules: – host: shop.lab.local http: paths: – path: / pathType: Prefix backend: service: name: web port: number: 80 EOF ingress.networking.k8s.io/web created kubectl get ingress -n shop NAME CLASS HOSTS ADDRESS PORTS AGE web contour shop.lab.local 10.60.14.28 80 9s

Nine seconds, and no new address consumed. Every application added after this one costs zero VIPs. That single arithmetic difference is the reason objective 4.12 lists ingress controllers alongside pods.

Verification, rollback and a failure lookup

Green looks like three things agreeing: an Envoy service holding an external address, an Ingress reporting that same address, and an HTTP 200 arriving with the Host header set. Checking only the first two is how people convince themselves a broken route works.

kubectl get svc -n projectcontour envoy NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE envoy LoadBalancer 198.51.100.87 10.60.14.28 80:30501/TCP,443:30173/TCP 6m kubectl get pods -n projectcontour NAME READY STATUS RESTARTS AGE contour-6d7f8b9c4-lm2vt 1/1 Running 0 6m contour-6d7f8b9c4-x8jrq 1/1 Running 0 6m contour-certgen-2wq9f 0/1 Completed 0 6m envoy-p4rkc 2/2 Running 0 6m # Prove the route, not just the address curl -s -o /dev/null -w "%{http_code}n" –resolve shop.lab.local:80:10.60.14.28 http://shop.lab.local/ 200 # Rollback: routes first, controller last, so nothing is stranded kubectl delete ingress web -n shop kubectl patch svc web -n shop -p='{"spec":{"type":"LoadBalancer"}}’ kubectl delete namespace projectcontour

Rollback order matters more than it looks. Delete the projectcontour namespace while ingress objects still reference it and those objects survive with no controller behind them, reporting a stale address that answers nothing. I have handed a stale address to an application team and spent twenty minutes of their time proving my own mess.

Below is the artifact worth keeping from this Part: a failure to cause lookup covering the four exposure and registry errors that account for nearly every broken deployment I have debugged on VKS, each with the layer that actually owns the fix.

What you see Real cause Where the fix lives
Service EXTERNAL-IP stuck at pending past two minutesAvi VIP range exhausted, or the Supervisor was enabled with no load balancer at allSupervisor, not the workload cluster. Extend the range or switch the application to ingress.
ImagePullBackOff with 401 UnauthorizedNo imagePullSecrets on the pod spec, or a secret in the wrong namespaceWorkload namespace. Secrets are namespaced and do not travel with a deployment.
ImagePullBackOff with x509 certificate signed by unknown authorityRegistry CA absent from the cluster trust bundleCluster spec on the Supervisor. Needs a rolling update, not a pod restart.
Ingress created but ADDRESS column stays emptyNo ingress controller installed, or ingressClassName names a class that does not existWorkload cluster. Compare against kubectl get ingressclass.
404 from Envoy while the address answersHost header does not match the rule, or pathType was omitted so nothing matchedIngress object. Test with curl and an explicit resolve before blaming DNS.
Time from apply to a first HTTP 200 Measured on one VKS 3.3.1 cluster, Kubernetes 1.32, Avi in the Supervisor load balancing path Route on a running controller 9 s, zero new VIPs Service type LoadBalancer 41 s, one VIP consumed Route plus published DNS name 74 s, zero new VIPs First Contour install plus route 186 s 0 s 200 s

Read that chart the way an architect would rather than the way a stopwatch would. Installing the controller is the slowest bar and it happens once per cluster. Every bar after it is faster than the LoadBalancer path and free in address terms. Optimising the one time cost is the wrong instinct.

Exam focus for objective 4.12

Objective 4.12, what it actually tests: Whether you can choose an exposure model for a described application and say what it consumes, and whether you can look at a broken pod and name the layer that owns the fix. Expect this objective in scenario framed multiple choice, in matching items pairing an error string to a remediation, and in point and click or hot area items on a manifest where you identify the field that is wrong. The trap that catches experienced vSphere admins: they read ingress controller as a platform feature that the Supervisor supplies, in the way it supplies storage classes and VM classes. It does not. Kubernetes ships no ingress controller, and a VKS workload cluster arrives with none installed, so an Ingress object applied to a fresh cluster is accepted by the API server, reports an empty ADDRESS column, and routes nothing. A second, quieter trap: assuming the imagePullSecret and the registry CA are interchangeable fixes. One is a namespaced secret you can add in seconds; the other lives in the cluster spec and costs a rolling update of every node.

Objective checkpoint

Three original questions written from the published objective wording. No real exam items appear here.
1. A team applies an Ingress to a newly provisioned VKS cluster. It is accepted, but the ADDRESS column stays empty for ten minutes and no traffic arrives. Which action resolves this fastest? (a) Extend the Avi VIP range on the Supervisor. (b) Install an ingress controller in the cluster and set ingressClassName to match. (c) Recreate the backing Service as type LoadBalancer. (d) Add the registry CA to the cluster spec.
Answer: b. An Ingress is only a set of rules; with no controller watching for them, nothing is programmed and no address is claimed. Option c would work but spends a VIP to solve a problem that costs none.
2. Pods report ImagePullBackOff and the events show x509 certificate signed by unknown authority against an internal Harbor. Which remediation is correct? (a) Create a docker-registry secret in the workload namespace. (b) Restart the deployment. (c) Add the Harbor CA to the trust settings in the cluster spec and let the rolling update complete. (d) Change imagePullPolicy to IfNotPresent.
Answer: c. An x509 failure happens before authentication, so credentials are irrelevant. Trust material is part of the cluster specification and reaching the nodes requires a rolling update.
3. An application has a containerised front end in a VKS cluster and a licensed database on a VM Service virtual machine. Where is the database exposed from? (a) An Ingress in the VKS cluster. (b) A Service in the VKS cluster selecting the VM. (c) The vSphere Namespace on the Supervisor that owns the machine. (d) A NodePort on the VKS worker nodes.
Answer: c. A VM Service machine is a Supervisor object living in a vSphere Namespace. No selector in the workload cluster can reach it, because it is not a member of that cluster.

Exposure defaults I would standardise on

My verdict is narrow and I will defend it. Install one ingress controller per VKS cluster on day one, before any application arrives, and treat a Service of type LoadBalancer as an exception that needs a sentence of justification. Reserve it for protocols ingress cannot carry, which in practice means databases, message brokers and anything doing mutual TLS that must terminate at the application. Avoid the pattern of one LoadBalancer per microservice entirely, even though it is what every quick start demonstrates and what most people build their first estate on.

I earned that opinion badly. On a customer demo estate sized with a 30 address VIP pool, a platform team shipped fourteen services of type LoadBalancer across two clusters over about six weeks, one per microservice, exactly as their tutorials showed. Service fifteen sat at pending, and I spent forty minutes convinced Avi had a health problem because the Avi console looked entirely healthy: nothing was down, the pool was simply full. Reversing it meant collapsing eleven of those services behind a single Contour instance in an evening change window, and the whole estate then ran on three virtual IPs instead of fourteen. Nobody noticed a latency change. What we bought back was eleven addresses and a routing layer we could put certificates and host rules on, which the previous design had no place for.

Second recommendation, smaller and less obvious. Take your ingress controller from the standard package repository rather than from an upstream manifest on the internet, and pin the version by reading it out of the repository. Broadcom own published Contour tutorial is the argument for this: it has drifted far enough that following it produces three separate errors on a supported cluster, and an upstream manifest you pasted a year ago will drift the same way. Package repositories in Part 23 and Harbor as the image source in Part 19 exist precisely so this Part has somewhere to pull from. Broader product context for all of it sits in the VKS Series, and the exam map is on the guide page.

A clean result looks like this: One ingress controller running in the cluster with an Envoy service holding exactly one external address. An ingressclass named contour visible to kubectl. One application reachable by host name through that shared address, verified with curl and an explicit resolve rather than by DNS luck. Zero services of type LoadBalancer that could have been routes instead. A deployment whose pods pull from Harbor using a namespaced pull secret, with both registry failure modes seen at least once, the 401 and the x509. And a note somewhere of how many addresses remain in your Supervisor VIP range, because that number is the budget every future application spends from.

Tonight, in your own lab, do one thing: count your remaining virtual IPs, then take an application that currently owns one and move it behind an ingress route. Watch the address return to the pool. Twenty minutes, and the trade off in objective 4.12 becomes something you have felt rather than something you memorised. Next Part moves to backup and restore with Velero.

VCAP-VKS Exam Series · Part 27 of 34
« Previous: Part 26  |  Guide  |  Next: Part 28 »

References

Deploying Workloads on VKS Clusters, VMware Cloud Foundation 9.0 documentation
Ingress Using Contour, Broadcom TechDocs, the tutorial discussed in step 3
Using Private Registries with VKS Clusters, Broadcom TechDocs
Deprecated API Migration Guide, Kubernetes documentation, Ingress v1beta1 removal in 1.22 and PodSecurityPolicy removal in 1.25
VMware Certified Advanced Professional vSphere Kubernetes Service exam guide, 3V0-24.25

About The Author


Discover more from Journal of Intelligent Infrastructure

Subscribe to get the latest posts sent to your email.

Leave a Reply

Your email address will not be published. Required fields are marked *

Architect’s Toolkit

About the Author

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

Discover more from Journal of Intelligent Infrastructure

Subscribe now to keep reading and get access to the full archive.

Continue reading