, ,

vSphere Pod and VM Service Workload Deployment on a Supervisor (VCAP-VKS Exam Series, Part 18)

Objective 4.3 as one lab: a vSphere Pod and a VM Service virtual machine dropped into the same vSphere Namespace on VCF 9.0, with the real errors each model throws and the Supervisor setting sitting behind them.

VCAP-VKS Exam Series · Part 18 of 34
Key takeaways: Objective 4.3, published wording Create workloads as Supervisor Pods or via VM Service. A vSphere Pod, which is what the blueprint calls a Supervisor Pod, only runs on a Supervisor configured with NSX; a Supervisor built on vSphere Distributed Switch will accept your Deployment manifest and never schedule it. VM Service virtual machines run on any Supervisor networking mode, but need three namespace side bindings in place first: a VM class, a storage class projected from a storage policy, and a content library supplying VirtualMachineImage objects. Both models bill against the same vSphere Namespace quota, so a guaranteed VM class quietly consumes the ceiling you set in Part 17. Headline preflight is kubectl get virtualmachineclass,virtualmachineimage –namespace ns-platform-a before you apply anything at all.
$ kubectl apply -f dpj-web.yaml deployment.apps/dpj-web created $ kubectl get pods –namespace ns-platform-a NAME READY STATUS RESTARTS AGE dpj-web-6c4f9d5b78-2xk4n 0/1 Pending 0 14m dpj-web-6c4f9d5b78-p9r7t 0/1 Pending 0 14m

Fourteen minutes of Pending on a two replica deployment, on a Supervisor reporting Ready, in a namespace with more than half its quota free. Nothing in that output mentions networking. It is entirely a networking answer. That Supervisor was activated on vSphere Distributed Switch, and a vSphere Pod cannot exist without NSX. Objective 4.3 asks you to create workloads as Supervisor Pods or via VM Service, and a large slice of what it actually tests is knowing which of those two routes the Supervisor beneath you is capable of before you write a single line of YAML.

Last Part carved this estate into vSphere Namespaces and mapped them onto zones. This Part puts workloads inside one of them, by both routes the blueprint names, using the NSX VPC backed Supervisor rather than the vDS one. A vSphere Pod is a small virtual machine with its own Photon based Linux kernel running one or more OCI containers, placed on an ESX host by DRS and attached to Kubernetes by a per host process called spherelet. A VM Service virtual machine is a full guest operating system described by a VirtualMachine custom resource and reconciled by VM Operator. Same namespace, same quota, two completely different failure vocabularies.

Who this is for: vSphere admins and platform engineers who can already hand out a namespace and now need to prove workloads land in it. This Part covers Objective 4.3, published wording Create workloads as Supervisor Pods or via VM Service, and assumes you have completed Part 17. General container and Kubernetes mechanics are not re-taught here; the VKS Series owns that ground, and NSX segment behaviour belongs to the NSX Series.

Preflight for both workload models

Four objects decide whether either model will start, and all four are Supervisor side rather than workload side. Networking mode decides vSphere Pods. VM classes decide VM Service sizing. Storage policies projected as StorageClasses decide where disks land. Content library association decides which VirtualMachineImage objects appear. Run the discovery block below before you write a manifest, because every one of these produces an error message that names something other than the thing that is actually missing.

# Tested on: VCF 9.0, vCenter 9.0.0.0 build 24755230, Supervisor 9.0.1, # VKS 3.3.1, kubectl v1.32.3, kubectl-vsphere plugin 9.0.0, VCF CLI 9.0.0 $ kubectl vsphere login –server=sup-a.lab.drpranayjha.com –vsphere-username $DPJ_SSO_USER –insecure-skip-tls-verify # password is read from the prompt, never from a flag or a file $ kubectl config use-context ns-platform-a Switched to context "ns-platform-a". $ kubectl get virtualmachineclass –namespace ns-platform-a NAME CPU MEMORY AGE best-effort-medium 2 8Gi 6d guaranteed-small 2 4Gi 6d $ kubectl get virtualmachineimage –namespace ns-platform-a NAME DISPLAY-NAME OSTYPE AGE vmi-0a1b2c3d4e5f60718 ubuntu-24.04-server-cloud ubuntu64Guest 4d vmi-77aa31b5c9d0e2f43 photon-5-cloud-init vmwarePhoton64 4d $ kubectl get storageclass NAME PROVISIONER AGE vsan-default-storage-policy csi.vsphere.vmware.com 6d vsan-high-performance csi.vsphere.vmware.com 6d

An empty result from either of the first two commands is your stop signal. No VirtualMachineClass rows means nobody added a VM class to this namespace in vCenter. No VirtualMachineImage rows means either the content library is not associated with the namespace or it has not finished syncing, which Part 9 covers in full. Neither shortage stops kubectl apply from succeeding at the CLI, which is precisely why both catch experienced admins.

Keep this table. It is the artifact worth returning to when someone asks which model a given application belongs in, and it answers the comparison items this objective tends to produce.

CriterionvSphere PodVM Service VMPod inside a VKS cluster
Supervisor networking requiredNSX or NSX VPC onlyAny mode, vDS includedAny mode, vDS included
Object you createPod, Deployment, StatefulSetVirtualMachine custom resourcePod, inside the guest cluster API
SchedulerDRS through sphereletDRS through VM Operatorkube-scheduler in the cluster
Median time to Ready, warm image6 seconds95 seconds to reported IP11 seconds
Quota billed toNamespace, per podNamespace, per VM classNamespace, via node VMs
Kubernetes version you getSupervisor version, not yoursNot applicableChosen Kubernetes release
Best forShort lived and platform side containersAnything that resists containerisingApplication teams wanting a real cluster
flowchart TD
  A[Workload manifest ready] --> B{Supervisor networking mode}
  B -->|NSX or NSX VPC| C{Runs as a Linux container}
  B -->|vDS only| D[vSphere Pod unavailable on this Supervisor]
  C -->|Yes| E[vSphere Pod in the vSphere Namespace]
  C -->|No| F[VM Service virtual machine]
  D --> F
  D --> G[Pod inside a VKS cluster]
  E --> H[Namespace quota charged per pod]
  F --> H
  G --> I[Namespace quota charged through node VMs]
Model selection starts at the Supervisor networking mode, not at the workload.

vSphere Pod deployment procedure

Step 1, confirm the Supervisor speaks NSX. Step 2, apply a Deployment with explicit resource requests and limits. Step 3, expose it. Nothing about the manifest is special, which is the point of the model, but step 2 carries a trap that costs you quota rather than an error message.

# Step 1. Prove NSX is the networking stack for this Supervisor $ kubectl get network –namespace ns-platform-a NAME AGE ns-platform-a 6d # Step 2. Deployment with explicit sizing. Each replica becomes one vSphere Pod. $ cat dpj-web.yaml apiVersion: apps/v1 kind: Deployment metadata: name: dpj-web namespace: ns-platform-a spec: replicas: 2 selector: matchLabels: app: dpj-web template: metadata: labels: app: dpj-web spec: containers: – name: web image: harbor.lab.drpranayjha.com/library/nginx:1.27 ports: – containerPort: 80 resources: requests: cpu: 100m memory: 128Mi limits: cpu: 500m memory: 256Mi $ kubectl apply -f dpj-web.yaml deployment.apps/dpj-web created $ kubectl get pods –namespace ns-platform-a -o wide NAME READY STATUS RESTARTS AGE IP NODE dpj-web-7f5c884b6d-4wq2s 1/1 Running 0 9s 172.26.1.7 esx-a01 dpj-web-7f5c884b6d-nc8vl 1/1 Running 0 9s 172.26.1.8 esx-a03 # Step 3. Expose through the load balancer path built in Part 11 $ kubectl expose deployment dpj-web –type=LoadBalancer –port=80 –target-port=80 –namespace ns-platform-a service/dpj-web exposed $ kubectl get svc dpj-web –namespace ns-platform-a NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE dpj-web LoadBalancer 10.96.0.142 10.20.40.51 80:31240/TCP 38s

Omit the resources stanza and nothing fails. A vSphere Pod is sized to its requests and limits, so with none supplied the namespace LimitRange default applies, and on a freshly created namespace that default is 1 vCPU and 512 MiB per container. Two replicas of a web server that needs 128 MiB then reserve 2 vCPU and 1 GiB of hard reservation against a namespace ceiling. I have watched a team run out of a 64 GiB namespace at roughly forty pods and conclude the Supervisor was broken.

Storage for a vSphere Pod arrives as three separate VMDK types governed by two separate policy decisions, and that split confuses people for years. Container image cache and ephemeral disks are placed by a storage policy chosen when the Supervisor is activated, and no namespace level change will move them. Persistent volumes are placed by the storage policies assigned to the vSphere Namespace, which is the layer Part 7 covered. A pod can therefore pull, start and serve traffic while its PersistentVolumeClaim hangs Pending, and the two answers live on different pages of the vSphere Client.

One more number worth carrying into the exam room. Image cache warmth is per ESX host, not per namespace and not per Supervisor. On this estate a 187 MB image pulled onto a cold host took 41 seconds to reach Running against 6 seconds once cached, and DRS is free to place the next replica on a host that has never seen that image. Scale a Deployment from two replicas to eight across a three cluster zonal namespace and start times fan out across that whole range with no configuration change whatsoever. Anybody quoting a single vSphere Pod start time is quoting host history rather than platform behaviour.

Now the failure that opened this Part, reproduced deliberately on the vDS backed Supervisor in the same estate.

$ kubectl describe pod dpj-web-6c4f9d5b78-2xk4n –namespace ns-vds-b | tail -n 5 Events: Type Reason Age From Message —- —— —- —- ——- Warning FailedScheduling 14m default-scheduler 0/3 nodes are available: 3 node(s) had untolerated taint {node-role.kubernetes.io/control-plane: }. preemption: 0/3 nodes are available: 3 Preemption is not helpful for scheduling.

Only three nodes exist on that Supervisor and all three are control plane VMs, because without NSX there are no spherelet backed ESX hosts joined as schedulable nodes. Read that event as a networking mode statement, not a taint problem, and never spend an hour writing a toleration for it.

VM Service deployment procedure

VM Service takes three inputs and one optional bootstrap. Class, image, storage class, plus cloud-init user data if you want the guest to be reachable. Build the bootstrap secret from an environment variable rather than pasting a key into the manifest, because a VirtualMachine object is readable by everyone with namespace edit rights.

# Step 1. Bootstrap secret, key material supplied from the shell environment $ export DPJ_SSH_PUBKEY=$(cat ~/.ssh/id_ed25519.pub) $ cat bootstrap.yaml.tmpl apiVersion: v1 kind: Secret metadata: name: dpj-vmsvc-01-bootstrap namespace: ns-platform-a stringData: user-data: | #cloud-config users: – name: dpj sudo: ALL=(ALL) NOPASSWD:ALL ssh_authorized_keys: – ${DPJ_SSH_PUBKEY} $ envsubst < bootstrap.yaml.tmpl | kubectl apply -f – secret/dpj-vmsvc-01-bootstrap created # Step 2. VirtualMachine object. v1alpha1 is deprecated in VCF 9.0. $ cat dpj-vmsvc-01.yaml apiVersion: vmoperator.vmware.com/v1alpha3 kind: VirtualMachine metadata: name: dpj-vmsvc-01 namespace: ns-platform-a spec: className: best-effort-medium imageName: vmi-0a1b2c3d4e5f60718 storageClass: vsan-default-storage-policy powerState: PoweredOn bootstrap: cloudInit: rawCloudConfig: name: dpj-vmsvc-01-bootstrap key: user-data $ kubectl apply -f dpj-vmsvc-01.yaml virtualmachine.vmoperator.vmware.com/dpj-vmsvc-01 created # Step 3. Watch it settle $ kubectl get vm –namespace ns-platform-a NAME POWERSTATE CLASS IMAGE PRIMARY-IP AGE dpj-vmsvc-01 poweredOn best-effort-medium vmi-0a1b2c3d4e5f60718 10.244.6.19 1m52s

PRIMARY-IP staying blank past two minutes is nearly always the bootstrap, not the network. A key name mismatch between the secret and rawCloudConfig.key leaves cloud-init with nothing to apply, the guest boots fine, and VM Operator reports no address because VMware Tools never handed one up. Check the guest console in vCenter before you touch a segment.

Power state on a VM Service virtual machine is declarative rather than a one time instruction. Patch spec.powerState to PoweredOff and VM Operator powers the guest down and holds it there, and a manual power on from the vSphere Client gets reconciled straight back off inside a minute. Administrators used to driving VMs from the client find that behaviour surprising the first time they meet it, and it is precisely the sort of reconciliation loop a scenario item can be built around.

Here is the admission failure worth memorising, produced by asking for a VM class that exists in vCenter but was never added to this namespace.

$ kubectl apply -f dpj-vmsvc-02.yaml Error from server (Forbidden): error when creating "dpj-vmsvc-02.yaml": admission webhook "default.validating.virtualmachine.v1alpha3.vmoperator.vmware.com" denied the request: spec.className: Invalid value: "guaranteed-large": no VirtualMachineClass with that name is associated with namespace ns-platform-a

That message is a gift, because it names the namespace. Most objective 4.3 failures are not this polite. Note also that VKS is the product formerly called TKG Service, and the rename has not reached the code: this same VM Operator builds the node VMs of every VKS cluster, and on the cluster side you will still meet tkg in API groups and resource names.

Against the usual advice: Nearly every walkthrough reaches for a guaranteed VM class because it sounds like the safe production choice. On a namespace with CPU and memory limits set, guaranteed classes create hard reservations that count against the ceiling whether or not the guest uses them, and you hit the wall at a fraction of the VM count you sized for. Use best-effort classes in any namespace that also runs VKS clusters or vSphere Pods, and reserve guaranteed for a namespace with no limits at all. Same conclusion as Part 17, arrived at from the workload side.

Verification, rollback and failure signatures

Green looks like this: pods Running with an ESX host name in the NODE column rather than a control plane VM name, a Service holding an external address from your load balancer range, and a VirtualMachine showing poweredOn with a primary IP. Anything else, go to the lookup table below rather than guessing.

# Verification, one pass over both models $ kubectl get pods,svc,vm –namespace ns-platform-a NAME READY STATUS RESTARTS AGE pod/dpj-web-7f5c884b6d-4wq2s 1/1 Running 0 4m pod/dpj-web-7f5c884b6d-nc8vl 1/1 Running 0 4m NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/dpj-web LoadBalancer 10.96.0.142 10.20.40.51 80:31240/TCP 3m NAME POWERSTATE PRIMARY-IP virtualmachine.vmoperator.vmware.com/dpj-vmsvc-01 poweredOn 10.244.6.19 $ curl -s -o /dev/null -w ‘%{http_code}n’ http://10.20.40.51 200 # Rollback. Order matters only for the Service, which holds a virtual IP. $ kubectl delete svc dpj-web –namespace ns-platform-a $ kubectl delete deployment dpj-web –namespace ns-platform-a $ kubectl delete vm dpj-vmsvc-01 –namespace ns-platform-a $ kubectl delete secret dpj-vmsvc-01-bootstrap –namespace ns-platform-a

Deleting a VirtualMachine removes the guest and its boot disk. Any PersistentVolumeClaim you attached through spec.volumes survives, which is a mercy in a lab and a surprise in production when storage quota does not fall after a cleanup. Releasing the Service before the Deployment returns the virtual IP promptly instead of leaving it held while the pods drain.

What you seeSupervisor object behind itRemediation
Pod Pending, FailedScheduling, untolerated control plane taintSupervisor activated on vDS, no NSXRun the workload in a VKS cluster, or rebuild the Supervisor with NSX
Pod Pending, exceeded quota on cpu or memoryNamespace LimitRange default sizing the podAdd explicit requests and limits to every container
ErrImagePull, x509 certificate signed by unknown authorityHarbor CA not trusted by the SupervisorAdd the CA to the Supervisor trust store, per Part 10
ErrImagePull, unauthorizedNo registry secret referenced by the pod specCreate the docker-registry secret and set imagePullSecrets
Admission denied on spec.classNameVM class never added to this vSphere NamespaceAdd the class to the namespace in vCenter, then reapply
Admission denied on spec.imageNameContent library unassociated or still syncingCheck virtualmachineimage output, then resync the library
VirtualMachine created, PRIMARY-IP blank past two minutescloud-init key mismatch, or Tools not reportingMatch rawCloudConfig.key to the secret key, check the console
VirtualMachine stuck Creating, storageclass not foundStorage policy not assigned to the namespaceAssign the policy, wait for the StorageClass to project
Median seconds from apply to Ready Five runs each, ns-platform-a, VCF 9.0 with NSX VPC networking, 187 MB image vSphere Pod, cached 6 s vSphere Pod, first pull 41 s VKS cluster pod, cached 11 s VKS cluster pod, first pull 33 s VM Service, powered on 22 s VM Service, IP reported 95 s Image cache warmth moves a vSphere Pod by 35 seconds. Guest boot moves nothing.
Measured on the running lab estate, five runs per row, medians reported.

Exam focus for objective 4.3

EXAM FOCUS, objective 4.3: Published wording is Create workloads as Supervisor Pods or via VM Service. What it expects you to be able to do is pick the correct model for a described environment and name the prerequisite objects each one needs, then read a workload level symptom back to the Supervisor or namespace setting that produced it. Expect this material in matching and drag and drop items pairing a symptom to a cause, in build list items ordering the create sequence for a VM Service VM, and in point and click items on the namespace configuration card in the vSphere Client. The trap that catches experienced admins is treating a Pending vSphere Pod as a scheduling or quota problem. On a vDS backed Supervisor there are no ESX nodes joined to Kubernetes at all, so no amount of quota, tolerations or node selectors will ever place it. Read the node count in the FailedScheduling event first, and if it equals your control plane VM count, the answer is the networking mode the Supervisor was activated with.

Objective checkpoint

1. A Supervisor was activated with vSphere Distributed Switch networking and an Avi load balancer. A developer applies a two replica Deployment into their vSphere Namespace and both pods sit Pending. Which change makes the workload run without altering the Supervisor?
Answer: Provision a VKS cluster in that namespace and deploy into the cluster. Reasoning: vSphere Pods require NSX, so the only container path on a vDS Supervisor is a workload cluster.

2. A VirtualMachine manifest is rejected at apply time with an admission webhook error naming spec.className. Which object is missing, and where is it added?
Answer: A VirtualMachineClass associated with that vSphere Namespace, added by the vSphere administrator on the namespace card in the vSphere Client. Reasoning: VM classes exist Supervisor wide but must be bound per namespace before VM Operator will accept them.

3. Two vSphere Pods and one VM Service virtual machine run in a namespace with a 32 GiB memory limit. Which of the three contributes a reservation that persists even while the guest sits idle?
Answer: The virtual machine, if it uses a guaranteed VM class. Reasoning: guaranteed classes create full memory reservations, whereas vSphere Pods reserve only what their container requests declare.

Workload model call for this estate

Field note. On a customer estate last year I lost most of a working day to exactly the opening symptom, on a Supervisor somebody else had activated. Two hours went on quota arithmetic, another hour on a toleration I did not need, and the resolution arrived when a colleague asked which networking mode had been chosen at activation. It had been vDS, picked because NSX Edge capacity was not ready on the day, and that single choice removed vSphere Pods from that estate permanently without ever producing an error that said so. Total cost, roughly five hours plus a rebuild scheduled six weeks later.

Verdict for this lab, and for most production estates. Use vSphere Pods for platform side and short lived containers where the 6 second start and per pod visibility in vCenter earn their keep, use VM Service for anything that resists containerising, and push application teams into VKS clusters rather than vSphere Pods so they own their own Kubernetes version. Avoid guaranteed VM classes inside any namespace that also carries limits. Avoid, above all, applying a container manifest before you have confirmed the networking mode.

Clean result checklist: two pods Running with ESX host names in the NODE column, a Service holding an address from your load balancer range answering 200, a VirtualMachine reporting poweredOn with a primary IP, no bootstrap material written into any manifest, and a namespace whose used memory rose by roughly what you asked for rather than by a LimitRange default.

Tonight, in your own lab: deploy the same Deployment twice, once with explicit requests and limits and once without, then compare the namespace memory used figure on the vSphere Client namespace card. That single number is the fastest way to make the LimitRange behaviour permanent in your memory before exam day. Next Part installs Supervisor add-on services, starting with Harbor and external-dns, which is where those image pull failures in the table above finally get their fix. Full map in the VCAP-VKS Exam Complete Guide.

VCAP-VKS Exam Series · Part 18 of 34
« Previous: Part 17  |  Guide  |  Next: Part 19 »

References

About The Author


Discover more from Journal of Intelligent Infrastructure

Subscribe to get the latest posts sent to your email.

Leave a Reply

Your email address will not be published. Required fields are marked *

Architect’s Toolkit

About the Author

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

Discover more from Journal of Intelligent Infrastructure

Subscribe now to keep reading and get access to the full archive.

Continue reading