Two hundred and fourteen gigabytes. That is what an immediate synchronisation of the public Kubernetes release library pulled onto a lab datastore before I caught it, in an estate that was running two clusters off one image. Nobody teaches this objective as a capacity problem, and that is exactly why it bites.
VKr distribution model in VCF 9
A vSphere Kubernetes release, shortened to VKr everywhere in the documentation, is one signed and supported Kubernetes distribution packaged as an OVA template, which is the virtual machine appliance format vSphere has used for years. Broadcom publishes VKrs to a content delivery network. A vCenter content library, meaning the vSphere object that stores templates and optionally synchronises them from somewhere else, pulls them down. VKS, which is vSphere Kubernetes Service and the product Broadcom used to call TKG Service and before that TKGS, reads that library and clones each template into cluster node virtual machines. Disk size for those nodes is fixed by the OVA. CPU and memory come from the VM class you bound to the vSphere Namespace back in Part 5, not from the image. Resource names, API groups and half the documentation URLs still say tkg, so a command aliased to kr and a command aliased to tkr return the same objects.
Three properties of that pipeline decide most of what this objective asks. First, VKrs ship on their own cadence, released and updated independently of both VKS and Supervisor, so an image can be sitting in your library, fully synchronised, and still refuse to build anything because it is not compatible with the VKS version you are running. Second, and this is the sentence to memorise, a release library is bound to the Supervisor rather than to a vSphere Namespace. Broadcom states it plainly: the same library is used for all vSphere Namespaces on a Supervisor instance, and for that reason changing it is only permitted at the Supervisor level. Third, matching what you asked for in YAML against what the library actually holds is done by a resolver controller reading annotations, and when that resolver cannot find a match it denies the request outright rather than choosing something near enough.
Which library type you build is a design decision you make once and live with for years, so it belongs in your notes as a table rather than as prose. Air gapped in this context does not mean paranoid, it means the vCenter appliance has no route to a Broadcom endpoint, which describes a great many regulated estates. General content library mechanics belong to vSphere and are covered in the VKS Series; here I only care about what VKS does differently.
| Library type | Use it when | How images arrive | Main cost |
|---|---|---|---|
| Subscribed, When needed | vCenter can reach the depot or the public repository. Default choice. | Metadata syncs automatically, image files download on first use | First cluster of a new version waits for a template pull |
| Subscribed, Immediately | Provisioning latency matters more than datastore capacity, and only then | Metadata and every image file sync on schedule | Large, growing datastore footprint, plus orphaned files after upstream deletions |
| Local, air gapped | vCenter has no route out. Regulated, classified or offline sites. | You download OVA files on a connected host and import them by hand | Manual naming discipline, and a human step on every patch |
| Local, published to a subscriber | Several offline vCenters behind one internal distribution point | One local library publishes, remote sites subscribe internally | An internal endpoint and certificate you now own and must renew |
Preflight and version record
Four conditions gate this work. Your Supervisor must be Ready, because association triggers a reconcile that a busy Supervisor will accept and then quietly defer. You need vSphere 9.0 or later with VKS 3.3 or later installed if you want more than one library attached at a time, which is the capability that makes a mixed subscribed and local arrangement possible at all. You need enough free capacity on the chosen datastore, and I mean measured rather than assumed. And you need the correct subscription endpoint, which differs depending on whether your estate runs a Software Depot, the VCF component that caches Broadcom binaries locally.
Note how uninformative that last result is. An empty kubectl get kr has at least four distinct causes: no library exists, a library exists but is not associated with the Supervisor, a library is associated but has never synchronised, or a library synchronised and every item failed OVF certificate validation. Candidates who treat empty as one symptom rather than four lose time on the troubleshooting objectives later in the blueprint, and they lose it in the lab too. Learn to reach for govc library.ls before kubectl, because it splits those four causes into two groups in one command.
Subscribed library, from subscription URL to a ready VKr
Step 1, obtain the correct subscription URL
Two answers exist and picking the wrong one is a silent failure, not an error. Estates without a Software Depot, which includes most VMware vSphere Foundation deployments, use the standard Broadcom public repository at https://wp-content.broadcom.com/v2/latest/lib.json. Estates that do run a Software Depot must build the URL from their own depot: log in to vCenter, open Developer Center, then API Explorer, issue a GET against the lcm/depot/services API, and assemble the returned depot address and base path into a URL ending /PROD/COMP/VKR/lib.json. Use the public repository against a depot backed estate and you will pull images your lifecycle tooling does not know about.
Step 2, create the subscribed library
That –sub-ondemand=true flag is the CLI equivalent of choosing When needed in the New Content Library wizard, and it is the recommended setting in Broadcom documentation for one blunt reason: metadata is small and images are not. On my lab the metadata for the full catalogue landed in under 3 GB and six minutes. Choosing Immediately instead has a second cost people forget about. When items are deleted from the published library upstream, their contents stay in your subscribed library storage and you have to delete them by hand. That is how a datastore quietly fills with Kubernetes versions nobody can even provision any more.
Step 3, associate the library with the Supervisor
No supported CLI exists for this and it is a vSphere Client action, which is precisely why it turns up as a point and click item. Navigate to Supervisor Management, then Supervisors, select your Supervisor instance, open Configure, then General, and find the Kubernetes Service card. On vSphere 9.0 with VKS 3.3 or later you get Add and Remove buttons and can attach more than one library. On vSphere 8.x and earlier you get a single Content Library field with an EDIT link and one library only. Add your subscribed library, then wait for it to appear in the list before you navigate away, because the write is asynchronous.
Step 4, confirm releases are published and usable
Two strings, one image, and only one of them works. On the v1alpha3 API you put the NAME string in tkr.reference.name. On v1beta1 and v1beta2 you put it in topology.version. Broadcom prints VERSION in the same table for human readability and it is the wrong value to copy every single time. If your team pastes cluster manifests from a wiki, this is worth a lint rule rather than a training session.
Local library for an air-gapped estate
Here is the war story, because it is short and it cost me real time. Building the offline half of this lab, I created a local library, downloaded the OVA on a jump host, carried it across, and imported it with the item name taken straight from the VERSION column, plus sign and all. Import succeeded. Library showed one item. And kubectl get kr returned nothing, for ninety minutes, while I checked the Supervisor, the storage policy, the zones and eventually the Pinniped pods, because at that point I was guessing. Broadcom documents the rule in one sentence: if you are using a local content library, make sure the name you give the VKr matches the NAME string. The subscribed library does that naming for you, which is exactly why nobody who has only ever run subscribed libraries knows the rule exists. I reversed my decision to import by hand and wrote a four line script instead.
One warning about running both libraries at once, which VKS 3.3 lets you do and which is genuinely useful for testing an offline path beside a live one. If two attached libraries hold the same image, matching on Kubernetes version, name, VKr version and architecture, the resolver finds multiple candidates and denies the request rather than choosing. Broadcom documents an annotation on the cluster specification to nominate a library and break the tie. Plan for that before you attach the second library, not after somebody reports that cluster creation broke for no reason.
Verification, rollback and a failure lookup
Three checks prove the work, and they should be run in this order because each one narrows the search space for the next. Confirm the library exists and holds items with govc library.ls. Confirm the Supervisor publishes them with kubectl get kr, reading both boolean columns rather than just the first. Then confirm the resolver agrees with your manifest by inspecting a cluster that already exists, since a successful resolution stamps the cluster with a label naming the release it picked.
Rollback is where the sharp edge lives. Backing out a library change means selecting it under Configure, General, Kubernetes Service and clicking Remove, and existing clusters keep running because their nodes were cloned long ago. Do it in that order. If somebody deletes the library from vCenter first, it stays linked to VKS as a dangling reference, and you then cannot add or remove any other library until you unlink the deleted one, with an error message telling you so. I have watched a change window burn twenty minutes on exactly that sequence. Remove from VKS, then delete from vCenter, never the reverse.
| Symptom | Cause | Remediation |
|---|---|---|
| kubectl get kr returns No resources found | No library associated, or association never reconciled | Check govc library.ls first, then re-add under Configure, General, Kubernetes Service |
| Library has items but no releases publish | Local library item name does not match the VKr NAME string | Rename the item to the triple hyphen NAME form and wait for the controller |
| could not resolve VKr for version, on apply | Manifest carries the VERSION string with a plus sign | Replace with the NAME string in topology.version or tkr.reference.name |
| Release listed but COMPATIBLE is False | VKr newer than the installed VKS version supports | Upgrade VKS, or pick a release whose COMPATIBLE column reads True |
| Release READY False, reason deactivated | Superseded by a newer patch after a critical bug was found | Move to the superseding patch. Deactivated is intentional, not a sync fault. |
| Apply denied with multiple matching images | Two attached libraries hold the identical release | Annotate the cluster to nominate one library, or detach the duplicate |
| Item tagged Verification Failed after sync | OVF signing certificate rejected by the applied security policy | Metadata is kept but files will not sync. Fix trust, then resynchronise the item. |
| Cannot add or remove any library | A library was deleted from vCenter while still linked to VKS | Unlink the deleted library from VKS first, then repeat the operation |
Exam focus for objective 2.5
Objective checkpoint
| Original practice question | Answer and reasoning |
|---|---|
| An administrator changes the release content library while viewing a single vSphere Namespace. Which namespaces are affected? | All of them on that Supervisor. The association is not namespace scoped, which is why the change is only permitted at Supervisor level in the first place. |
| A local library contains one imported OVA, yet kubectl get kr reports nothing. What is checked first? | Item name. In a local library it must match the release NAME string exactly, including the triple hyphen separator. Subscribed libraries generate that name for you, local libraries do not. |
| A release appears in the listing with READY True and COMPATIBLE False. Can a cluster be provisioned from it? | No. Releases are versioned independently of VKS, so an image can be fully synchronised and still be unusable until VKS itself is upgraded. |
Library plan for the reference estate
My recommendation for this lab, and for most connected estates that resemble it, is one subscribed library set to When needed, attached at the Supervisor, with a second local library attached only while you are rehearsing the offline workflow and detached immediately afterwards. Avoid the pattern that most tutorials still teach, which is Immediately for the sake of faster first provisioning. Broadcom recommends on demand, my own numbers put the difference at 214 GB against 3 GB for an identical catalogue, and the eleven minutes you save on a first cluster is not worth a datastore you have to garbage collect by hand. Avoid, equally, leaving two libraries attached permanently unless you have already written the disambiguating annotation into your cluster templates.
Tonight, in your own lab, run kubectl get kr and then run govc library.ls against the same Supervisor, and reconcile the two lists by hand. Any item present in vCenter and absent from Kubernetes is either misnamed, incompatible, deactivated or failed verification, and working out which one takes about ten minutes and teaches this objective better than any amount of reading. Part 10 carries the second objective printed as 2.5, covering CNIs, NSX networking objects and TLS certificates, and it assumes the library you built here.
References
- Using Kubernetes Releases with VKS Clusters, vSphere Supervisor 9.0 Services and Standalone Components, Broadcom TechDocs
- Create a Subscribed Content Library, vSphere Supervisor 9.0, Broadcom TechDocs
- Add or Update VKS Content Libraries on a Supervisor, vSphere Supervisor 9.0, Broadcom TechDocs
- VKr and OS Image Resolution Process, vSphere Supervisor 9.0, Broadcom TechDocs
About The Author
DrJha