illegal base64 data at input byte 0. That single line is everything a VKS cluster gives you when a certificate authority bundle carries one stray character, and the cluster it kills sits in Provisioning looking entirely healthy from the vSphere Client. Nothing in that message mentions a certificate. Nothing mentions whitespace. It cost me most of an afternoon the first time I met it, and it is why this objective belongs in your notes as an encoding problem wearing a security costume.
Preflight and default CNI at Supervisor level
A container network interface, CNI for short, is the plugin that hands every pod an IP address and enforces network policy inside a Kubernetes cluster. VKS supports exactly two: Antrea, whose data plane is Open vSwitch, and Calico, which routes over the Linux bridge with BGP and enforces policy with Linux IP tables. Antrea is the system defined default. Half the resource names you will type still carry tkg, because VKS is what TKG Service and before that TKGS became, so a custom resource in the cni.tanzu.vmware.com API group is current rather than legacy. Last Part attached a subscribed release library at the Supervisor. This one changes nothing about which images arrive and everything about what those images trust and how their pods get wired.
Default CNI is set in the vSphere Client, under Supervisor Management, then Supervisors, then your Supervisor instance, then Configure, then Kubernetes Service, then Default CNI. Broadcom exposes no supported CLI for it in 9.0, which is worth knowing because a point and click item on this objective almost has to test that path. Two properties of that dialog are examinable and both catch people. Changing it is a global operation for that Supervisor, not a per namespace one, which is the same Supervisor scoping rule the release library follows. And existing clusters are unchanged, so switching from Antrea to Calico on a Friday afternoon does absolutely nothing to the twelve clusters already running and everything to the next one somebody builds.
Popular study advice says flip the default to Calico if you want serious network policy, on the grounds that Calico has been doing it longer. In a VCF estate that advice is backwards. Antrea is the CNI that the NSX integration path expects, the one the Antrea NSX adapter attaches to, and the one that gives you pod level visibility inside NSX rather than a black box hanging off a segment. Pick Calico when a workload team already owns Calico policy manifests they refuse to rewrite. Otherwise the cost of leaving the default alone is zero and the cost of changing it is that every NSX security conversation for the next two years starts with an apology.
Per cluster CNI choice and Antrea tuning
Below the Supervisor default sit two per cluster levers. Calico can be named directly in the cluster manifest, which overrides the default for that one cluster and leaves everybody else alone. Antrea behaviour is tuned through an AntreaConfig custom resource in the cni.tanzu.vmware.com API group, and on VCF 9.0.x you create that object yourself in the vSphere Namespace before the cluster exists. Naming is not optional. Metadata name must be the cluster name followed by the suffix -antrea-package, exactly, or the controller never associates it and your carefully written feature gates are silently ignored.
An empty SECRETREF column means the object exists and nothing has consumed it. Ninety percent of the time that is a name typo. Editing an AntreaConfig after the cluster is running is a legitimate operation, but the change does not reach any node until a rolling update replaces it, which is the recurring theme of this whole objective and the reason the chart further down measures minutes rather than seconds.
NSX objects a VKS cluster consumes
Broadcom publishes a small table mapping cluster networking endpoints to their providers, and it repays memorising because it tells you which failures are Kubernetes problems and which are NSX problems. Pod connectivity, ClusterIP and network policy are handled entirely by the CNI and create no NSX object at all. NodePort is kube-proxy opening a port on each worker. Only service type LoadBalancer reaches into the fabric, and on NSX Load Balancer that means one virtual server per service type definition, which is the sentence that turns into a virtual IP exhaustion incident three months later. Ingress ships with nothing: VKS expects you to install a third party controller such as Contour, and an Ingress object sitting with no address is usually a missing controller rather than a broken fabric. Routable pod networking is the one case where pod addresses leave the overlay, selected by naming antrea-nsx-routed as the CNI. NSX itself is owned by the NSX Series, so here I only care about the seam.
| CNI option | Data plane and policy | Pod addresses | Pick it when |
|---|---|---|---|
| Antrea, the default | Open vSwitch for both traffic and policy | Private to the cluster overlay | Always, unless something below forces your hand |
| antrea-nsx-routed | Open vSwitch, policy enforceable in NSX | Routable, source IP preserved outside the cluster | An external system must identify or filter individual pods by address |
| Calico | Linux bridge with BGP, policy in IP tables | Private to the cluster overlay | A team owns existing Calico policy and will not port it |
Trusted CA certificates in a cluster spec
Node virtual machines in a VKS cluster are immutable. You do not install anything on them, you declare what they should contain and let a rolling update rebuild them. Extra certificate authority certificates, the thing you need before a node will pull an image from a Harbor instance signed by an internal CA, are declared as a cluster topology variable called osConfiguration, under trust, under additionalTrustedCAs, each entry pointing at a key inside a Kubernetes secret. PEM, meaning the base64 text format certificates normally arrive in, is what goes inside that key, and it goes in encoded twice.
Encoded twice is where everybody falls over, and the reason is worth one clear sentence. Kubernetes already base64 encodes whatever you put in a secret. VKS then expects the value it decodes out of that secret to itself be base64 of the PEM. So if you write the secret as raw YAML with a data field, you supply the string double encoded by hand. If you build the same secret with kubectl create secret generic and from-file, kubectl performs the outer encoding for you and you supply it encoded only once. Both routes are correct and mixing them produces the two failures that fill the knowledge base.
Now the failure I promised, reproduced exactly, because a happy path here teaches nothing. My certificate file ended with a blank line. One newline byte and four spaces, invisible in every editor I opened it in.
Verification, rollback and failure lookup
Reading the cluster object back tells you the patch was accepted. It does not tell you the certificate reached a node. Only a pull from the registry signed by that CA proves trust is live, so that is the verification I use and the one I would want to see in a lab exercise.
Rollback has two speeds. Removing the osConfiguration variable, or repointing secretRef at a known good secret, backs the change out cleanly but costs another full rolling update, because trust only changes when nodes are replaced. Backing out a CNI decision has no fast path at all: a cluster cannot change its CNI in place, so the rollback is a new cluster and a workload migration. Plan accordingly, and never let a CNI experiment run on the cluster that carries the PostgreSQL StatefulSet.
Here is the artifact worth keeping from this Part, a failure to cause lookup covering both halves of the objective. I keep it printed next to the desk and I have used every row of it at least twice.
| What you see | Actual cause | Fix |
|---|---|---|
| illegal base64 data at input byte 0 | Stray character or trailing whitespace inside the encoded certificate | Decode twice, inspect the tail bytes, strip blank lines, re-encode |
| First node builds, no further nodes appear, connection refused on port 6443 | Expired or single encoded CA broke cloud init write-files on the node | Build in a fresh vSphere Namespace with a correctly double encoded certificate |
| x509: certificate signed by unknown authority on image pull | Trust declared but nodes never rolled, or CA added to the wrong cluster | Confirm machinedeployment UPDATED equals REPLICAS before blaming the registry |
| AntreaConfig exists, SECRETREF column empty, feature gates ignored | Metadata name is not cluster name plus the -antrea-package suffix | Rename the object, recreate it before the cluster, rebuild the cluster |
| Default CNI switched but new cluster still runs Antrea | Cluster manifest names a CNI explicitly, which beats the Supervisor default | Remove the CNI entry from the manifest or accept the per cluster override |
| Service type LoadBalancer stuck Pending with no external address | Load balancer out of virtual IPs, one virtual server is consumed per service | Widen the virtual IP range at Supervisor level, covered properly in Part 11 |
Exam focus for objective 2.5
Objective checkpoint
| Original practice question | Answer and reasoning |
|---|---|
| An administrator sets the Supervisor default CNI to Calico. Twelve VKS clusters are already running Antrea. How many change? | None. Changing the default is global for the Supervisor but applies only to clusters created afterwards, and a cluster cannot swap CNI in place. |
| A cluster sits in Provisioning and the only message references a failed patch from the runtime extension handler. Which object is inspected next? | The secret named in additionalTrustedCAs. Patch generation validates trust configuration, and an encoding fault there surfaces as a generic patch failure with no mention of certificates. |
| Which cluster networking endpoint creates one virtual server per definition on NSX Load Balancer? | Service type LoadBalancer. Pod connectivity, ClusterIP and network policy are handled by the CNI and create no fabric object at all. |
CNI and trust plan for this estate
My pick for the reference estate, and for most VCF estates that look like it, is Antrea left as the Supervisor default, an AntreaConfig created per cluster so the feature gates are explicit rather than inherited, and antrea-nsx-routed reserved for the one or two clusters that genuinely need externally addressable pods. Avoid a global switch to Calico. Avoid, equally, hand editing certificates onto running nodes over SSH, which works for about six hours and then vanishes on the next rolling update while you are asleep. Harbor and its internal CA are the usual reason any of this comes up, and Harbor itself is covered in the Harbor series; the product level walkthrough of VKS lives in the VKS Series.
Tonight, take whatever CA bundle your registry uses, run it through base64 twice, decode it twice, and pipe the last twenty four bytes into xxd. If anything follows the final hyphen of END CERTIFICATE, you have just found the fault that would have cost you an afternoon, and you found it in forty seconds. Part 11 moves into planning and design and takes on load balancer sizing, which is where that Pending service in the lookup table above actually gets solved.
References
- Set the Default CNI for VKS Clusters, VMware Cloud Foundation 9.0, Broadcom TechDocs
- VKS Cluster Networking, VMware Cloud Foundation 9.0, Broadcom TechDocs
- Resolving the illegal base64 data error when adding a second additional trusted CA, Broadcom Knowledge Base 437848
- VKS cluster node deployment with additional trusted CAs fails during cloud init, Broadcom Knowledge Base 402780
About The Author
DrJha