, ,

Security Context Constraints and Pod Admission, the Biggest Portability Gap (TKGI to OpenShift Series, Part 7)

Security Context Constraints reject workloads that ran for years on TKGI, and no migration tool fixes that for you. Field level mapping, five real rejection errors, and the remediation paths ranked by cost.

TKGI to OpenShift Series · Part 7 of 26

That deployment had run for three years on Tanzu Kubernetes Grid Integrated Edition (TKGI, formerly Enterprise PKS) without anybody thinking about it. Same image, same manifest, same replica count, moved across with the same tooling that moved everything else. On OpenShift it never got as far as starting a container. Nothing about networking, storage or the migration toolchain was involved. A validating admission plugin read one integer in the pod spec and refused.

# Versions tested: TKGI 1.18, OpenShift Container Platform 4.20, oc client 4.20 $ oc apply -f payments-api-deployment.yaml deployment.apps/payments-api created $ oc get pods -n payments NAME READY STATUS RESTARTS AGE No resources found in payments namespace. $ oc get events -n payments –field-selector reason=FailedCreate LAST SEEN TYPE REASON OBJECT MESSAGE 19s Warning FailedCreate replicaset/payments-api-7c9d4b8f6d Error creating: pods "payments-api-7c9d4b8f6d-" is forbidden: unable to validate against any security context constraint: [provider "restricted-v2": .spec.securityContext.runAsUser: Invalid value: 0: must be in the ranges: [1000670000, 1000679999]]
Key takeaways:
• Your TKGI 1.18 clusters are not running Pod Security Policy. Kubernetes removed it at 1.25, so there is nothing to export and nothing to translate.
• On OpenShift, Pod Security Admission is telemetry and a Security Context Constraint (SCC) is enforcement. Copying namespace labels across buys you nothing.
• Fastest debugging move: oc get pod POD -o jsonpath='{.metadata.annotations.openshift.io/scc}’ tells you which SCC admitted a running pod.
• On a 118 workload estate, 61 passed restricted-v2 untouched and 57 did not. Of those 57, 31 needed only a manifest edit.
• Treat every SCC grant as a defect you owe somebody, not a fix you shipped.
Who this is for: You have the wave list from Part 6 and you are about to find out what your Wave 1 workloads look like to an OpenShift Container Platform 4 (OCP) API server. Terms defined here on first use: SCC is a Security Context Constraint, the cluster scoped OpenShift object that validates pod security fields; PSP is PodSecurityPolicy, the removed Kubernetes equivalent; PSA is Pod Security Admission, its namespace label based replacement; RBAC is role based access control; MCS is Multi Category Security, the SELinux labelling scheme OpenShift allocates per project. If you are still weighing VMware Kubernetes Service as the landing place instead, the TKGI to VKS guide covers that path and this admission problem largely goes away there.

Admission on TKGI, and What Actually Enforced It

Nobody in a TKGI shop thinks about admission, and there is a structural reason for that. PSP was the original mechanism, a cluster scoped object that validated pod specs against an allow list and reached workloads through RBAC. Kubernetes removed PSP outright in 1.25, which means a TKGI 1.18 estate is not running PSP at all, whatever your runbooks still say. Broadcom guidance required replacing PSP configurations with PSA before upgrading past TKGI 1.16, and TKGI 1.18 went further by dropping support for the SecurityContextDeny admission controller entirely.

PSA is what your clusters run now. It works on namespace labels: three levels, privileged, baseline and restricted, crossed with three modes, enforce, audit and warn. Documentation makes that sound tidy. Real estates are not. In every TKGI environment I have inventoried, most namespaces carry no PSA label at all, or carry privileged because somebody hit a wall during an upgrade and labelled their way out of it. An unlabelled namespace inherits the cluster default, and on TKGI that default is almost always permissive, because tightening it would have broken workloads that were already running.

So your effective posture is: anything goes. Containers run as uid 0 because a base image said USER root in 2019 and nobody revisited it. A log shipper mounts hostPath. A monitoring agent asks for SYS_TIME. None of that was a decision anybody made. It accumulated, quietly, for years, and it worked because nothing was checking.

Now count what it costs. Our reference estate is the one from Part 5 and Part 6: 47 namespaces, 118 Deployments, StatefulSets and DaemonSets between them. I ran every one of those specs through a server side dry run against a fresh OCP 4.20 cluster and recorded the first blocking reason for each. Sixty one passed untouched. Fifty seven did not, and the shape of that failure set is the whole story of this part.

What restricted-v2 does to 118 unmodified TKGI workloads Counted by first blocking reason, so each workload appears exactly once. Admitted unchanged 61 runAsUser 0 31 hostPath volume 11 Added capabilities 7 Host network or port 5 Privilege escalation 3 47 namespaces, TKGI 1.18 source, OCP 4.20 target, server side dry run.
One field, runAsUser, accounts for more than half of every rejection. Fix that one pattern and your Wave 1 shrinks by a third.

SCC Evaluation Order and Default Assignment

An SCC does roughly what PSP did, and Red Hat kept it alive when upstream dropped PSP. Two things about how it is granted trip up people arriving from PSA. First, an SCC reaches a workload through RBAC and not through a namespace label, so you need a Role or ClusterRole permitting use of that SCC plus a binding that ties it to a service account. Second, restricted-v2 is already granted to every authenticated user by default, which means every workload in your cluster has at least one SCC available whether you configured anything or not.

Evaluation order is where the surprises live. When a pod create request arrives, admission collects every SCC the requesting service account may use, sorts them by restrictiveness, and tries them from least privileged upward. First SCC that validates the entire pod spec wins, and OpenShift stamps the winner onto the pod as an openshift.io/scc annotation. If nothing validates, you get the field level rejection at the top of this article. Reading that annotation on a running pod is the fastest debugging move available to you, and it is the one command I wish somebody had shown me in week one.

PSA still exists on OpenShift, and it is not doing what you would guess. OpenShift enforces the privileged level globally at the PSA layer and leaves restricted set only to warn and audit on new namespaces. A separate controller then synchronises those warn and audit labels upward to match the most privileged SCC any service account in that namespace can use, specifically so a legitimately privileged workload does not spray warnings across your logs. Net effect: on OpenShift, PSA reports and SCC decides. That single fact invalidates the most common migration plan I see written down.

Contradicts the usual advice: Every migration checklist I have been handed says export your pod security policies and build matching SCCs on the target. Do not. Your PSPs were deleted by a Kubernetes upgrade years ago, and your PSA namespace labels carry no authority to OpenShift because OpenShift already enforces privileged at that layer. Teams who follow that plan spend a week authoring SCC objects nobody needs and still get rejected pods on day one, because the real constraint was never in a policy object. It was in the pod spec and in the image.
flowchart TD
  A[Pod create request] --> B[PSA namespace check]
  B --> C[OpenShift enforces privileged globally so PSA rarely blocks]
  C --> D[SCC admission gathers every SCC the service account may use]
  D --> E[Sort by restrictiveness, least privileged tried first]
  E --> F{Does one validate the whole spec}
  F -->|Yes| G[Pod annotated openshift.io/scc with the winner]
  F -->|No| H[Request rejected with a field level error]
Admission path for a pod on OpenShift 4.20. Granting a permissive SCC does not relax the rules, it adds a later candidate that a stricter one no longer has to satisfy.

Mapping PSA Labels and PSP Rules onto SCCs

Work in the opposite direction from the checklist. Start at what each workload asks for in its pod spec, and look up what restricted-v2 is willing to give it. That is a field level exercise rather than a policy translation exercise, and it produces something useful: a list of image and manifest changes owned by application teams, instead of a list of cluster objects owned by you.

Below is the artifact worth keeping from this part. Print it, put it beside the wave list from Part 6, and walk every rejected workload through it. Call it the admission field mapping table. It is the thing I reach for in every migration triage call, and it has not changed shape in three engagements.

What the workload asks forWhere it was permitted on TKGIOpenShift SCC fieldDefault under restricted-v2What happens if you ignore it
runAsUser 0Privileged or unlabelled namespacerunAsUser strategy MustRunAsRangeUnset, or inside the project uid rangeRejected at create, no container ever starts
allowPrivilegeEscalation trueKubernetes default when unsetallowPrivilegeEscalationfalsePod admitted, container exits with operation not permitted
Capabilities such as NET_ADMIN or SYS_TIMEPSP allowedCapabilities, later nothing at allallowedCapabilities and requiredDropCapabilitiesDrop ALL, allow only NET_BIND_SERVICERejected at create
hostPath volume mountPrivileged or unlabelled namespaceallowHostDirVolumePlugin and volumes listfalse, hostPath absent from allowed volumesRejected at create, needs a redesign not a grant
hostNetwork or hostPortPrivileged namespace, common for NCP era agentsallowHostNetwork and allowHostPortsfalseRejected at create, needs hostnetwork-v2
fsGroup for a shared volumeAny value, uncheckedfsGroup strategy MustRunAsProject allocated supplemental group rangePod runs green, first write to the PVC fails with permission denied
seccompProfile left unsetUnconfined in practiceseccompProfilesDefaulted to runtime/default by the SCCUsually fine, custom profiles need an explicit grant
No SELinux context setNot enforcedseLinuxContext MustRunAs with an MCS labelProject allocated MCS labelRelabel failures on shared block storage at mount time
Namespace labelled PSA privilegedNamespace labelNo equivalent, OpenShift enforces privileged globallySynced from the highest SCC in useCopying the label changes nothing, SCC still decides

Two rows deserve extra attention because they behave differently from the rest. fsGroup and SELinux bite at runtime rather than at create time, which makes them worse to debug: your pod reaches Running, your readiness probe goes green, and the application dies on first write to a persistent volume with a bare permission denied. Stateful migration in Part 19 inherits that problem directly, so flag those workloads now rather than discovering them during a cutover window.

Five Rejection Patterns from a Real Estate

Read what restricted-v2 declares once, in full, rather than trusting anybody summarising it for you. Note that Users and Groups are both empty, because since OCP 4.11 access to restricted-v2 comes from a ClusterRole rather than from the fields on the SCC object itself. People new to OpenShift see those empty fields and conclude nobody has it. Everybody has it.

$ oc describe scc restricted-v2 Name: restricted-v2 Priority: <none> Access: Users: <none> Groups: <none> Settings: Allow Privileged: false Allow Privilege Escalation: false Default Add Capabilities: <none> Required Drop Capabilities: ALL Allowed Capabilities: NET_BIND_SERVICE Allowed Seccomp Profiles: runtime/default Allowed Volume Types: configMap,downwardAPI,emptyDir,ephemeral,persistentVolumeClaim,projected,secret Allow Host Network: false Allow Host Ports: false Allow Host PID: false Allow Host IPC: false Read Only Root Filesystem: false Run As User Strategy: MustRunAsRange SELinux Context Strategy: MustRunAs FSGroup Strategy: MustRunAs
SymptomError you will actually seeRoot causeFix that holds
Deployment created, zero podsprovider restricted-v2: .spec.securityContext.runAsUser: Invalid value: 0Explicit runAsUser 0 in the manifest or chart valuesDelete the field and let the project range default it
Pod reaches Running then CrashLoopBackOffexec /usr/bin/nc: operation not permittedBinary carries file capabilities or a setuid bit, blocked by allowPrivilegeEscalation falseRebuild the image without the capability bits
DaemonSet admits nothingprovider restricted-v2: .spec.volumes[0]: Invalid value: hostPath: hostPath volumes are not allowed to be usedNode level agent reading host log pathsReplace with the cluster logging stack, or a tightly scoped custom SCC
Pod rejected on capabilitiescapabilities.add: Invalid value: NET_ADMIN: capability may not be addedSidecar doing packet level work that NSX-T used to doDrop the sidecar, move the behaviour to a NetworkPolicy in Part 16
Pod green, application writes failopen /data/pgdata/PG_VERSION: permission deniedfsGroup outside the project supplemental group range, PVC owned by the wrong gidRemove the hardcoded fsGroup, let the SCC allocate it

Now look at where those ranges come from, because this is the production gotcha that catches teams twice.

$ oc describe ns payments | grep sa.scc openshift.io/sa.scc.mcs=s0:c26,c10 openshift.io/sa.scc.supplemental-groups=1000670000/10000 openshift.io/sa.scc.uid-range=1000670000/10000
Production gotcha: That uid range is allocated per project in creation order, and it is not stable across clusters. A uid of 1000670000 that validates in the payments project on your pilot cluster will be rejected in the payments project on production, because production handed that block to a project created earlier. Anything that hardcodes a uid to satisfy admission is building on sand. Leave runAsUser unset and let the range default it, every time.

Remediation Paths Compared by Cost and Risk

Six ways out of an admission rejection, and they are absolutely not equivalent. Effort and blast radius pull in opposite directions here, which is exactly why teams under schedule pressure pick the wrong one: the cheapest fix in minutes is also the one that costs most in audit findings six months later.

PathEffortBlast radiusAudit postureVerdict
Delete the offending manifest fieldMinutes per workloadOne workloadClean, stays on restricted-v2First choice, always try this before anything else
Rebuild the image for an arbitrary uidHours to days, plus team schedulingOne image, often many workloadsCleanCorrect fix when the image is genuinely at fault, budget for it
Grant nonroot-v2 to the service accountMinutesOne service accountAcceptable, still never rootReasonable when an image declares its own fixed non zero uid
Grant anyuid to the service accountMinutesRoot inside the containerWeak, flagged in every reviewTemporary only, with a dated exception and a ticket
Author a scoped custom SCCHalf a day plus reviewEverything you bind it toGood if the grant is narrow and documentedRare, for genuine platform agents such as storage or CNI
Grant privilegedMinutesWhole nodeFails audit outrightAvoid for application workloads, no exceptions

Here is what the top path looks like in practice, on the deployment that failed at the top of this article. Removing three lines and adding four was the entire fix.

# before, inherited from the TKGI manifest spec: securityContext: runAsUser: 0 fsGroup: 2000 # after, portable across every OpenShift project spec: securityContext: runAsNonRoot: true containers: – name: payments-api securityContext: allowPrivilegeEscalation: false capabilities: drop: [ALL] $ oc apply -f payments-api-deployment.yaml deployment.apps/payments-api configured $ oc get pod -n payments -o jsonpath='{.items[0].metadata.annotations.openshift.io/scc}{"n"}’ restricted-v2

Leave seccompProfile unset and the SCC defaults it to runtime/default for you, which keeps the manifest portable to clusters that predate your target. Setting it explicitly is fine on OCP 4.20 and harmless, but it is one more thing to keep correct in a chart, so I leave it out. When a grant genuinely is the answer, scope it to a named service account rather than the default one, and never to a group.

$ oc create sa payments-api -n payments serviceaccount/payments-api created $ oc adm policy add-scc-to-user nonroot-v2 -z payments-api -n payments clusterrole.rbac.authorization.k8s.io/system:openshift:scc:nonroot-v2 added: "payments-api" # prove which service accounts hold a permissive SCC, run this weekly $ oc get rolebindings,clusterrolebindings –all-namespaces -o json | grep -c scc:anyuid 1

Field Note from a Wave 1 Admission Sweep

We put nine namespaces into Wave 1 and I budgeted two days for admission fixes. It took eleven.

First mistake was entirely mine. On day one, with four workloads rejected and a stakeholder demo at 16:00, I ran add-scc-to-user anyuid against the default service account in all four namespaces. Everything came up green inside forty minutes and the demo went well. That was the problem. A fix that works in forty minutes under pressure becomes the pattern everyone copies, and two other engineers copied it out of the migration channel within a fortnight.

Three weeks later our security team ran their first review of the new cluster and produced a list of every service account bound to anyuid. Nine of them. Every one had to be unwound, and unwinding is far more expensive than fixing properly at the start, because by then applications had been promoted, images had been retagged, and two teams had built pipeline steps around the working state. Undoing those nine cost roughly six engineer days against maybe four hours if I had done it correctly on day one.

Second thing I got wrong was assuming the image was the hard part. For six of those nine workloads it was not. Image already ran fine as an arbitrary uid. Only reason it failed was a runAsUser: 0 line sitting in a Helm values file that somebody had added in 2021 to work around a TKGI storage permission problem that had not existed for years. Deleting one line fixed it. I had assumed rebuild, so I never checked, and I burned two days scheduling calls with teams who did not need to be on them.

Actual split once we finished: 31 workloads needed a manifest edit only, 14 needed a genuine image rebuild, 8 landed on nonroot-v2, 3 needed a scoped custom SCC for a storage agent, and 1 stayed on anyuid with a dated exception and a ticket. Median time for a manifest only fix was 43 minutes including a test deploy. Median for an image rebuild was closer to two days, almost all of it waiting on the owning team rather than on the build. Three months on, that single anyuid exception is still in place. They always are. Write the expiry date down anyway.

Fix Images First, Grant SCCs Last

Recommendation, unambiguous: treat every SCC grant as a defect you owe somebody, not a fix you shipped. Order of preference is delete the field, then rebuild the image, then nonroot-v2, then a narrowly scoped custom SCC. anyuid is a dated exception with a ticket attached and never a resolution. privileged is not on the list for application workloads at all.

Second recommendation, which costs nothing and saves weeks: stop trying to reproduce your TKGI security posture on OpenShift. There is nothing to reproduce. PSP was deleted by an upgrade you already did, PSA labels carry no authority on the target, and the only artifact that survives the move is the pod spec. Every hour spent authoring SCC objects before you have run a dry run is an hour spent on the wrong problem.

Worth saying plainly, because it comes up in every kickoff: this gap is not an argument against OpenShift. Kubernetes upstream moved in exactly this direction and your TKGI clusters would have faced the same reckoning eventually. Migration just makes you pay it in one visible block instead of spreading it across three upgrades. Broader trade offs for the platform choice sit in the complete guide, and if a permissive posture is genuinely non negotiable for a subset of workloads, that is one of the few honest reasons to look at the TKGI to VKS path for those specific applications instead.

Do this on Monday. Take your Wave 1 namespaces, run every workload spec through oc apply with server side dry run against a scratch OpenShift project, and record the first blocking reason for each one. You want the histogram from the top of this article for your own estate, because that number, not your wave list, is your real Wave 1 schedule. Part 8 leaves admission behind and takes on the other half of the portability problem, NSX-T and NCP against OVN-Kubernetes.

TKGI to OpenShift Series · Part 7 of 26
« Previous: Part 6  |  Guide  |  Next: Part 8 »

References

About The Author


Discover more from Journal of Intelligent Infrastructure

Subscribe to get the latest posts sent to your email.

Leave a Reply

Your email address will not be published. Required fields are marked *

Architect’s Toolkit

About the Author

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

Discover more from Journal of Intelligent Infrastructure

Subscribe now to keep reading and get access to the full archive.

Continue reading