Posted on July 30, 2026 by Sascha Grunert
CNCF projects highlighted in this post
The widely used container supply chain verification tools today operate at the Kubernetes API layer as admission webhooks (such as Kyverno, OPA Gatekeeper, and Sigstore Policy Controller). They intercept pod creation, check signatures and attestations, and either admit or reject the pod.
However, admission webhooks depend on explicit configuration and network connectivity. Misconfigured namespace selectors can silently skip verification. An admission webhook outage forces a choice between cluster lockup and silent bypass. Static pods and direct kubelet API access bypass admission webhooks entirely.
What if verification moved one layer lower—to the container runtime itself—where every container must pass through regardless of how it was scheduled?
That is what the Supply Chain NRI Plugin does. Originally proposed as an in-tree CRI-O feature, community feedback favored a plugin approach: it works with both CRI-O and containerd, ships on its own release cycle, and keeps the runtime’s critical path simple.
The Gap Below the API Server
Kubernetes documents these bypass paths explicitly. For example, static pods are managed directly by the kubelet. Even when their mirror pod fails admission, the container still runs. A static pod with an invalid namespace name becomes invisible to the API entirely. None of these containers pass through admission webhooks, so any image verification configured at that layer simply does not apply.
These are not theoretical risks. Techniques using mutating webhook manipulation to confuse validating webhooks have been disclosed, meaning verification can be silently bypassed.
There is also the availability side: OPA Gatekeeper’s documentation on failing closed describes a deadlock scenario where fail-closed webhooks prevent node recovery after a cluster-wide node deletion. Admission webhooks can both miss containers (a security gap) and block recovery (an availability gap).
Admission webhooks are a necessary first layer, but they are not sufficient as the only layer. Defense-in-depth means enforcing policy at a point that cannot be skipped, no matter how the container was scheduled.
Verification at the Runtime Level
The Node Resource Interface (NRI) is a plugin API supported by both CRI-O (1.28+) and containerd (1.7+, enabled by default since 2.0). It lets long-lived plugins (persistent daemons, not per-container one-shot hooks) hook into container lifecycle events. When a plugin subscribes to CreateContainer, the runtime calls it synchronously before the container starts. If the plugin returns an error, the container is rejected.
The Supply Chain NRI Plugin uses this exact hook point. On every CreateContainer call, it:
- Extracts the image reference and digest from runtime annotations.
- Fetches supply chain attestations from the OCI registry.
- Verifies them against a per-namespace policy.
- Allows or rejects the container.
This complements CRI-O’s existing containers-policy.json, which validates image signatures but does not cover attestation content like provenance, vulnerability status, or verification summaries.
+-------------------------------------------------------------------+
| Kubernetes API |
| (Bypassed by static pods, direct kubelet access, or outages) |
+-------------------------------------------------------------------+
|
v
+-------------------------------------------------------------------+
| Container Runtime |
| (CRI-O / containerd) |
+-------------------------------------------------------------------+
|
Synchronous NRI Hook Call
v
+-------------------------------------------------------------------+
| Supply Chain NRI Plugin |
| (Verifies SLSA, VEX, & VSA attestations before container start) |
+-------------------------------------------------------------------+
Admission webhooks can be skipped, but the NRI hook cannot be bypassed from the API layer because the runtime calls it synchronously for every container. This includes containers from pre-pulled or cached images. Verification happens at container creation, not at image pull, so even images pulled hours ago without an attestation check get verified before running.
Every container on the node passes through the runtime. There is no label to forget, no webhook network path to disrupt, and no gap between namespaces.
Note on Threat Model: This approach assumes node integrity. An attacker with root access can kill the plugin process, replace policy files on disk, or disable NRI in the runtime configuration entirely (e.g., nri: { disable: true } in containerd, --enable-nri=false in CRI-O). Runtime verification closes the gap between the API server and the container, but it does not defend against a fully compromised node.
What Gets Verified
The plugin verifies three types of attestations, each answering a distinct question about the container image:
SLSA (Supply-chain Levels for Software Artifacts) Provenance (How was this built?): Proves the image was produced by a specific builder, from specific source code, in a specific build environment. The plugin verifies the attestation signature, checks the builder identity against a trusted list, validates the source repository, and confirms the build type using the SLSA Provenance v1 predicate type.
VEX (Vulnerability Exploitability eXchange) (Are known vulnerabilities actually exploitable?): A VEX document states whether a known CVE is not_affected, under_investigation, fixed, or affected. Any affected status blocks the container in enforce mode. When multiple VEX documents exist, the most restrictive status wins. The implementation follows the OpenVEX v0.2.0 specification.
VSA (Verification Summary Attestation) (Has someone trusted already verified this?): A VSA comes from a trusted verification service that has already performed full SLSA and VEX verification. If the plugin finds a valid, passing VSA from a trusted verifier, it short-circuits all other checks and admits the image right away. This allows verification to scale across large clusters without repeating cryptographic work per node.
All three attestation types are discovered through a single OCI Referrers API call per image digest (with a cosign tag-based fallback). Attestation signatures are verified cryptographically using sigstore-go, supporting both keyless (Fulcio/OIDC) and key-based verification.
Verification Flow & Resilience
The flow starts when the container runtime calls the plugin’s CreateContainer hook. The plugin extracts the image digest, looks up the per-namespace policy, checks the local cache, and (on a cache miss) fetches attestations from the registry.
Design decisions embedded in the workflow:
- VSA Priority: Checked first. A failed VSA results in a hard rejection with no fallback.
- Parallel Execution: If no trusted VSA exists, SLSA provenance and VEX checks run in parallel.
Caching & Pre-warming: Results are cached per image digest and namespace with configurable TTLs. On startup, the plugin receives all running containers via NRI’s Synchronize callback and pre-warms the cache in the background, so restarts don’t cause a thundering herd of registry requests.
The plugin also deduplicates concurrent requests: when multiple containers reference the same image, only one verification runs and the rest share its result. Per-registry circuit breakers prevent cascading failures when a registry goes down. A concurrency semaphore limits parallel fetches, and retry with exponential backoff handles transient errors.
Configuration: two layers by design
An operator tuning cache TTL shouldn’t need to dig through trust root definitions in the same file. The plugin intentionally separates operational settings from security policy.
The operational config is a TOML file that controls the plugin’s behavior: verification mode, timeouts, cache TTL, metrics endpoint, circuit breaker settings.
verification = "enforce"
fetch_timeout = "30s"
fetch_failure_policy = "warn"
cache_ttl = "24h"
cache_failure_ttl = "5m"
policy_dir = "/etc/nri-supply-chain/policies"The policy files are JSON documents in a directory, one per Kubernetes namespace. default.json applies to all namespaces unless overridden. A file named production.json applies to the production namespace. Namespace policies can inherit from the default and override specific sections, or replace it entirely.
{
"trust": {
"issuers": ["https://token.actions.githubusercontent.com"],
"sanPatterns": ["https://github.com/saschagrunert/nri-supply-chain/**"],
"sources": ["github.com/saschagrunert/*"]
},
"slsa": { "missingPolicy": "deny" },
"vex": { "missingPolicy": "deny" }
}
Operational settings like cache size and timeouts change with infrastructure. Security policy, who you trust and what you require, changes with compliance requirements. Keeping them apart means they can be managed by different teams with different change cadences.
Try it out
The plugin binary supports a --verify-image flag to test a single image against the configured policy without connecting to NRI. The --validate flag checks config and policy files for errors without contacting any registry.
Using the policy from the previous section:
$ nri-supply-chain --config config.toml \
--verify-image ghcr.io/saschagrunert/nri-supply-chain:0.1.5
{
"image": "ghcr.io/saschagrunert/nri-supply-chain:0.1.5",
"digest": "sha256:1a8b39eeff74b8bb3e20c7f9fa773d4a9935241f7cc4e1217067c8186c2cee3c",
"namespace": "default",
"allowed": true,
"checkResults": [
{
"type": "slsa",
"passed": true,
"status": "pass",
"detail": "SLSA provenance verified"
},
{
"type": "vex",
"passed": true,
"status": "pass",
"detail": "VEX verification passed"
}
]
}
The output confirms two things: the image was built by a trusted builder from a trusted source (SLSA), and no known exploitable vulnerabilities were found (VEX). allowed: true means the image passes verification.
To enable VSA-accelerated verification, add a trust.verifiers entry to the policy:
{
"trust": {
"issuers": ["https://token.actions.githubusercontent.com"],
"sanPatterns": ["https://github.com/saschagrunert/nri-supply-chain/**"],
"sources": ["github.com/saschagrunert/*"],
"verifiers": [
{
"id": "https://github.com/saschagrunert/nri-supply-chain/.github/workflows/release.yml"
}
]
},
"slsa": { "missingPolicy": "deny" },
"vex": { "missingPolicy": "deny" }
}With this policy, the output changes:
{
"image": "ghcr.io/saschagrunert/nri-supply-chain:0.1.5",
"digest": "sha256:1a8b39eeff74b8bb3e20c7f9fa773d4a9935241f7cc4e1217067c8186c2cee3c",
"namespace": "default",
"allowed": true,
"reason": "VSA verification passed, skipping direct verification",
"checkResults": [
{
"type": "vsa",
"passed": true,
"status": "pass",
"detail": "VSA verification passed"
}
]
}The difference: instead of two separate checks (SLSA and VEX), a single VSA from a trusted verifier short-circuits the entire verification chain. The reason field confirms that direct verification was skipped. For production at scale, a CI/CD pipeline can verify images during build, attach a signed VSA, and push it alongside the image. Nodes then validate the VSA instead of fetching and verifying SLSA and VEX separately, reducing per-node registry traffic and verification latency.
Deployment
The plugin ships as a single static binary, a multi-arch container image, and DEB/RPM packages. There are three ways to deploy it:
As a Kubernetes DaemonSet, which is the most common path. The provided manifests include a security-hardened container with a NetworkPolicy restricting egress to DNS and HTTPS only. The container runs non-root with a read-only rootfs, no capabilities, a seccomp profile, and system-node-critical priority. Policies come from ConfigMap entries, volume-mounted into the container’s policy_dir.
As a systemd service for non-Kubernetes hosts or traditional Linux infrastructure. The unit file applies strict sandboxing: ProtectSystem=strict, memory limits, empty CapabilityBoundingSet, syscall filtering.
As a pre-installed NRI plugin (binary in /opt/nri/plugins/), auto-launched by the container runtime with no external process management. Best for immutable node images.
The plugin authenticates to registries using the standard Docker credential chain (~/.docker/config.json and configured credential helpers), so it works with any registry credentials available on the node.
A practical rollout
Supply chain verification is not something you flip on overnight. The plugin ships with verification = “disabled” by default, meaning it does nothing until explicitly configured. From there, it supports a gradual path from observation to enforcement.
Phase 1: Observe. Create a default.json policy with permissive settings (missingPolicy: “allow”) and set verification = “warn”. The plugin runs verification on every container but never blocks anything. Failed verifications get logged and counted. The nri_supply_chain_verification_total{result=”fail”} Prometheus metric counts how many containers would be blocked. The repo includes example alerting rules for circuit breaker trips, fetch error rates, and verification failures. This phase answers a question: what is our supply chain posture right now?
Phase 2: Tighten per namespace. Add namespace-specific policy files. Production namespaces get missingPolicy: “deny” for SLSA provenance, while dev namespaces stay permissive. Infrastructure images (pause containers, node-local images) can be excluded via glob patterns. This phase reveals which images in which namespaces lack attestations.
Phase 3: Enforce. Switch to verification = “enforce”. The plugin will log warnings at startup if permissive defaults are still in place, like fetch_failure_policy set to warn or any missingPolicy set to allow. It actively tells you what to tighten. Containers failing verification are now rejected, and the error shows up in pod events as a structured message:
supply chain verification failed: SLSA provenance: builder “X” not in trusted builders list
At this point, the runtime-level enforcement is active. Admission webhooks remain as the first layer, and the NRI plugin is the second. All configuration changes across every phase are hot-reloaded via SIGHUP and filesystem watching, with no pod restarts needed. The cache only gets cleared when policy-affecting fields actually change.
Failure modes and trade-offs
Every verification system has to decide what happens when things go wrong. The plugin makes these trade-offs explicit and configurable.
When a registry is unreachable, the fetch_failure_policy setting controls the behavior. The default is warn: containers pass, a warning gets logged, and the fetch error counter increments. This is explicitly fail-open, and the documentation calls it out. Setting it to deny trades availability for security: registry outages prevent new containers from starting.
The circuit breaker amplifies this trade-off. After consecutive failures to a registry host, all fetches to that host short-circuit for a configurable cooldown period. In warn mode, this means new, never-verified images from that registry bypass verification during the cooldown. Images verified before the outage are still served from cache. In deny mode, the circuit breaker prevents cascading timeouts from blocking the entire node.
Missing attestations are controlled per type. You might require SLSA provenance (missingPolicy: “deny”) but tolerate missing VEX documents (missingPolicy: “allow”) if your organization hasn’t adopted VEX in its build pipeline yet.
What’s next
The SLSA specification also defines a Source Track alongside the Build Track, covering how source revisions are created and protected. Unlike the Build Track, the Source Track is still maturing and tooling support is limited. It will be a natural addition once the ecosystem catches up.
Vulnerability scan result attestations and test result attestations are defined in the in-toto/cosign ecosystem but not yet widely produced. Once adoption grows, support will follow.
Notation/Notary v2 support alongside Sigstore would cover organizations using Microsoft’s signing infrastructure (common in Azure environments).
The plugin currently requires registry connectivity to fetch attestations at container creation time, which means air-gapped environments are not yet supported. Local attestation caching would let attestations be bundled alongside images in local registries or on-disk stores.
Further out, extending CRI error types upstream in Kubernetes would let the kubelet surface precise attestation failure reasons instead of a generic CreateContainerError. This requires changes to the CRI API itself and is a separate upstream effort.
The bottom line
Admission webhooks verify what the API server sees. An NRI plugin verifies what the container runtime actually executes. If your threat model includes admission webhook bypasses, static pods, or direct node access, runtime-level verification offers a robust defense-in-depth layer.
To learn more about the Node Resource Interface and container runtime ecosystem, check out the documentation on landscape.cncf.io.
0 Comments
Log in to join the conversation.No comments yet. Be the first to share your thoughts.