Three security scanners gave my cluster a clean bill of health. Then I deployed an attacker into a DaemonSet and discovered what static analysis actually misses.

The Setup

My homelab cluster is nothing exotic: a few Hyper-V VMs, a Raspberry Pi worker, and a Talos Linux control plane running the usual suspects: ArgoCD for GitOps, Cilium for CNI, Longhorn for storage. My workload is a typical cloud-native stack: a Go backend, a React frontend, PostgreSQL, the works.

I did exactly what the security playbooks tell you to do:

  • Trivy for container image scanning
  • kube-bench for CIS Kubernetes Benchmark compliance
  • Polaris for best-practice configuration checks

Every single tool reported green. I took a screenshot of the dashboard and felt safe.

Then something felt wrong.

A scanner audits what you declared. It does not test what happens when things go wrong. It is like a home inspector checking your door locks while ignoring the open window around the corner. I wanted a burglar on my side.

So I built one.

Introducing NEMESIS

NEMESIS is a Kubernetes-native purple team platform. It lives inside the cluster as a DaemonSet, not outside as a scheduled scan. It does not read manifests and compare them against registries. It executes attack scenarios from the same pod network your real workloads use.

The Stack

The platform splits along two languages for very specific reasons:

  • Python Controller — The orchestration brain. Schedules attack scenarios, correlates events, and emits findings. Python was the right call here: rapid scenario development and rich security libraries.
  • Go Attack Agent — A DaemonSet on every node. Executes the actual attack primitives: token abuse, lateral movement, network probing. Go gives me a single static binary, sub-second cold-start, and zero dependency hell inside a distroless container.

Attack Engine

Scenarios are not hard-coded exploits. They are a composable library of primitives:

  • Recon — Enumerate service accounts, roles, secrets, and network policies from inside the pod
  • Token Abuse — Mount and abuse service account tokens against the API server
  • Lateral Movement — Test east-west connectivity between namespaces with no external tooling
  • Exfil Probe — DNS and HTTP egress tests to measure actual network policy enforcement

eBPF Integration

Because the cluster runs Cilium, NEMESIS hooks into eBPF-based network telemetry. Every packet the attack agent sends is observable at the kernel layer. This means I do not just know that an attack path exists; I have flow-level proof of exactly which packet traversed which interface.

The Attack Chain That Scanners Missed

Here is one scenario, step by step. I anonymized nothing else because the configuration is common.

Step 1: Recon

The attack agent starts inside a pod in the default namespace. It enumerates service accounts:

curl -s https://kubernetes.default.svc/api/v1/namespaces/default/serviceaccounts \
  --header "Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)"

Enter fullscreen mode Exit fullscreen mode

It finds the default service account. Then it checks its bindings:

curl -s https://kubernetes.default.svc/apis/rbac.authorization.k8s.io/v1/namespaces/default/rolebindings

Enter fullscreen mode Exit fullscreen mode

Result: a ClusterRoleBinding named cluster-reader-default grants cluster-reader to the default SA across all namespaces.

Step 2: Token Abuse

With cluster-reader permissions, the agent queries every namespace for secrets:

curl -s https://kubernetes.default.svc/api/v1/secrets --all-namespaces

Enter fullscreen mode Exit fullscreen mode

It finds a ConfigMap in the backend namespace named db-config containing a PostgreSQL connection string with a hard-coded password.

What the scanner said: "ConfigMap db-config exists. No hardcoded secrets detected in source image."

What NEMESIS found: The ConfigMap is readable by any pod using the default service account, which is every pod in the cluster that does not explicitly override it.

Step 3: Lateral Movement

The agent lists pods across namespaces:

curl -s https://kubernetes.default.svc/api/v1/pods --all-namespaces

Enter fullscreen mode Exit fullscreen mode

It identifies the backend API pod in the backend namespace. It checks the NetworkPolicy for that namespace:

# backend-network-policy.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny
  namespace: backend
spec:
  podSelector: {}
  policyTypes:
  - Ingress

Enter fullscreen mode Exit fullscreen mode

What the scanner said: "NetworkPolicy exists. Default-deny posture implemented."

What NEMESIS found: The policy only specifies Ingress. There is no Egress policy. The backend pod can open outbound connections to anything.

Step 4: Data Exfil Test

The agent attempts a DNS lookup to an external domain controlled by the test harness:

nslookup exfil-test.nemesis.internal

Enter fullscreen mode Exit fullscreen mode

Cilium's eBPF probe captures the packet at the kernel socket layer. The DNS query reaches the cluster DNS, which forwards it upstream. No policy blocked it.

NEMESIS logs this as: "DNS exfiltration path confirmed — no egress filtering on namespace backend."

The Gap: What You Declare vs. What You Get

Out of 47 unique findings NEMESIS surfaced across three weeks of continuous testing, here is the breakdown versus what the traditional scanner suite caught:

Finding Category NEMESIS Scanner Suite Gap
Lateral movement paths 12 0 100%
Service account token abuse 9 1 89%
Network exfiltration (DNS/HTTP) 8 0 100%
Network policy bypass 6 0 100%
Runtime container escape 3 1 67%
Image CVEs 2 7 Covered by scanner
CIS misconfigurations 4 5 Partial overlap
Manifest best practices 3 6 Covered by scanner
Total 47 20 ~81% unique to NEMESIS

To be completely fair: the scanners caught every image vulnerability and most CIS misconfigurations. They are excellent at what they do. They just do not test runtime behavior.

Why This Changes How You Think About K8s Security

The industry has optimized for speed of detection. But detection speed means nothing if you are detecting the wrong things.

Scanners answer: "Is this configuration compliant?"

Purple teams answer: "What happens when someone already inside tries to move?"

Both questions matter. But in a world where supply chain attacks, compromised CI pipelines, and insider threats are the dominant risk models, the second question is the one that actually keeps you awake.

NEMESIS is not a replacement for scanning. It is an additional layer that validates your assumptions at runtime, continuously, from the same vantage point a real attacker would have.

What I Learned Building This

The Go-vs-Python split was not philosophical. It was operational. The agent needs to start in under a second when a node scales up, handle network timeouts gracefully, and compile to a single binary that fits in a scratch container. Go wins here without debate.

The controller needs to parse YAML attack definitions, talk to the Kubernetes API, and integrate with Cilium's Hubble for flow data. Python's ecosystem made this painless.

The hardest part was not the attack code. It was building the telemetry correlation: matching an eBPF flow event to a specific attack scenario step so you can say with certainty: "The DNS leak on step 4 of scenario backend-exfil-001 originated from pod backend-api-7d9f4b8c5-x2k9m at 14:23:07 UTC."

Without that correlation, purple team findings are just noise.

Where NEMESIS Goes Next

The current milestone focuses on Kubernetes, but the architecture is generic. The same DaemonSet pattern applies to:

  • Container runtime testing (Docker socket abuse, privileged escalation)
  • Cloud metadata service abuse (IMDSv2 bypass attempts on AWS/Azure nodes)
  • Storage layer attacks (Longhorn volume snapshot tampering, RWO bypass)

The goal is not to build a scanner with a different skin. The goal is to shift the security posture from "we read your configuration" to "we prove what an attacker can actually do."

Try It Yourself

If you want to see what your scanners are missing, the fastest path is deliberate:

  1. Spin up a throwaway pod in your cluster
  2. kubectl exec into it
  3. Run curl --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt -H "Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" https://kubernetes.default.svc/api/v1/namespaces/default/secrets
  4. See what comes back

If you see more than an empty list, you have found your first gap without writing a single line of code.

NEMESIS is an open-source Kubernetes-native purple team platform. If you want to follow the build, attack scenario library, or contribute, the repository is at github.com/beltagyy/nemesis.

Built with Go, Python, Cilium, and enough caffeine to make a kernel panic look calm.