cgroup-based security in kubernetes

containers in a kubernetes cluster are linux processes sharing the same kernel, hardware, and pid space as everything else on the node. cgroups are the kernel mechanism that prevents one container from consuming all available memory and destabilizing the node.

cgroups are well understood in principle but frequently misconfigured or left at defaults. missing or incorrect cgroup configuration is behind many unexpected node failures.

the short version

cgroups - control groups - place processes into groups and cap what resources each group can use: cpu, memory, disk i/o, pid count. when kubernetes runs a container, it creates a cgroup for it and translates the pod spec’s resources.requests and resources.limits into kernel constraints.

set limits.memory: "512Mi" and the kernel oom-kills the process if it crosses that boundary. set limits.cpu: "2" and the scheduler throttles it after 200ms per 100ms period. these are hard enforcement at the kernel level.

the problem is what happens when they are not set.

what goes wrong without limits

a container without memory limits has no memory.max in its cgroup. it can allocate freely until the node runs out of memory entirely. at that point the kernel’s oom killer picks something to terminate - and it might pick kubelet. once kubelet is gone, the node is unmanageable from the control plane.

a common scenario: a memory leak in a sidecar container running besteffort because the resource block was omitted. it consumes 90% of node memory over several hours. when the oom killer fires, it terminates the container runtime instead of the leaking process, taking down every pod on the node.

the same applies to pids. kubelet’s --pod-max-pids defaults to unlimited. a fork bomb - intentional or accidental - fills the pid table, preventing any new process from spawning on the node. health checks fail, graceful shutdowns fail, and manual intervention at the node level becomes necessary.

qos classes and the eviction order

kubernetes groups pods into three quality-of-service classes based on their resource configuration, and this directly determines which cgroup subtree they land in:

/sys/fs/cgroup/kubepods.slice/
├── kubepods-guaranteed.slice/
├── kubepods-burstable.slice/
└── kubepods-besteffort.slice/

guaranteed means requests equal limits for both cpu and memory. the kernel protects these pods most aggressively - they’re last to be evicted, and their memory is protected from reclaim via memory.min.

besteffort means no resource spec at all. these pods are evicted first under memory pressure. since they have no memory.max, they can also be the source of that pressure - consuming unbounded memory while having no eviction protection.

running besteffort pods alongside critical workloads means an unbounded container can trigger node-level memory pressure that affects everything on that node.

hardening

the relevant kubelet configuration:

cgroupDriver: systemd
enforceNodeAllocatable: [pods, system-reserved, kube-reserved]
systemReserved:
  cpu: "500m"
  memory: "1Gi"
kubeReserved:
  cpu: "500m"
  memory: "1Gi"
podPidsLimit: 4096

enforceNodeAllocatable is frequently overlooked. without it, there’s no cgroup boundary protecting kubelet and the container runtime from pod resource pressure. systemReserved and kubeReserved carve out space that pods can never touch. at the namespace level, a LimitRange provides default limits:

apiVersion: v1
kind: LimitRange
metadata:
  name: default-limits
spec:
  limits:
    - default:
        cpu: "1"
        memory: "512Mi"
      defaultRequest:
        cpu: "100m"
        memory: "128Mi"
      type: Container

this catches every container that ships without a resource block. it won’t fix bad limits, but it eliminates the “no limits at all” failure mode.

for monitoring, cgroup v2’s pressure stall information - psi - provides early warning. cadvisor exposes it as prometheus metrics. a rising some on cpu means workloads are being throttled. non-zero full on memory means the kernel is actively reclaiming pages from that cgroup. these metrics surface before failures occur, providing time to respond.

what cgroups don’t cover

cgroups handle resource exhaustion. they don’t prevent kernel exploits, network scanning, or unauthorized data access. those require namespaces, seccomp profiles, network policies, and pod security standards.

that said, resource exhaustion remains the most common way a container destabilizes a node. proper cgroup configuration eliminates this class of failure.

references

[1] linux kernel documentation. “control group v2.”
docs.kernel.org/admin-guide/cgroup-v2.html

[2] kubernetes documentation. “managing resources for containers.”
kubernetes.io/docs/concepts/configuration/manage-resources-containers

[3] kubernetes documentation. “reserve compute resources for system daemons.”
kubernetes.io/docs/tasks/administer-cluster/reserve-compute-resources

[4] kubernetes documentation. “pod quality of service classes.”
kubernetes.io/docs/concepts/workloads/pods/pod-qos

[5] kubernetes documentation. “limit ranges.”
kubernetes.io/docs/concepts/policy/limit-range

[6] kubernetes documentation. “process id limits and reservations.”
kubernetes.io/docs/concepts/policy/pid-limiting