systemd-nspawn Isolation

How the unbounded-agent uses systemd-nspawn to isolate Kubernetes worker node components.

The unbounded-agent runs all Kubernetes worker node components inside a systemd-nspawn container. This provides lightweight OS-level isolation (separate filesystem, PID, and cgroup namespaces) while sharing the host kernel. The container runs a full systemd init tree so that kubelet, containerd, and their dependencies are managed as regular systemd services, decoupled from anything running on the host.

Why nspawn

Running kubelet directly on the host requires careful package management and risks conflicting with existing services. An nspawn container gives the agent a clean, reproducible environment:

• Filesystem isolation. Kubernetes binaries, configuration, and runtime state live inside the container’s rootfs and do not interfere with the host.
• Process isolation. The container has its own PID namespace. A full systemd instance (PID 1 inside the container) supervises containerd and kubelet with automatic restart policies.
• Cgroup isolation. The container gets its own cgroup subtree. Nested containers (pods) created by runc operate under cgroups v2 inside this subtree.
• Shared network namespace. The container shares the host’s network stack (VirtualEthernet=no), so kubelet uses the host’s routable IP directly and CNI plugins can create interfaces (WireGuard tunnels, overlay bridges) that are visible on the host.
• Simplified upgrades. Upgrading Kubernetes is a matter of replacing the container rootfs and restarting the nspawn machine. The host OS remains untouched, eliminating the need to coordinate package upgrades across the host and the node stack.

Container Rootfs

The rootfs is a plain directory tree at /var/lib/machines/<MachineName>. It is created during the rootfs bootstrap phase by one of two methods:

After the rootfs exists, the agent downloads Kubernetes binaries (kubelet, kubectl, kube-proxy), containerd, runc, and CNI plugins directly into the rootfs before the container boots. Configuration files (containerd config, kubelet service units, bootstrap kubeconfig, CA certificates, hostname, and resolv.conf) are also written into the rootfs at their standard absolute paths (e.g. /var/lib/machines/<MachineName>/etc/containerd/config.toml becomes /etc/containerd/config.toml inside the container).

OCI Images

The agent ships with two pre-built rootfs images based on Ubuntu 24.04 (Noble). The correct image is selected automatically based on whether NVIDIA GPUs are detected on the host.

The agent pins a specific image tag by default at build time. The OCIImage field in the agent config can override the full image reference for custom or pinned builds.

Image sources are maintained in the images/ directory of the repository.

Machine Configuration

The agent configures the nspawn machine to share the host network namespace (VirtualEthernet=no) so that kubelet uses the host’s routable IP directly and CNI plugins can manage interfaces visible on the host. It also makes /proc/sys/net writable inside the container so that CNI plugins and kube-proxy can set network sysctls, while the rest of /proc/sys stays read-only. Cgroups v2 is forced inside the container to ensure consistent behavior regardless of the systemd version in the rootfs.

The default systemd-nspawn@.service launcher may still show flags such as --network-veth. Those flags come from the systemd template, but the generated .nspawn file is written to the trusted /etc/systemd/nspawn/ directory and the template runs with --settings=override. As a result, the generated VirtualEthernet=no setting is authoritative and the machine shares the host network namespace. Host interfaces, host firewall and routing rules, and loopback listeners are therefore visible from inside the nspawn machine.

When NVIDIA GPUs are detected on the host, the agent automatically bind-mounts the GPU device nodes (e.g. /dev/nvidia0, /dev/nvidiactl) and the host’s driver libraries into the container, and grants the necessary cgroup device permissions. See NVIDIA GPU Support for the full GPU pipeline.

When the KVM character device (/dev/kvm) is present on the host, the agent automatically bind-mounts it into the container so that workloads inside the container can use hardware virtualisation (e.g. QEMU/KVM virtual machines).

The configuration is written to two files on the host before the machine boots:

Customization points

System call filter

The SystemCallFilter=@keyring bpf perf_event_open setting keeps nspawn’s default syscall filtering, but explicitly allows syscalls used by the worker stack:

• @keyring allows the kernel keyring syscall group that containerd uses for snapshotter and container setup operations.
• bpf allows eBPF program and map operations used by runc, cgroup handling, and eBPF CNIs.
• perf_event_open allows eBPF CNIs such as Cilium to create per-CPU perf ring buffers for datapath events. If nspawn blocks it, Cilium startup fails when creating those perf rings.

What Runs Inside the Container

The nspawn container hosts the complete Kubernetes worker node stack:

• systemd (PID 1 inside the container)
• containerd (container runtime)
• runc (OCI runtime)
• kubelet (Kubernetes node agent)
• kubectl, kube-proxy
• CNI plugins (under /opt/cni/bin)
• All Kubernetes pod containers (managed by containerd and runc)

The unbounded-agent itself and host-side management tools such as machinectl and systemctl remain on the host. machinectl is used to start, inspect, and access the machine, while the lifecycle of the backing systemd-nspawn@<MachineName>.service is managed via systemctl. NVIDIA kernel drivers and userspace libraries also stay on the host and are forwarded into the container via bind-mounts.

Lifecycle

Startup

The agent’s three-phase bootstrap drives the nspawn lifecycle:

1. Host preparation. Installs the systemd-container package (provides systemd-nspawn and machinectl), sets kernel sysctl values that the container cannot write (because /proc/sys is read-only), and installs a nftables-flush.service oneshot that clears stale firewall rules before any systemd-nspawn@.service starts.

2. Rootfs preparation. Creates the rootfs, writes the .nspawn config and service override, downloads Kubernetes and container runtime binaries, and configures the OS inside the rootfs (hostname, DNS, kernel modules).

3. Node start. Starts the nspawn machine, polls until it is responsive, then enables containerd and kubelet inside it.

Networking

The container operates in the host’s network namespace (VirtualEthernet=no):

• No virtual ethernet pair, bridge, or NAT is created.
• Kubelet binds to 0.0.0.0 and uses the host’s routable IP.
• CNI plugins create network interfaces (WireGuard, VXLAN, overlay bridges) that are visible on both the host and inside the container.
• Each nspawn machine gets a private bpffs mount at /sys/fs/bpf, backed by /run/bpffs/<MachineName> on the host. This isolates pinned BPF object paths between alternating nspawn machines, but it does not by itself detach host-visible TC/XDP/cgroup programs or remove CNI-created network devices.
• /proc/sys/net is writable inside the container so that CNI plugins and kube-proxy can configure network sysctls.
• Host-level sysctl values (net.ipv4.ip_forward, net.bridge.bridge-nf-call-iptables, etc.) are pre-set on the host because the rest of /proc/sys is read-only.
• DNS resolution uses a static copy of the host’s resolv.conf. systemd-resolved is masked inside the container to prevent it from overwriting the file. The host’s stub resolver at 127.0.0.53 is reachable because the network namespace is shared.

Key Paths

See Also

• Agent Configuration: JSON config file specification including MachineName and OCIImage.
• NVIDIA GPU Support: How GPU devices and libraries are forwarded into the nspawn container.
• Agent Guide: End-to-end walkthrough of the three-phase bootstrap.
• Agent Operations: Upgrading the agent and resetting hosts.