Agent

How the unbounded-agent bootstraps a host into a Kubernetes worker node.

The unbounded-agent is a single binary that turns a Linux host into a Kubernetes worker node. It is the final convergence point for every provisioning path: regardless of how the host was delivered (SSH, cloud-init, PXE), the agent runs the same bootstrap sequence. Typically the agent is invoked by a controller (machina, metalman, or Karpenter), but it can also be run manually to bootstrap a machine and join it as a worker node.

How It Works

When unbounded-agent start is invoked, the agent runs through three phases in sequence:

  1. Host preparation (host) – installs required packages, configures kernel parameters for Kubernetes networking, and disables services that conflict with kubelet (Docker, swap).
  2. Rootfs preparation (rootfs) – detects host settings such as CPU architecture and GPU devices, creates a systemd-nspawn machine rootfs (from an OCI image), and downloads containerd, runc, CNI plugins, and Kubernetes binaries.
  3. Node start (node-start) – boots the nspawn machine and starts containerd and kubelet inside it. Kubelet performs TLS bootstrapping against the API server using the configured bootstrap token.

Retrying bootstrap and repairing the daemon

Initial bootstrap records installation ownership before changing the host. If a stage fails or the process is interrupted, rerun the same saved bootstrap script or invoke unbounded-agent start with the same original configuration.

Every stage runs again on every attempt, and each decides what it still has to do by looking at the host. Packages already present are not reinstalled. A rootfs a machine is already registered from is left in place rather than rebuilt. A running nspawn machine is left running. Node service configuration is rewritten, and the service is restarted only if that configuration actually changed. The nftables ruleset a running node depends on is not flushed.

This is why a retry is safe to run repeatedly, and why a host that was modified between attempts is repaired rather than skipped past: nothing is assumed from how far a previous attempt got.

The ownership record is /var/lib/unbounded/agent/install-state.json. It is an internal file, not a configuration input. Keep it intact when retrying. It records which phase the installation is in, never how much of it has been done, so it cannot fall out of step with the host. A different machine name, Kubernetes version, rootfs image, or API server endpoint is rejected and requires an explicit reset before a new initial installation. Credentials and artifact locations can be refreshed for a retry.

A retry that finds the node already running does not rewrite <machine>-applied-config.json, because that file records what the running node was built from and this attempt did not build it. Configuration that is allowed to change between attempts but only takes effect at node registration, node labels in particular, therefore reaches the node through the daemon’s ordinary drift repave rather than through the retry itself.

After completion, the same start invocation checks required daemon files, executable permissions, and enabled/active service state, and repairs the daemon when they are missing or stopped. It does not compare unit contents or overwrite working local unit customizations. This path uses the current applied configuration, including after an ordinary repave has switched from kube1 to kube2. It does not resolve the original node image or binary-download sources. The bootstrap shell still needs its agent download; to repair without that download, run an available agent executable directly with the original config:

sudo env UNBOUNDED_AGENT_CONFIG_FILE=/path/to/original-agent-config.json \
    /path/to/unbounded-agent start

The agent daemon does not run while an installation is unfinished. If it starts in that state, on a reboot or because an interrupted bootstrap had already enabled it, it logs a warning and exits with status 69. The unit reports inactive, not failed, and is not restarted; systemctl status unbounded-agent-daemon showing inactive (dead) together with that warning means the install still has to be completed, not that the daemon is broken. Run the bootstrap again and the daemon starts with it. Do not reset the host to clear this state.

Reset also completes on a host where bootstrap never got far enough to install the packages it inspects with, and on one whose ownership record cannot be parsed. Both are cases where refusing would leave nothing able to clear the record, so reset proceeds and removes it.

Bootstrap, reset, node lifecycle operations, and binary activation share an installation lock. Last-resort daemon rollback does not wait on these locks; reset stops the recovery unit before teardown. A busy lock is retryable, and MachineOperation handlers requeue instead of starting concurrent host mutations. Reset retains a resetting record when cleanup fails; correct the reported error and run reset again. Ownership is removed only after teardown and its filesystem synchronization barriers succeed. The lock file in /run remains and must not be deleted to force an operation through.

Recovery applies to initial installations that carry an ownership record. Existing installations without one retain ordinary daemon operations and reset support; start requires a clean host before creating new ownership. Interrupted ordinary repaves still use the existing repave model, without a persistent repave recovery operation.

Configuration

The agent reads a JSON config file whose path is set through the UNBOUNDED_AGENT_CONFIG_FILE environment variable. The manual-bootstrap command generates this config automatically, but it can also be authored by hand. A minimal example:

{
  "MachineName": "mysite-worker-01",
  "Cluster": {
    "CaCertBase64": "<base64-encoded CA certificate>",
    "ClusterDNS": "10.0.0.10",
    "Version": "1.33.1"
  },
  "Kubelet": {
    "ApiServer": "https://api.example.com:6443",
    "BootstrapToken": "abc123.0123456789abcdef",
    "Labels": {
      "unbounded-cloud.io/site": "mysite"
    },
    "RegisterWithTaints": []
  }
}

See the Configuration reference for the full field specification.

Running the Agent

What You’ll Do

Join an existing Ubuntu VM to your Kubernetes cluster as a worker node by manually running the unbounded-agent on the target host.

Prerequisites

Steps

  1. Generate and run the bootstrap script on the target VM:
kubectl unbounded machine manual-bootstrap my-node --site mysite | ssh user@host sudo bash
Note

To inspect the script before running it, save it to a file first:

kubectl unbounded machine manual-bootstrap my-node --site mysite > bootstrap.sh

Reusable bootstrap payloads (VMSS / Auto Scaling)

The NAME argument is optional. When you omit it, the generated payload leaves the machine name unset and the unbounded-agent resolves it on the host at startup. This lets a single payload be reused across many instances, such as an Azure VMSS or an AWS Auto Scaling group, where each instance derives its own name.

# Generate a reusable cloud-init document with no hard-coded machine name:
kubectl unbounded machine manual-bootstrap --site mysite --variant cloud-init > user-data.yaml

At startup the agent resolves the machine name in this order:

  1. The AGENT_MACHINE_NAME environment variable, if set on the host.
  2. The host hostname.

The resolved value is lowercased and validated as a Kubernetes node name (it becomes the Machine CR name), and the agent logs which source it used:

journalctl -u unbounded-agent | grep "resolved unbounded MachineName"

To pin a specific name on a host without editing the payload, export the environment variable before the agent runs, for example via cloud-init:

runcmd:
  - export AGENT_MACHINE_NAME=my-custom-node

Where the agent installs

The agent keeps its own host-side files under /opt/unbounded/agent: the daemon binaries and helper scripts in /opt/unbounded/agent/bin, and the LocalDNS network helper in /opt/unbounded/agent/libexec. Paths inside the nspawn machine are always relative to the machine directory, and /etc/unbounded/agent and /var/lib/unbounded are separate. /opt/unbounded/agent/bin is not on the default PATH, so run the agent on the host as /opt/unbounded/agent/bin/unbounded-agent.

/opt/unbounded/agent is reserved for the agent. The rest of /opt/unbounded is yours, for example for offline artifacts or a local OCI layout: the agent creates /opt/unbounded if it is missing, and otherwise never changes or removes it or anything else in it. Mount a volume at a subdirectory such as /opt/unbounded/artifacts rather than over /opt/unbounded, so it does not hide the agent.

The filesystem that holds /opt/unbounded/agent must allow running programs, so /opt must not be mounted noexec. Preflight fails where it is.

Releases before /opt/unbounded/agent installed these files under /usr/local. A host installed by one of them moves to /opt/unbounded/agent in two stages:

  1. Linked. The first command of a newer agent that changes the host links /opt/unbounded/agent to /usr/local, creating /opt/unbounded if needed. That happens when the daemon starts after an AgentUpgrade, when start or agent-upgrade runs, or when the nspawn pre-start hook runs. The files stay where they are, and every path the units name is unchanged. The older release is still the last-good binary, so the daemon can roll back to it, and an AgentUpgrade back to it works. If the link cannot be made, the pre-start hook leaves the machine’s config as it is and the machine still starts; the daemon refuses to start, and its journal says why.

  2. Moved. Once neither the current nor the last-good binary is from a release before /opt/unbounded/agent, the daemon copies the files into a real /opt/unbounded/agent, rewrites the units to use them, and restarts itself from there. The restarted daemon then removes the files from /usr/local. That is at the first daemon start after the AgentUpgrade that follows the older release out of the last-good slot. To move a host that is not due another upgrade, apply the release it already runs as an AgentUpgrade. A move that is interrupted is finished or started over at the next daemon start.

    Before any unit names the copy, the daemon runs it. A host whose /opt is mounted noexec, or where the copy does not run for another reason, stays linked: the daemon logs why, removes any copy it made, points the units back at /usr/local if a move that was interrupted had already pointed them at the copy, and tries again at its next start.

unbounded-agent reset removes the agent’s files from both locations at any stage, then /opt/unbounded/agent itself: the link the agent made, or the directory once nothing else is left in it.

Releases before /opt/unbounded/agent cannot run on a host whose agent is in a real /opt/unbounded/agent, whether it was installed there or moved there, and an AgentUpgrade to one is not supported. Nothing refuses it, and what happens depends on /usr/local:

  • Where it is read-only, that release’s daemon fails at startup, because it writes its nspawn lifecycle helper there. The agent rolls back to the last-good binary and the operation fails.
  • Where it is writable, the operation reports success, and the next AgentUpgrade fails, because that release looks for its binaries under /usr/local.

To recover, copy a newer release’s unbounded-agent binary to the host and run sudo ./unbounded-agent agent-upgrade with it. That leaves the older release as the last-good binary, so apply the newer release as an AgentUpgrade once more to replace it there. Alternatively, run sudo ./unbounded-agent reset with the newer binary and bootstrap the host again.

The agent refuses to run on a host that has an installation under both locations, or an installation under /usr/local beside an existing /opt/unbounded/agent directory, except while a move between them is under way. Run unbounded-agent reset first.

The install script also places the agent binary at /usr/local/bin/unbounded-agent where it can, because releases before /opt/unbounded/agent look for it there. Newer releases do not use it, and the daemon removes it once nothing it runs is under /usr/local: once the host is installed in a real /opt/unbounded/agent, or in a directory outside /usr/local that you linked /opt/unbounded/agent to.

Immutable hosts (read-only /usr)

Some images mount /usr read-only and provide no package manager, so there is no shell-based provisioning path at first boot. Azure Container Linux is one such image. /opt, and with it the agent’s files under /opt/unbounded/agent, is on the writable root filesystem there.

For these hosts, generate an Ignition config. The agent has to be a release that installs under /opt/unbounded/agent: earlier releases install under /usr/local, which is read-only on these hosts. Set VERSION to that release’s tag:

VERSION=vX.Y.Z
curl -fsSLO "https://github.com/Azure/unbounded/releases/download/${VERSION}/checksums.txt"
kubectl unbounded machine manual-bootstrap my-node --site mysite \
    --variant ignition \
    --agent-url "https://github.com/Azure/unbounded/releases/download/${VERSION}/unbounded-agent-linux-amd64" \
    --agent-sha256 "$(grep ' unbounded-agent-linux-amd64$' checksums.txt)" \
    > config.ign

Ignition declares state rather than running commands, so this variant cannot resolve a version, detect an architecture, or extract an archive at boot. It therefore requires --agent-url pointing at the bare agent binary rather than the release tarball, and --agent-sha256 to verify it. The digest for each release binary is published in checksums.txt.

The config writes the agent config and the binary, and enables unbounded-agent-bootstrap.service, which runs preflight and then start on every boot. On an installed host that does nothing unless the daemon needs repair. It retries a failed bootstrap with a delay that grows to five minutes on systemd 254 and later, and every 10 seconds on older versions. Follow it on the host with journalctl -u unbounded-agent-bootstrap.service. unbounded-agent reset stops, disables, and removes it.

config.ign carries the agent config, including the bootstrap token. Treat it as a secret, and deliver it only to the host it is for.

Customizing the agent download

By default the bootstrap script downloads the latest published unbounded-agent release directly from GitHub. This means new releases are picked up automatically without editing scripts or docs. The download can be customized when generating the bootstrap payload or at runtime via environment variables:

  • --agent-version / AGENT_VERSION: pin to a specific release tag instead of tracking latest. Example: --agent-version v0.0.10.
  • --agent-base-url / AGENT_BASE_URL: point at a self-hosted mirror of the release assets. The layout under the base URL must match the GitHub releases layout (<base>/latest/download/<asset> and <base>/download/<tag>/<asset>). Useful for air-gapped environments.
  • --agent-url / AGENT_URL: fully qualified download URL for the agent tarball. Overrides the version / base URL resolution entirely.

Examples:

# Pin the agent to a specific release:
kubectl unbounded machine manual-bootstrap my-node --site mysite \
    --agent-version v0.0.10 | ssh user@host sudo bash

# Self-host the release assets under an internal mirror:
kubectl unbounded machine manual-bootstrap my-node --site mysite \
    --agent-base-url https://releases.internal.example.com/unbounded \
    | ssh user@host sudo bash

The same overrides can also be exported on the target host before running a previously generated bootstrap script:

export AGENT_BASE_URL=https://releases.internal.example.com/unbounded
bash bootstrap.sh

The same agent-download flags are also accepted by kubectl unbounded machine register, which creates a Machine CRD. The machina controller then SSH’s into the host and exports the same AGENT_* environment variables when running the install script, so pinning an agent version or pointing at a mirror works uniformly across all three provisioning paths (manual-bootstrap, register, and the machina reconciler):

kubectl unbounded machine register \
    --site mysite \
    --host 10.0.0.5 \
    --ssh-username azureuser \
    --ssh-private-key ~/.ssh/id_ed25519 \
    --agent-version v0.0.10 \
    --agent-base-url https://releases.internal.example.com/unbounded

These values are persisted on the Machine object under spec.agent.{version,baseURL,url} so the controller reproduces the same override on every reconcile:

apiVersion: unbounded-kube.io/v1alpha3
kind: Machine
metadata:
  name: mysite-worker-01
spec:
  agent:
    version: v0.0.10
    baseURL: https://releases.internal.example.com/unbounded
  ssh:
    host: 10.0.0.5
    # ...

Mirroring the rootfs downloads (air-gapped deployments)

After installing the unbounded-agent, the agent downloads five additional binaries into the nspawn machine rootfs:

ArtifactUpstream default
kubelet / kubectl / kube-proxyhttps://dl.k8s.io
containerdhttps://github.com/containerd/containerd/releases/download
runchttps://github.com/opencontainers/runc/releases/download
CNI pluginshttps://github.com/containernetworking/plugins/releases/download
crictl (cri-tools)https://github.com/kubernetes-sigs/cri-tools/releases/download

Each default can be overridden at registration time. Every artifact exposes three flags, mirroring the --agent-* flags above:

  • --<artifact>-base-url: replaces the upstream host + path prefix. Mirrors must preserve the same file layout as the upstream project (e.g. <base>/v<version>/<asset>).
  • --<artifact>-url: full URL template; %s fmt placeholders are substituted with version / arch fields in the same order as the upstream default.
  • --<artifact>-version (or --kubernetes-binary-version): pin the artifact to a specific version independently of the cluster Kubernetes version. The default crictl minor-alignment behavior (Kubernetes vX.Y.Z -> cri-tools vX.Y.0) is preserved when no crictl version is set.

The flags are available on both manual-bootstrap and register, and the machina controller forwards them through the agent JSON config (Downloads block) so an air-gapped operator can register a node against an internal mirror without touching source:

kubectl unbounded machine register \
    --site mysite \
    --host 10.0.0.5 \
    --ssh-username azureuser \
    --ssh-private-key ~/.ssh/id_ed25519 \
    --agent-base-url      https://mirror.internal/unbounded \
    --kubernetes-base-url https://mirror.internal/k8s \
    --containerd-base-url https://mirror.internal/containerd/releases/download \
    --runc-base-url       https://mirror.internal/runc/releases/download \
    --cni-base-url        https://mirror.internal/plugins/releases/download \
    --crictl-base-url     https://mirror.internal/cri-tools/releases/download

The resulting Machine persists the overrides under spec.agent.downloads so the controller reproduces them on every reconcile:

apiVersion: unbounded-kube.io/v1alpha3
kind: Machine
metadata:
  name: mysite-worker-01
spec:
  agent:
    baseURL: https://mirror.internal/unbounded
    downloads:
      kubernetes:
        baseURL: https://mirror.internal/k8s
      containerd:
        baseURL: https://mirror.internal/containerd/releases/download
      runc:
        baseURL: https://mirror.internal/runc/releases/download
      cni:
        baseURL: https://mirror.internal/plugins/releases/download
      crictl:
        baseURL: https://mirror.internal/cri-tools/releases/download
  ssh:
    host: 10.0.0.5
    # ...

Leaving any artifact unset preserves the upstream default, so you can mirror a subset of the artifacts and let the rest fall back to the public CDN.

A successful run looks like this (timestamps shortened for readability):

[I] starting unbounded-agent version=dev commit=unknown
[I] [install-packages] started
[I] [install-packages] completed duration=7.35s status=ok
[I] [parallel(configure-os, configure-nftables, disable-docker, disable-swap)] started
[I] [disable-swap] started
[I] [configure-os] started
[I] [configure-nftables] started
[I] [disable-docker] started
[I] [disable-swap] completed status=ok duration=3.86ms
[I] [configure-os] completed duration=18.04ms status=ok
[I] [configure-nftables] completed duration=1.07s status=ok
[I] [disable-docker] completed status=ok duration=1.09s
[I] [parallel(configure-os, configure-nftables, disable-docker, disable-swap)] completed duration=1.09s status=ok
[I] [apply-attestation] started
[I] attestation not configured, skipping
[I] [apply-attestation] completed duration=12.81µs status=ok
[I] [ensure-nspawn-workspace] started
[I] [oci-download-rootfs] started
[I] pulling OCI image image=ghcr.io/azure/agent-ubuntu2404:... dest=/var/lib/machines/my-node
[I] OCI image extraction complete dest=/var/lib/machines/my-node
[I] [oci-download-rootfs] completed duration=2.10s status=ok
[I] [ensure-nspawn-workspace] completed duration=2.23s status=ok
[I] [parallel(download-kube-binaries, download-cri-binaries, download-cni-binaries, configure-os, disable-resolved)] started
[I] [download-kube-binaries] started
[I] [download-cri-binaries] started
[I] [download-cni-binaries] started
[I] [download-kube-binaries] completed status=ok duration=2.06s
[I] [download-cni-binaries] completed duration=3.21s status=ok
[I] [download-cri-binaries] completed duration=3.32s status=ok
[I] [parallel(download-kube-binaries, download-cri-binaries, download-cni-binaries, configure-os, disable-resolved)] completed duration=3.33s status=ok
[I] [parallel(configure-containerd, configure-kubelet)] started
[I] [configure-containerd] completed duration=5.57ms status=ok
[I] [configure-kubelet] completed duration=13.31ms status=ok
[I] [parallel(configure-containerd, configure-kubelet)] completed duration=13.46ms status=ok
[I] [start-nspawn-machine] started
[I] [start-nspawn-machine] completed duration=654.85ms status=ok
[I] [setup-nvidia] started
[I] NVIDIA runtime not enabled or no host libraries found, skipping
[I] [setup-nvidia] completed status=ok duration=18.40µs
[I] [start-containerd] started
[I] [start-containerd] completed status=ok duration=319.89ms
[I] [start-kubelet] started
[I] [start-kubelet] completed duration=180.11ms status=ok
  1. Verify the node has joined:
kubectl get nodes -o wide

NAME       STATUS   ROLES    AGE   VERSION   INTERNAL-IP      EXTERNAL-IP   OS-IMAGE               KERNEL-VERSION      CONTAINER-RUNTIME
my-node    Ready    <none>   20s   v1.33.1   192.168.100.10   <none>        Ubuntu 24.04.4 LTS     6.8.0-106-generic   containerd://2.0.4

GPU Support

When GPUs are detected on the host, the agent automatically exposes the host paths needed by Kubernetes device plugins:

  • For NVIDIA GPUs, bind-mounts GPU devices and driver libraries into the nspawn machine, generates a CDI spec, and registers the NVIDIA container runtime with containerd.
  • For AMD GPUs, bind-mounts /dev/kfd and DRM device nodes and exposes AMD sysfs paths read-only so the AMD Kubernetes device plugin can detect GPUs inside nspawn.

The agent does not deploy GPU device plugins. Install the vendor device plugin for Kubernetes resource advertisement. See the GPU reference for details.

Troubleshooting

Agent fails to start – Verify that the config file path is correct and contains valid JSON. Check that all required fields are present.

Node not joining the cluster – Ensure the host has HTTPS connectivity to the API server and that the bootstrap token has not expired. Inspect the kubelet logs inside the nspawn machine:

machinectl shell <machine-name> /bin/journalctl -u kubelet

GPU not detected – Confirm that the vendor driver is installed and loaded on the host. For NVIDIA, check /dev/nvidia*. For AMD, check /dev/kfd, /dev/dri/card*, and /dev/dri/renderD*.

See Also

Edit this page on GitHub