Configuration

All flags, environment variables, ConfigMap settings, and tuning guidance for unbounded-net.

This document describes all configuration options for unbounded-net components. For a conceptual introduction, see Networking Concepts.

Runtime Configuration

Both the controller and node agent load runtime settings from a shared YAML file mounted from the unbounded-net-config ConfigMap.

• Default path: /etc/unbounded-net/config.yaml
• Override: --config-file=<path>
• Startup behavior: fail-fast if the config file is missing or invalid.
• CLI flags still work as explicit overrides when set.

Summary publication and explicit details

Routine node publications contain overview metadata and aggregate counts, not peer lists, route entries, or BPF maps. Detailed data is collected for an explicit single-node request and kept temporarily. Changing these settings requires restarting the affected controller or node-agent pods.

The cache lifetime starts when actual details arrive, not on summary updates or reads. Continuous legacy full publications refresh it; on-demand duplicate or late replies do not. The request timeout covers all delivery attempts together.

Requests are coalesced per node, but simultaneous requests for different nodes start independent dispatch workers. Deadline and TTL cleanup bound how long they remain, not peak concurrency or memory during a burst. The controller’s background aggregate-pull concurrency limit does not apply to these explicit requests.

Upgrade controllers and their dashboard/CLI consumers before restarting agents in summary mode. An incompatible controller is reported explicitly; agents do not silently resume full streaming. For node-publishing rollback, set node.statusDetailMode: full and restart agents. This restores full/delta publication, not detailed bulk exports or the retired connectivity views.

Bulk APIs and global broadcasts remain summary-only in either mode. With no active capable node WebSocket, an explicit request tries HTTP pull first, regardless of the background-pull toggle. Failed pulls leave a command for the next authenticated status POST response. Both publishers may remain disabled: local HTTP diagnostics still work, but a failed pull has no polling fallback until outbound POST publication resumes.

Requests and details are leader-local and in memory. Restart or leadership change may require an explicit retry. Releasing expired references permits garbage collection but does not guarantee an immediate RSS decrease.

Config Structure

common:
  azureTenantId: ""           # Only for Azure Portal links in the UI
  apiserverURL: ""            # Override API server URL (empty = in-cluster)
  logLevel: 2                 # klog verbosity (0-10), watched for live changes

controller:
  healthPort: 9999
  nodeAgentHealthPort: 9998
  informerResyncPeriod: 300s
  statusStaleThreshold: 40s
  registerAggregatedAPIServer: true
  managedKubeProxy:
    enabled: true
    image: ""
  leaderElection:
    enabled: true
    leaseDuration: 30s
    renewDeadline: 15s
    retryPeriod: 10s

node:
  cniConfDir: /host/etc/cni/net.d
  cniConfFile: 10-unbounded.conflist
  bridgeName: cbr0
  wireGuardDir: /host/etc/wireguard
  wireGuardPort: 51820
  mtu: 0
  healthPort: 9998
  tunnelDataplaneMapSize: 16384
  tunnelIPFamily: IPv4
  preferredPrivateEncap: GENEVE
  preferredPublicEncap: WireGuard
  genevePort: 6081
  geneveVni: 1
  vxlanPort: 4789

Controller Configuration

Managed kube-proxy

The controller creates a kube-proxy DaemonSet for each Site whose nodes are not covered by the provider-managed kube-proxy. Each DaemonSet uses the Site’s pod CIDR and runs only on nodes labeled for that Site and for unbounded-managed kube-proxy.

The image setting defaults to an empty string. When it is empty, the controller reuses the image from the kube-system/kube-proxy DaemonSet. If that DaemonSet does not exist, it uses registry.k8s.io/kube-proxy:<server version>.

Set controller.managedKubeProxy.image to override the image for all unbounded-managed kube-proxy DaemonSets, for example when pulling kube-proxy through a private registry. Set controller.managedKubeProxy.enabled to false to disable their creation.

Leader Election

The node mutating webhook assigns only site labels, including to nodes that already have pod CIDRs. It never allocates pod CIDRs. Nodes without a matching site are admitted unchanged; reconciliation handles nodes whose internal IPs appear later.

With leader election enabled, the controller allocates pod CIDRs during node sync only while it has confirmed it holds the lease: until the renew deadline has elapsed since the start of its last successful lease write. A leader whose renewals stop succeeding stops allocating before another replica can acquire the lease. The lease holder assigns CIDRs after node creation during node sync.

New assignment allocators are seeded before they become available to node workers. Seeding preserves existing and unconfirmed allocations, including blocks of a different size that overlap candidate CIDRs. A pending allocation can move to a different allocator only when its IP families, block sizes, and pool containment match. Otherwise, reconciliation waits until a changed Node UID or resourceVersion proves the previous patch cannot apply before allocating replacement CIDRs. Successful patches remain reserved until the Node informer observes their CIDRs, even if the original assignment is removed in the meantime. Reservations copied into new allocators are tracked and released from every recipient when abandoned, unless another Node or pending assignment still owns the CIDR.

Health and Monitoring

Logging

Node Agent Configuration

Large-cluster neighbor tables

The node agent automatically sizes the IPv4 and IPv6 neighbor tables from the mesh endpoints it uses. No ConfigMap setting or command-line flag is required. Configured tunnel endpoints and currently learned WireGuard endpoints are included; a learned-only peer cannot be counted until its endpoint is known. Each reconciliation resolves the endpoints’ routes and counts unique next-hop IP/interface pairs separately for each address family. Directly connected endpoints each need a neighbor entry; endpoints reached through the same router on the same interface share one. Live multipath next hops are included, while local routes, loopback, NOARP devices, and managed tunnel devices are excluded.

The agent raises these thresholds when needed:

Existing higher values are never reduced. Higher lower thresholds also act as floors for the upper thresholds to keep them ordered. The agent raises the hard limit first. Peer changes and periodic reconciliation recheck the routes even when mesh membership is unchanged. Removing peers or switching to shared routers never lowers existing values. A family with no resolved neighbor next hops is left untouched, without any sysctl reads or writes.

An unexpected route lookup error, or a required sysctl that cannot be read, parsed, or raised, fails that reconciliation with an explicit error and is retried on a subsequent reconciliation. Unreachable endpoints are skipped. On hosts without the IPv6 neighbor table (net.ipv6.neigh.default.gc_thresh1 is absent), the agent logs that IPv6 tuning is skipped; permission errors, invalid values, and other missing thresholds still fail reconciliation.

Despite the default path, Linux implements these as table-wide limits shared by all interfaces, including subsequently created ones. There is no neigh/all/gc_thresh* or per-interface gc_thresh* to set. IPv6 has a separate neighbor table sized from its own next-hop count. See the kernel IP sysctl reference and neighbor sysctl registration.

This is a minimum for mesh traffic, not a complete host-capacity estimate. Neighbor entries can also include local pods and unrelated directly connected hosts, which are not counted here. Existing larger operator-set values are preserved. Sizing is reapplied after an agent restart or host reboot; the agent does not write host /etc/sysctl.d files or restore lower values at shutdown.

Neighbor table exhaustion warnings

Before tuning, each reconciliation checks the host network namespace’s actual IPv4 and IPv6 neighbor occupancy against that family’s current gc_thresh3. Permanent, externally learned, externally validated, and loopback entries are excluded because they do not count toward the kernel’s GC limit. Reaching or exceeding the limit produces a neighborTableFullIPv4 or neighborTableFullIPv6 warning: forced garbage collection may be needed before new neighbors can be allocated.

The agent also watches per-CPU table_fulls counters in /proc/net/stat/arp_cache and /proc/net/stat/ndisc_cache. Increases indicate actual allocation failures at the GC hard limit and produce neighborTableOverflowIPv4 or neighborTableOverflowIPv6 warnings. These kernel counters also catch short-lived exhaustion between samples and failures caused by demand in other network namespaces. The initial sample establishes a baseline; historical failures are not repeatedly reported as new events.

Warnings are logged and sent to the controller in both full and summary node status. An observed pressure or overflow warning remains visible for one minute after its last observation, even if automatic tuning raises the limit in the same cycle. Observation failures are reported separately as neighborTableCheckFailed, not as healthy or exhausted tables. Missing IPv6 thresholds and unavailable proc counter files are skipped.

Other scale-sensitive sysctls

These require workload measurements, not a universal multiplier of node count, so automatic neighbor sizing does not change them:

Fragment reassembly thresholds are memory limits in bytes, not fragment or node counts. The corresponding retention timeouts are net.ipv4.ipfrag_time and net.ipv6.ip6frag_time, in seconds. Longer timeouts retain incomplete datagrams longer and can increase memory pressure. Inspect IpReasmFails/IpReasmTimeout and Ip6ReasmFails/Ip6ReasmTimeout when diagnosing reassembly loss. net.ipv4.ipfrag_low_thresh is obsolete on modern kernels; it is not an additional capacity knob to increase.

Do not tune net.ipv4.route.max_size: Linux no longer uses the IPv4 route cache. IPv6 route.max_size is also deprecated on modern kernels. Do not adjust neigh.default.gc_interval, which is unused, to address table overflow.

References: IP sysctls, network core sysctls, and conntrack sysctls.

General

CNI Configuration

MTU Guidance

The node.mtu setting caps the automatically selected MTU on tunnel and bridge interfaces. Zero enables automatic selection. Encapsulation overhead:

The node agent resolves the route to each peer and subtracts the selected encapsulation overhead. For example, WireGuard over a 1500-byte route uses 1420.

Behavior:

• Each peer’s effective MTU is the lowest applicable configured cap, CRD tunnelMTU, and route MTU after encapsulation overhead.
• Forwarding-path scopes constrain a node. Direct mesh peers use both endpoint Sites and their enabled SitePeering. Gateway paths use assignments targeting the worker’s own Site, the selected GatewayPool, and GatewayPoolPeerings present in an advertised transit path.
• Direct-mesh components also share the lowest configured Site or enabled mesh-peering MTU. If A meshes with B at 1200 and C at 1400, A’s shared pod interfaces use 1200, so C must also use 1200 when sending to A. This propagates an interface constraint, not B-C routes. Disabled and meshNodes: false peerings, missing Sites, and disconnected components do not contribute. Gateway assignments and pool peerings do not join direct-mesh components.
• The CNI MTU is the lowest effective MTU among actual mesh and gateway peers and the configured direct-mesh component cap.
• At startup and whenever the effective CNI MTU changes, the node agent updates the CNI bridge and both endpoints of existing pod veth pairs.
• Forwarded TCP SYN packets on every interface are MSS-clamped to the site-wide fabric MTU (MTU - 40 for IPv4 and MTU - 60 for IPv6).
• A value of 0 enables route-specific automatic MTU selection.

WireGuard

GENEVE

VXLAN

The narrow source port range (32 ports) limits distinct flows from VMs, helping avoid flow table limits on cloud platforms (e.g., Azure).

IPIP

Tunnel interface names

The three shared tunnel device names (geneve0, vxlan0, ipip0) and the per-port WireGuard interface prefix (default wg) are configurable via the flags above and via the matching YAML fields under node: (geneveInterfaceName, vxlanInterfaceName, ipipInterfaceName, wireGuardInterfacePrefix). Operators rarely need to change them.

Constraints, enforced at startup:

• Shared tunnel names: non-empty, at most 15 bytes (Linux IFNAMSIZ limit), pairwise distinct, must not equal unbounded0.
• WireGuard prefix: non-empty, at most 10 bytes (so prefix + 5-digit UDP port fits in IFNAMSIZ), no /, must not equal unbounded0.
• unbounded0 (the agent’s eBPF dummy device) is not configurable.

When the names are changed, the agent automatically removes previous-run devices it can confidently identify, scoped narrowly so other CNIs (e.g. Cilium) running flow-based GENEVE / VXLAN on different UDP ports are not touched. IPIP is not auto-cleaned: remove a renamed-away IPIP device manually with ip link del <old-name>.

Tunnel Dataplane

The node agent uses an eBPF dataplane: BPF LPM tries on the unbounded0 interface drive per-destination tunnel redirection.

Tunnel Protocol Selection

The tunnelProtocol field is available on all scope CRDs:

When Auto, links using external IPs use WireGuard; internal-only links use the preferred private encap (default GENEVE). The security-wins rule ensures WireGuard is used if any scope explicitly requests it.

See Routing Flows for the full protocol selection algorithm.

Status Push

Health Check (UDP Probes)

Health check sessions are automatically created for all routes with nexthops. Route metric adjustment on failure provides fast failover.

Firewall Requirements

Resource Requirements

Controller

resources:
  requests: { cpu: 10m, memory: 64Mi }
  limits:   { cpu: 100m, memory: 128Mi }

Node Agent

resources:
  requests: { cpu: 10m, memory: 32Mi }
  limits:   { cpu: 100m, memory: 64Mi }

Scaling

Environment Variables

Controller

Node Agent

Tuning Recommendations

High Availability

replicas: 2
args:
  - --leader-elect=true
  - --leader-elect-lease-duration=30s
  - --leader-elect-renew-deadline=15s

Large Clusters (1000+ nodes)

# Controller
args:
  - --informer-resync-period=600s

# Node agent
args:
  - --informer-resync-period=7200s

Next Steps

• Custom Resources – CRD specifications.
• Routing Flows – Packet-level routing details.
• Operations – Deployment, monitoring, and troubleshooting.