Runtime and node reference

Reference for the parts of a self-hosted install that run agents. How much cluster to plan is on Sizing and requirements.

Agent workspaces

Each agent whose harness is xpander and that uses a workspace gets its own pod, created on demand and reclaimed when idle. Size for concurrently active workspaces, not total agents.

Resource	Value
CPU	250m request, 2000m limit
Memory	256 Mi request, 3 Gi limit
Storage	5 GiB persistent volume per agent
Idle shutdown	5 minutes without activity (configurable via `SANDBOX_IDLE_TIMEOUT` on the Agent Controller)
Maximum lifetime	3 hours, then the pod is recycled

Idle workspaces use ~50 Mi. An active one uses ~0.5-1 vCPU and up to 1 GiB, bursting to its limits. The volume persists after the pod is reclaimed, so plan 5 GiB per workspace-using agent, not per pod.

The workspace image is ~1 GiB and is pulled on the first start on each node. Budget ~2 GiB of node disk for the image cache.

The executor fleet

The fleet is a fleet-executor StatefulSet. Each replica is one PodSecurity baseline pod over one XFS volume: no privileged container, no host namespaces, no user namespaces and no node preparation. The pod holds the executor container, which claims turns, plus one slot-<i> container per turn slot (fleet.turnsMax). Each turn runs in its own slot container, as the agent’s own uid. It has its own processes, /proc, /tmp and memory and CPU limit, behind a seccomp profile, no_new_privs and Landlock. An agent’s directory lives on exactly one executor’s volume, and its turns always run there.

The fleet is always rendered by both charts (there is no enable flag). It starts warm with one executor (fleet.replicas: 1 on Hybrid, global.fleet.replicas: 1 on Air-Gapped). Harness agents (Claude Code, Codex, OpenCode) run only on the executors, never as a pod per agent. So the chart always keeps at least one: replicas: 0 (or a blank value) renders one executor. The one case that renders zero is a live cluster whose executor volume can never bind (see Idle guard). Agent turns then wait in the queue until it is fixed.

	Executor fleet
What an agent is	A directory (its home, CLI state, working tree) on the executor’s XFS volume, under a 20 GiB quota
What runs a turn	A free slot container on the executor that holds the agent’s directory, as the agent’s own uid, limited by Landlock (unless `fleet.landlock.enabled: false`) and the seccomp profile
Cost of an idle agent	Disk only; no pod, no requests
Cost of a new agent	`mkdir` on the first turn, seconds
Scaling axis	Concurrent turns, never the number of agents
Model credentials	None in the pod. Each turn calls the in-cluster LLM Gateway directly, with the run’s own short-lived token

Per executor

Defaults per executor pod: 2 / 8 vCPU and 8 / 28 GiB (request / limit), 12 slot containers, a 500 GiB XFS volume.

Executor resources, volume, grace and placement

Resource	Self-hosted chart default
Executor CPU request / limit	2 / 8 vCPU (`fleet.resources`)
Executor memory request / limit	8 / 28 GiB
Slot containers	One per slot (`fleet.turnsMax`, default 12); each requests 10m CPU / 64 Mi and is limited to the turn ceiling (`fleet.turn`, 4 vCPU / 8 GiB). Changing `turnsMax` rolls the executor pods
Volume	500 GiB, XFS, `prjquota`, gp3 encrypted, RWO
Per-agent disk quota (`quotaPerAgent`)	20 GiB (XFS project quota, hard; a ceiling, not a reservation)
`/dev/shm` (memory-backed, for Chromium)	4 GiB
Package cache (`emptyDir`, node disk)	20 GiB
Termination grace	7500 s (`fleet.executor.terminationGraceSeconds`), a ceiling: the executor first drains running turns for up to 3600 s (`fleet.executor.drainSeconds`)
Probes	readiness 10 s x 6, startup 5 s x 36 (3 min)
Placement	One executor per node (required anti-affinity); `fleet.nodeSelector` / `fleet.tolerations`

Size the executor’s memory against turnsMax. Each slot container is limited to the turn ceiling (8 GiB by default), and the node must back them. If the turns together use more memory than the node has, the kernel OOM-kills one. The chart default (12 slots, executor limit 28 GiB) is a conservative starting footprint. It assumes typical turns stay well below their ceiling. For a production fleet, either use memory-rich nodes sized toward turnsMax x 8 GiB or lower turnsMax.

Fleet values (Hybrid chart)

# Hybrid (xpander chart); on Air-Gapped the same keys live under global.fleet
fleet:
  replicas: 1                       # 0 or blank also renders 1
  turnsMax: 12
  turn: { memory: "8Gi", cpu: "4", pids: 1024 }
  landlock:
    enabled: true                   # false on nodes without Landlock
  executor:
    drainSeconds: 3600              # a roll waits this long for running turns
    terminationGraceSeconds: 7500   # pod grace; keep drainSeconds at least 120 s below it
  resources:
    requests: { cpu: "2", memory: "8Gi" }
    limits:   { cpu: "8", memory: "28Gi" }
  storage:
    className: fleet-xfs
    createClass: true          # false on Air-Gapped: bring your own XFS + prjquota class
    provisioner: ebs.csi.aws.com
    size: 500Gi
    quotaPerAgent: 20Gi
  networkPolicy:
    exceptCidrs: ["169.254.0.0/16"]
    exceptCidrsV6: ["fe80::/10"]
  shm:   { size: 4Gi }
  cache: { size: 20Gi }

Per turn

Ceilings per turn (fleet.turn). An agent can lower its own, never raise it.

Runner	Memory	CPU	Processes	Notes
Agent turn (Claude Code / Codex / OpenCode)	8 GiB	4 vCPU	1024 pids	Sized for real `node` / `python` builds inside the turn
Disk per agent	20 GiB			XFS project quota, hard limit; a `du`-based sweeper enforces it softly where `prjquota` is missing

Concurrency caps

Cap	Where	Default
Turns one executor accepts	`fleet.turnsMax` (`XPANDER_FLEET_TURNS_MAX`)	12
Turns above capacity	Queue in Redis and start as slots free	Nothing is dropped, only delayed

There is no per-organization cap. fleet.orgMaxTurns is removed, and a license’s agent cap does not limit concurrent turns. A values file that still sets it upgrades cleanly, and the key renders nothing.

Reading the runtime logs

One turn shows in two logs, the executor and the Agent Controller:

kubectl -n xpander logs -f fleet-executor-0 -c executor
kubectl -n xpander logs -f deploy/agent-controller

Executor log events, in order

Executor (fleet-executor-<n>, container executor), in order:

Event	Carries
`harness.fleet.turn_start`	The turn was claimed
`harness.cli.selected`	The CLI and where the choice came from, for example `cli=claude-code source=task`
`execution.runtime_selected`
`harness.sandbox.open`	The sandbox’s `uid` and `cgroup`
`harness.skills.sync`	`synced=` and `exposed=`, the skills placed in the sandbox
`harness.mcp_remote.resolved`	`servers=` and `tools=`, the MCP endpoints resolved for the turn and how many actions they expose
`harness.run.start`, `harness.cli.spawn`, `harness.cli.init`	The CLI process is up
`harness.cli.tool_call`, `harness.mcp.call`, `harness.mcp.result`, `harness.cli.tool_result`	One group per skill call the agent makes
`harness.cli.exit`	The CLI finished
`harness.sandbox.close`	`cpu_usec`, the turn’s CPU time
`harness.run.end`, `harness.fleet.turn_end`	The `outcome`

Agent Controller (deploy/agent-controller). The same turn shows as:

direct turn conversation=<id> task=<id> status=... received_to_enqueued_ms=... enqueued_to_claimed_ms=...: the message became a task and was picked up by the fleet. enqueued_to_claimed_ms is where a cold executor shows. The turn runs on the model the conversation picked. The agent record’s model is only the default for new conversations.
event=fleet.claimed executor=fleet-executor-0: which executor took it.
event=compute_usage.emitted cli=claude-code cpu_seconds=...: the compute record written when the turn ended.

LLM Gateway (llm-gw, kubectl -n xpander logs -f -l app=llm-gw). Each model call the turn made is one

event=llm_gw.call llm_route=gateway_direct status=... org=<id> agent=<id> task=<id> client=... provider=... request_id=<id>

line. To see the model a call used, paste its request_id into Request or trace ID and press Enter in Platform > AI governance > Logs, on your install’s own address.

Node prerequisites

All required: a cluster that fails one runs no agent turns. install.sh and the Air-Gapped preflight check each and name the fix. There is no Kubernetes 1.33, kernel 6.3, containerd 2.0 or user.max_user_namespaces minimum, and no node configuration.

Requirement	Why	How to check
amd64 / x86_64 nodes	The `agent-harness` image is amd64 only	Same rule as the rest of xpander
Release namespace admits PodSecurity `baseline`	The executor and slot containers are baseline pods (root with the runtime’s default capabilities, RuntimeDefault seccomp)	Unlabelled is fine; `restricted` is refused
Landlock on the executor nodes, or `fleet.landlock.enabled: false`	Landlock limits each turn to its own files and ports	`grep -o landlock /sys/kernel/security/lsm` prints `landlock`; the executor’s `/health` reports it too. See Landlock
A StorageClass that formats XFS and mounts it with `prjquota`	Hard per-agent quotas	Hybrid: `fleet.storage.createClass: true` (default) renders `fleet-xfs` for the EBS CSI driver (`ebs.csi.aws.com`, gp3, encrypted, XFS, `prjquota`; on EKS Auto Mode set `provisioner: ebs.csi.eks.amazonaws.com`, see EKS Auto Mode; on GKE `pd.csi.storage.gke.io`, on AKS `disk.csi.azure.com`). Air-Gapped: `global.fleet.storage.createClass: false` by default, so bring a class that mounts XFS with `prjquota`, or set it to `true` on EKS. Without `prjquota` the sweeper enforces quotas softly
RWO block storage, 500 GiB per executor	One volume per executor, via `volumeClaimTemplates`	The class exists and provisions
Cloud metadata blocked from the pod	The executor refuses to register while `169.254.169.254` answers from the pod	The fleet NetworkPolicy blocks it; with `fleet.networkPolicy.enabled: false`, block it yourself (your own policy, or IMDS hop limit 1 on EKS)
KEDA (optional)	Holds the executor count with a ScaledObject	Only with `fleet.keda.enabled`; KEDA is not bundled. Air-Gapped mirrors its images with `image_sync.sh --with-keda` and installs it first
Pinned image	The executor and slot containers run `agent-harness`	Air-Gapped render validation refuses an unpinned `global.images.agent-harness` tag or digest
Memory-rich nodes	Turn memory dominates; CPU is bursty	`r`-class (1:8 vCPU:GiB) fits the profile

Landlock on the executor nodes

Landlock limits each turn to the agent’s own files. It is on by default (fleet.landlock.enabled: true, or global.fleet.landlock.enabled on Air-Gapped). On a node without it, the executor takes no turn: /health answers 503, and its refusal field and the log say the kernel has no Landlock.

Landlock support by platform

Platform	Node image	Agent executors
Amazon EKS (incl. Auto Mode)	Amazon Linux 2023, Bottlerocket	Supported, nothing to set
Google GKE Standard	Ubuntu (`UBUNTU_CONTAINERD`)	Supported, nothing to set (recommended on GKE)
Google GKE Standard	Container-Optimized OS	Supported with `fleet.landlock.enabled: false`
Google GKE Autopilot	Container-Optimized OS	Supported with `fleet.landlock.enabled: false`
Azure AKS	Ubuntu, Azure Linux	Supported when the check above prints `landlock`, else with `fleet.landlock.enabled: false`

On GKE, keep the cluster on Container-Optimized OS and add one Ubuntu node pool for the executors:

gcloud container node-pools create xpander-agents --cluster <cluster> --image-type=UBUNTU_CONTAINERD \
  --machine-type <type> --num-nodes <n> --node-taints xpander.ai/agents=true:NoSchedule

fleet:
  nodeSelector:
    cloud.google.com/gke-nodepool: xpander-agents
  tolerations:
    - {key: xpander.ai/agents, operator: Equal, value: "true", effect: NoSchedule}

The taint is optional. If you cannot add a node pool, set fleet.landlock.enabled: false, and Landlock is off on every node.

What stays and what is lost without Landlock

Still in place without Landlock	Lost without Landlock
Each agent still runs as its own user. Its files and turn secrets are readable by that user only. Each turn runs in its own slot container behind the seccomp profile and `no_new_privs`. The slot is cleared between agents. Agents never hold platform keys.	Landlock’s file rules (a turn can read what ordinary file permissions allow) and its port rules

/health then shows landlock_enabled: false.

Fleet egress

The executors’ NetworkPolicy (fleet.networkPolicy.enabled, default true) gives agents the same reachability as any other pod, private ranges and in-cluster services included, minus link-local. fleet.networkPolicy.exceptCidrs defaults to ["169.254.0.0/16"] and exceptCidrsV6 to ["fe80::/10"]. Setting either replaces the whole list, so keep the link-local range in it. To close more ranges, use extraExceptCidrs. extraEgress appends raw egress rules. fleet.egress.allowHosts is still accepted for older agent-harness images and is no longer needed.

Autoscaling

An agent runs only on the executor that holds its home, so scaling in would strand agents. With fleet.keda.enabled: true the chart renders a KEDA ScaledObject that keeps every executor running:

Knob	Chart default
`fleet.keda.maxReplicas`	12; the ScaledObject sets both `minReplicaCount` and `maxReplicaCount` to it (`0` holds one). Set it to your fleet node count
`fleet.keda.minReplicas`	Removed; ignored if still set
Trigger	CPU utilization 70 % (`fleet.keda.cpuUtilization`)
Cooldown	300 s (`fleet.keda.cooldownSeconds`)

One executor per node, so grow the node group and maxReplicas together. Without KEDA (the default), set fleet.replicas. On a sealed network, a fixed count sized for peak is recommended.

Safe rollout, drain, and upgrades

A roll never kills a running turn

A rolling update, a KEDA scale-in and a pod delete all send SIGTERM, which the executor treats as a drain, not a kill:

The executor stops claiming work, and its heartbeat says draining. The Agent Controller routes around it: nothing new lands on it, and whatever was queued for it goes back to the shared queue. It still counts as fleet presence, so new turns wait for the replacement instead of starting anywhere else.
The turns in flight keep running for up to fleet.executor.drainSeconds (3600 s by default). A turn still running when that window ends is stopped the way a user stop is. The CLI is interrupted, its session is kept, and the task ends stopped with “The executor is restarting; send the message again to continue where it left off.” Sending the message again resumes the session on the new pod. No task is left hanging as executing.
Each slot container holds its turn until the executor’s health endpoint reports draining with no turn in flight. The pod grace (fleet.executor.terminationGraceSeconds, 7500 s: the CLI run budget plus five minutes) is a ceiling, never a wait. The pod leaves as soon as its containers exit.

A roll replaces one pod at a time, so a roll during a long turn takes as long as that turn, up to the drain window. So:

If your turns run long, raise drainSeconds toward the run budget. It must stay at least 120 s under the pod grace. The Air-Gapped chart refuses to render otherwise.
Never shorten a termination with --grace-period=0 or --force on a pod that still has turns. That is the one way to kill a turn.
Plan image rolls for quiet hours when you can. They are safe at any time, only slower under load.
The drain needs an agent-harness image that carries it. An older executor ignores the setting and exits as before.

Idle guard when the StorageClass is missing

The executor is Ready only when its volume binds on an XFS + prjquota StorageClass. So an install never waits on one that cannot bind, both charts check the live cluster:

What the chart renders, by situation

Situation	What the chart renders
`fleet.storage.createClass: false` and no StorageClass named `fleet.storage.className` exists	the executor StatefulSet with `replicas: 0`, annotated `xpander.ai/fleet-idle-reason`
`fleet.storage.createClass: true` but no `CSIDriver` named `fleet.storage.provisioner` is installed (the rendered class could never provision, the non-AWS cluster case)	same: idle, with the reason on the annotation
the class exists (or the driver is present for the rendered class)	the configured `replicas`

helm template and dry-runs always render the configured replicas (at least one). fleet.idleGuard: "off" removes the lookup for Argo CD, Flux and policy gates, and the executor stays Pending until its class exists. Create the class (or point provisioner at your CSI driver) and the next upgrade brings it up. Meanwhile turns queue.

`--wait`, `--atomic` and the installer

helm upgrade --wait (and --atomic) waits for the executor StatefulSet:

A running turn holds the roll. With --wait, an upgrade during a long turn can take up to the drain window plus pod start. Give --timeout at least the drain window (3600 s by default) or upgrade in a quiet period.
An idle executor never stalls an upgrade. With the idle guard above, a cluster without the storage prerequisites renders the executor at zero replicas and the upgrade completes.

install.sh handles both for you:

Run	What `install.sh` does
A first install	Runs without `--wait` or `--atomic` (the post-install smoke hook is the readiness gate), and prints the `xpander.ai/fleet-idle-reason` annotation when the executor is at zero
upgrades	Run `helm upgrade --atomic` with the installer’s timeout, then verify workloads, skipping zero-replica ones

Image and registry

Image details

Item	Value
Image	`agent-harness` (Docker Hub `xpanderaihub/agent-harness`, mirrored into your registry on Air-Gapped)
Compressed size	~1.5 GB (amd64)
On node disk	Plan ~4 GiB for the extracted image
Contents	Claude Code, Codex, OpenCode, Playwright Chromium, Node 22, Python 3.12, git, plus `kubectl`, `helm`, `aws`, `terraform`, `gh`, `docker` client and other developer utilities
Pull policy	Executors pull on every roll (`Always`); pin the tag or digest so a roll is deliberate

The executor and every slot use this image. Budget the ~1.5 GB pull per executor node. On Air-Gapped, mirror it at the pinned tag before the first executor starts.

Network egress

Executors hold no LLM provider key. A turn’s model calls go straight to the in-cluster LLM Gateway, which is always on, with the run’s own token. The gateway is the component that egresses to the model provider.

Egress paths

From	To	When
Executor	Agent Controller (registration, heartbeat, claim, run tokens) and the LLM Gateway Service on port 4000 (model calls)	Always, in-cluster (`fleet.controllerUrl`, default the in-cluster Service)
LLM Gateway	`api.anthropic.com`, `bedrock-runtime.<region>.amazonaws.com`, or your OpenAI-compatible endpoint	Every model call, on behalf of the CLIs
Executor	`registry.npmjs.org` (or your `XPANDER_NPM_REGISTRY` mirror)	Only when CLI auto-update is on (`cli_auto_update`, default on). The preinstalled CLIs need no egress; Air-Gapped uses a sealed CLI bundle instead
A turn’s own commands	npm / PyPI / apt / Homebrew, or their mirrors	Only when an agent’s runtime environment or the agent itself installs packages; follows the install’s package-mirror contract
Executor (NetworkPolicy)	DNS, the Agent Controller, the LLM Gateway, and every other address except link-local (`fleet.networkPolicy.exceptCidrs`), private ranges included	The chart renders this policy (Fleet egress); on Air-Gapped the internet leg is dropped with `global.network.internetEgress: false`

The executor refuses to register while the instance metadata endpoint (169.254.169.254) answers from the pod, so a turn can never read node credentials.

Model providers

Keys are saved in the organization (Platform > AI governance > Models) or on the install (secrets.static), and the LLM Gateway adds them to each call. Per-provider keys, Bedrock authentication and default harnesses: AI vendors.

Govern

Company Brain

LLM Gateway

Deploy

Integrations

Licensing and identity

Security and compliance

Runtime and node reference

Agent workspaces

The executor fleet

Per executor

Per turn

Concurrency caps

Reading the runtime logs

Node prerequisites

Landlock on the executor nodes

Fleet egress

Autoscaling

Safe rollout, drain, and upgrades

A roll never kills a running turn

Idle guard when the StorageClass is missing

`--wait`, `--atomic` and the installer

Image and registry

Network egress

Model providers

Next steps

Size your cluster

Set up an EKS cluster

​Agent workspaces

​The executor fleet

​Per executor

​Per turn

​Concurrency caps

​Reading the runtime logs

​Node prerequisites

​Landlock on the executor nodes

​Fleet egress

​Autoscaling

​Safe rollout, drain, and upgrades

​A roll never kills a running turn

​Idle guard when the StorageClass is missing

​--wait, --atomic and the installer

​Image and registry

​Network egress

​Model providers

​Next steps

Size your cluster

Set up an EKS cluster

Agent workspaces

The executor fleet

Per executor

Per turn

Concurrency caps

Reading the runtime logs

Node prerequisites

Landlock on the executor nodes

Fleet egress

Autoscaling

Safe rollout, drain, and upgrades

A roll never kills a running turn

Idle guard when the StorageClass is missing

`--wait`, `--atomic` and the installer

Image and registry

Network egress

Model providers

Next steps