> ## Documentation Index > Fetch the complete documentation index at: https://docs.xpander.ai/llms.txt > Use this file to discover all available pages before exploring further. # Runtime and node reference > Agent workspaces, the executor fleet, node prerequisites, Landlock, autoscaling, safe rollouts, the runtime image and network egress on a self-hosted install. Reference for the parts of a self-hosted install that run agents. How much cluster to plan is on [Sizing and requirements](https://docs.xpander.ai/self-hosted/sizing.md). ## Agent workspaces Each agent whose harness is **xpander** and that uses a workspace gets its own pod, created on demand and reclaimed when idle. Size for **concurrently active** workspaces, not total agents. | Resource | Value | | - | - | | CPU | 250m request, 2000m limit | | Memory | 256 Mi request, 3 Gi limit | | Storage | 5 GiB persistent volume per agent | | Idle shutdown | 5 minutes without activity (configurable via `SANDBOX_IDLE_TIMEOUT` on the Agent Controller) | | Maximum lifetime | 3 hours, then the pod is recycled | Idle workspaces use \~50 Mi. An active one uses \~0.5-1 vCPU and up to 1 GiB, bursting to its limits. The volume **persists after the pod is reclaimed**, so plan 5 GiB per workspace-using agent, not per pod. > [!NOTE] > > The workspace image is \~1 GiB and is pulled on the first start on each node. Budget \~2 GiB of node disk for the image cache. ## The executor fleet The fleet is a `fleet-executor` StatefulSet. Each replica is one PodSecurity **baseline** pod over one XFS volume: no privileged container, no host namespaces, no user namespaces and no node preparation. The pod holds the **executor** container, which claims turns, plus one `slot-` container per turn slot (`fleet.turnsMax`). Each turn runs in its own slot container, as the agent's own uid. It has its own processes, `/proc`, `/tmp` and memory and CPU limit, behind a seccomp profile, `no_new_privs` and Landlock. An agent's directory lives on exactly one executor's volume, and its turns always run there. > [!NOTE] > > The fleet is **always rendered** by both charts (there is no enable flag). It starts warm with one executor (`fleet.replicas: 1` on Hybrid, `global.fleet.replicas: 1` on Air-Gapped). Harness agents (Claude Code, Codex, OpenCode) run only on the executors, never as a pod per agent. So the chart always keeps at least one: `replicas: 0` (or a blank value) renders one executor. The one case that renders zero is a live cluster whose executor volume can never bind (see [Idle guard](#idle-guard-when-the-storageclass-is-missing)). Agent turns then wait in the queue until it is fixed. | | Executor fleet | | - | - | | What an agent is | A directory (its home, CLI state, working tree) on the executor's XFS volume, under a 20 GiB quota | | What runs a turn | A free **slot container** on the executor that holds the agent's directory, as the agent's own uid, limited by Landlock (unless `fleet.landlock.enabled: false`) and the seccomp profile | | Cost of an idle agent | Disk only; no pod, no requests | | Cost of a new agent | `mkdir` on the first turn, seconds | | Scaling axis | **Concurrent turns**, never the number of agents | | Model credentials | None in the pod. Each turn calls the in-cluster LLM Gateway directly, with the run's own short-lived token | ## Per executor Defaults per executor pod: 2 / 8 vCPU and 8 / 28 GiB (request / limit), 12 slot containers, a 500 GiB XFS volume. > [accordion] Executor resources, volume, grace and placement > > | Resource | Self-hosted chart default | > | - | - | > | Executor CPU request / limit | **2 / 8 vCPU** (`fleet.resources`) | > | Executor memory request / limit | **8 / 28 GiB** | > | Slot containers | One per slot (`fleet.turnsMax`, default **12**); each requests 10m CPU / 64 Mi and is limited to the turn ceiling (`fleet.turn`, 4 vCPU / 8 GiB). Changing `turnsMax` rolls the executor pods | > | Volume | **500 GiB**, XFS, `prjquota`, gp3 encrypted, RWO | > | Per-agent disk quota (`quotaPerAgent`) | 20 GiB (XFS project quota, hard; a ceiling, not a reservation) | > | `/dev/shm` (memory-backed, for Chromium) | 4 GiB | > | Package cache (`emptyDir`, node disk) | 20 GiB | > | Termination grace | 7500 s (`fleet.executor.terminationGraceSeconds`), a ceiling: the executor first **drains** running turns for up to 3600 s (`fleet.executor.drainSeconds`) | > | Probes | readiness 10 s x 6, startup 5 s x 36 (3 min) | > | Placement | One executor per node (required anti-affinity); `fleet.nodeSelector` / `fleet.tolerations` | > [!WARNING] > > Size the executor's **memory against `turnsMax`**. Each slot container is limited to the turn ceiling (8 GiB by default), and the node must back them. If the turns together use more memory than the node has, the kernel OOM-kills one. The chart default (12 slots, executor limit 28 GiB) is a conservative starting footprint. It assumes typical turns stay well below their ceiling. For a production fleet, either use memory-rich nodes sized toward `turnsMax x 8 GiB` or lower `turnsMax`. > [accordion] Fleet values (Hybrid chart) > > ```yaml theme={"dark"} > # Hybrid (xpander chart); on Air-Gapped the same keys live under global.fleet > fleet: > replicas: 1 # 0 or blank also renders 1 > turnsMax: 12 > turn: { memory: "8Gi", cpu: "4", pids: 1024 } > landlock: > enabled: true # false on nodes without Landlock > executor: > drainSeconds: 3600 # a roll waits this long for running turns > terminationGraceSeconds: 7500 # pod grace; keep drainSeconds at least 120 s below it > resources: > requests: { cpu: "2", memory: "8Gi" } > limits: { cpu: "8", memory: "28Gi" } > storage: > className: fleet-xfs > createClass: true # false on Air-Gapped: bring your own XFS + prjquota class > provisioner: ebs.csi.aws.com > size: 500Gi > quotaPerAgent: 20Gi > networkPolicy: > exceptCidrs: ["169.254.0.0/16"] > exceptCidrsV6: ["fe80::/10"] > shm: { size: 4Gi } > cache: { size: 20Gi } > ``` ## Per turn Ceilings per turn (`fleet.turn`). An agent can lower its own, never raise it. | Runner | Memory | CPU | Processes | Notes | | - | - | - | - | - | | Agent turn (Claude Code / Codex / OpenCode) | **8 GiB** | **4 vCPU** | 1024 pids | Sized for real `node` / `python` builds inside the turn | | Disk per agent | 20 GiB | | | XFS project quota, hard limit; a `du`-based sweeper enforces it softly where `prjquota` is missing | ### Concurrency caps | Cap | Where | Default | | - | - | - | | Turns one executor accepts | `fleet.turnsMax` (`XPANDER_FLEET_TURNS_MAX`) | 12 | | Turns above capacity | Queue in Redis and start as slots free | Nothing is dropped, only delayed | There is no per-organization cap. `fleet.orgMaxTurns` is removed, and a license's agent cap does not limit concurrent turns. A values file that still sets it upgrades cleanly, and the key renders nothing. ## Reading the runtime logs One turn shows in two logs, the executor and the Agent Controller: ```bash theme={"dark"} kubectl -n xpander logs -f fleet-executor-0 -c executor kubectl -n xpander logs -f deploy/agent-controller ``` > [accordion] Executor log events, in order > > **Executor** (`fleet-executor-`, container `executor`), in order: > > | Event | Carries | > | - | - | > | `harness.fleet.turn_start` | The turn was claimed | > | `harness.cli.selected` | The CLI and where the choice came from, for example `cli=claude-code source=task` | > | `execution.runtime_selected` | | > | `harness.sandbox.open` | The sandbox's `uid` and `cgroup` | > | `harness.skills.sync` | `synced=` and `exposed=`, the skills placed in the sandbox | > | `harness.mcp_remote.resolved` | `servers=` and `tools=`, the MCP endpoints resolved for the turn and how many actions they expose | > | `harness.run.start`, `harness.cli.spawn`, `harness.cli.init` | The CLI process is up | > | `harness.cli.tool_call`, `harness.mcp.call`, `harness.mcp.result`, `harness.cli.tool_result` | One group per skill call the agent makes | > | `harness.cli.exit` | The CLI finished | > | `harness.sandbox.close` | `cpu_usec`, the turn's CPU time | > | `harness.run.end`, `harness.fleet.turn_end` | The `outcome` | **Agent Controller** (`deploy/agent-controller`). The same turn shows as: * `direct turn conversation= task= status=... received_to_enqueued_ms=... enqueued_to_claimed_ms=...`: the message became a task and was picked up by the fleet. `enqueued_to_claimed_ms` is where a cold executor shows. The turn runs on the model the conversation picked. The agent record's model is only the default for new conversations. * `event=fleet.claimed executor=fleet-executor-0`: which executor took it. * `event=compute_usage.emitted cli=claude-code cpu_seconds=...`: the compute record written when the turn ended. **LLM Gateway** (`llm-gw`, `kubectl -n xpander logs -f -l app=llm-gw`). Each model call the turn made is one `event=llm_gw.call llm_route=gateway_direct status=... org= agent= task= client=... provider=... request_id=` line. To see the model a call used, paste its `request_id` into **Request or trace ID** and press Enter in Platform > AI governance > Logs, on your install's own address. ## Node prerequisites All required: a cluster that fails one runs no agent turns. `install.sh` and the Air-Gapped preflight check each and name the fix. There is no Kubernetes 1.33, kernel 6.3, containerd 2.0 or `user.max_user_namespaces` minimum, and no node configuration. | Requirement | Why | How to check | | - | - | - | | **amd64 / x86\_64 nodes** | The `agent-harness` image is amd64 only | Same rule as the rest of xpander | | **Release namespace admits PodSecurity `baseline`** | The executor and slot containers are baseline pods (root with the runtime's default capabilities, RuntimeDefault seccomp) | Unlabelled is fine; `restricted` is refused | | **Landlock on the executor nodes**, or `fleet.landlock.enabled: false` | Landlock limits each turn to its own files and ports | `grep -o landlock /sys/kernel/security/lsm` prints `landlock`; the executor's `/health` reports it too. See [Landlock](#landlock-on-the-executor-nodes) | | **A StorageClass that formats XFS and mounts it with `prjquota`** | Hard per-agent quotas | Hybrid: `fleet.storage.createClass: true` (default) renders `fleet-xfs` for the EBS CSI driver (`ebs.csi.aws.com`, gp3, encrypted, XFS, `prjquota`; on EKS Auto Mode set `provisioner: ebs.csi.eks.amazonaws.com`, see [EKS Auto Mode](https://docs.xpander.ai/self-hosted/eks-setup.md#eks-auto-mode); on GKE `pd.csi.storage.gke.io`, on AKS `disk.csi.azure.com`). Air-Gapped: `global.fleet.storage.createClass: false` by default, so bring a class that mounts XFS with `prjquota`, or set it to `true` on EKS. Without `prjquota` the sweeper enforces quotas softly | | **RWO block storage, 500 GiB per executor** | One volume per executor, via `volumeClaimTemplates` | The class exists and provisions | | **Cloud metadata blocked from the pod** | The executor refuses to register while `169.254.169.254` answers from the pod | The fleet NetworkPolicy blocks it; with `fleet.networkPolicy.enabled: false`, block it yourself (your own policy, or IMDS hop limit 1 on EKS) | | **KEDA** (optional) | Holds the executor count with a ScaledObject | Only with `fleet.keda.enabled`; KEDA is not bundled. Air-Gapped mirrors its images with `image_sync.sh --with-keda` and installs it first | | **Pinned image** | The executor and slot containers run `agent-harness` | Air-Gapped render validation refuses an unpinned `global.images.agent-harness` tag or digest | | **Memory-rich nodes** | Turn memory dominates; CPU is bursty | `r`-class (1:8 vCPU:GiB) fits the profile | ## Landlock on the executor nodes Landlock limits each turn to the agent's own files. It is on by default (`fleet.landlock.enabled: true`, or `global.fleet.landlock.enabled` on Air-Gapped). On a node without it, the executor takes no turn: `/health` answers 503, and its `refusal` field and the log say the kernel has no Landlock. > [accordion] Landlock support by platform > > | Platform | Node image | Agent executors | > | - | - | - | > | Amazon EKS (incl. Auto Mode) | Amazon Linux 2023, Bottlerocket | Supported, nothing to set | > | Google GKE Standard | Ubuntu (`UBUNTU_CONTAINERD`) | Supported, nothing to set (recommended on GKE) | > | Google GKE Standard | Container-Optimized OS | Supported with `fleet.landlock.enabled: false` | > | Google GKE Autopilot | Container-Optimized OS | Supported with `fleet.landlock.enabled: false` | > | Azure AKS | Ubuntu, Azure Linux | Supported when the check above prints `landlock`, else with `fleet.landlock.enabled: false` | On GKE, keep the cluster on Container-Optimized OS and add one Ubuntu node pool for the executors: ```bash theme={"dark"} gcloud container node-pools create xpander-agents --cluster --image-type=UBUNTU_CONTAINERD \ --machine-type --num-nodes --node-taints xpander.ai/agents=true:NoSchedule ``` ```yaml theme={"dark"} fleet: nodeSelector: cloud.google.com/gke-nodepool: xpander-agents tolerations: - {key: xpander.ai/agents, operator: Equal, value: "true", effect: NoSchedule} ``` The taint is optional. If you cannot add a node pool, set `fleet.landlock.enabled: false`, and Landlock is off on every node. > [accordion] What stays and what is lost without Landlock > > | Still in place without Landlock | Lost without Landlock | > | - | - | > | Each agent still runs as its own user. Its files and turn secrets are readable by that user only. Each turn runs in its own slot container behind the seccomp profile and `no_new_privs`. The slot is cleared between agents. Agents never hold platform keys. | Landlock's file rules (a turn can read what ordinary file permissions allow) and its port rules | `/health` then shows `landlock_enabled: false`. ### Fleet egress The executors' NetworkPolicy (`fleet.networkPolicy.enabled`, default `true`) gives agents the same reachability as any other pod, private ranges and in-cluster services included, minus link-local. `fleet.networkPolicy.exceptCidrs` defaults to `["169.254.0.0/16"]` and `exceptCidrsV6` to `["fe80::/10"]`. Setting either replaces the whole list, so keep the link-local range in it. To close more ranges, use `extraExceptCidrs`. `extraEgress` appends raw egress rules. `fleet.egress.allowHosts` is still accepted for older `agent-harness` images and is no longer needed. ## Autoscaling An agent runs only on the executor that holds its home, so scaling in would strand agents. With `fleet.keda.enabled: true` the chart renders a KEDA `ScaledObject` that keeps every executor running: | Knob | Chart default | | - | - | | `fleet.keda.maxReplicas` | 12; the ScaledObject sets both `minReplicaCount` and `maxReplicaCount` to it (`0` holds one). Set it to your fleet node count | | `fleet.keda.minReplicas` | Removed; ignored if still set | | Trigger | CPU utilization 70 % (`fleet.keda.cpuUtilization`) | | Cooldown | 300 s (`fleet.keda.cooldownSeconds`) | One executor per node, so grow the node group and `maxReplicas` together. Without KEDA (the default), set `fleet.replicas`. On a sealed network, a fixed count sized for peak is recommended. ## Safe rollout, drain, and upgrades ### A roll never kills a running turn A rolling update, a KEDA scale-in and a pod delete all send `SIGTERM`, which the executor treats as a **drain**, not a kill: 1. The executor stops claiming work, and its heartbeat says `draining`. The Agent Controller routes around it: nothing new lands on it, and whatever was queued for it goes back to the shared queue. It still counts as fleet presence, so new turns wait for the replacement instead of starting anywhere else. 2. The turns in flight keep running for up to `fleet.executor.drainSeconds` (3600 s by default). A turn still running when that window ends is stopped the way a user stop is. The CLI is interrupted, its session is kept, and the task ends `stopped` with "The executor is restarting; send the message again to continue where it left off." Sending the message again resumes the session on the new pod. No task is left hanging as executing. 3. Each slot container holds its turn until the executor's health endpoint reports draining with no turn in flight. The pod grace (`fleet.executor.terminationGraceSeconds`, 7500 s: the CLI run budget plus five minutes) is a ceiling, never a wait. The pod leaves as soon as its containers exit. A roll replaces one pod at a time, so **a roll during a long turn takes as long as that turn, up to the drain window.** So: * If your turns run long, raise `drainSeconds` toward the run budget. It must stay at least 120 s under the pod grace. The Air-Gapped chart refuses to render otherwise. * Never shorten a termination with `--grace-period=0` or `--force` on a pod that still has turns. That is the one way to kill a turn. * Plan image rolls for quiet hours when you can. They are safe at any time, only slower under load. * The drain needs an `agent-harness` image that carries it. An older executor ignores the setting and exits as before. ### Idle guard when the StorageClass is missing The executor is Ready only when its volume binds on an XFS + `prjquota` StorageClass. So an install never waits on one that cannot bind, both charts check the **live cluster**: > [accordion] What the chart renders, by situation > > | Situation | What the chart renders | > | - | - | > | `fleet.storage.createClass: false` and no StorageClass named `fleet.storage.className` exists | the executor StatefulSet with `replicas: 0`, annotated `xpander.ai/fleet-idle-reason` | > | `fleet.storage.createClass: true` but no `CSIDriver` named `fleet.storage.provisioner` is installed (the rendered class could never provision, the non-AWS cluster case) | same: idle, with the reason on the annotation | > | the class exists (or the driver is present for the rendered class) | the configured `replicas` | `helm template` and dry-runs always render the configured replicas (at least one). `fleet.idleGuard: "off"` removes the lookup for Argo CD, Flux and policy gates, and the executor stays Pending until its class exists. Create the class (or point `provisioner` at your CSI driver) and the next upgrade brings it up. Meanwhile turns queue. ### `--wait`, `--atomic` and the installer `helm upgrade --wait` (and `--atomic`) waits for the executor StatefulSet: * **A running turn holds the roll.** With `--wait`, an upgrade during a long turn can take up to the drain window plus pod start. Give `--timeout` at least the drain window (3600 s by default) or upgrade in a quiet period. * **An idle executor never stalls an upgrade.** With the idle guard above, a cluster without the storage prerequisites renders the executor at zero replicas and the upgrade completes. `install.sh` handles both for you: | Run | What `install.sh` does | | - | - | | A **first install** | Runs without `--wait` or `--atomic` (the post-install smoke hook is the readiness gate), and prints the `xpander.ai/fleet-idle-reason` annotation when the executor is at zero | | **upgrades** | Run `helm upgrade --atomic` with the installer's timeout, then verify workloads, skipping zero-replica ones | ## Image and registry > [accordion] Image details > > | Item | Value | > | - | - | > | Image | `agent-harness` (Docker Hub `xpanderaihub/agent-harness`, mirrored into your registry on Air-Gapped) | > | Compressed size | **\~1.5 GB** (amd64) | > | On node disk | Plan **\~4 GiB** for the extracted image | > | Contents | Claude Code, Codex, OpenCode, Playwright Chromium, Node 22, Python 3.12, git, plus `kubectl`, `helm`, `aws`, `terraform`, `gh`, `docker` client and other developer utilities | > | Pull policy | Executors pull on every roll (`Always`); pin the tag or digest so a roll is deliberate | The executor and every slot use this image. Budget the \~1.5 GB pull per executor node. On Air-Gapped, mirror it at the pinned tag before the first executor starts. ## Network egress Executors hold **no LLM provider key**. A turn's model calls go straight to the in-cluster LLM Gateway, which is always on, with the run's own token. The gateway is the component that egresses to the model provider. > [accordion] Egress paths > > | From | To | When | > | - | - | - | > | Executor | Agent Controller (registration, heartbeat, claim, run tokens) and the LLM Gateway Service on port 4000 (model calls) | Always, in-cluster (`fleet.controllerUrl`, default the in-cluster Service) | > | LLM Gateway | `api.anthropic.com`, `bedrock-runtime..amazonaws.com`, or your OpenAI-compatible endpoint | Every model call, on behalf of the CLIs | > | Executor | `registry.npmjs.org` (or your `XPANDER_NPM_REGISTRY` mirror) | Only when **CLI auto-update** is on (`cli_auto_update`, default on). The preinstalled CLIs need no egress; Air-Gapped uses a sealed CLI bundle instead | > | A turn's own commands | npm / PyPI / apt / Homebrew, or their mirrors | Only when an agent's runtime environment or the agent itself installs packages; follows the install's package-mirror contract | > | Executor (NetworkPolicy) | DNS, the Agent Controller, the LLM Gateway, and every other address except link-local (`fleet.networkPolicy.exceptCidrs`), private ranges included | The chart renders this policy ([Fleet egress](#fleet-egress)); on Air-Gapped the internet leg is dropped with `global.network.internetEgress: false` | The executor refuses to register while the instance metadata endpoint (`169.254.169.254`) answers from the pod, so a turn can never read node credentials. ## Model providers Keys are saved in the organization (**Platform > AI governance > Models**) or on the install (`secrets.static`), and the LLM Gateway adds them to each call. Per-provider keys, Bedrock authentication and default harnesses: [AI vendors](https://docs.xpander.ai/run/llm-gateway/providers-and-keys.md#keys-on-a-self-hosted-install). ## Next steps > [card] [Size your cluster](https://docs.xpander.ai/self-hosted/sizing.md) > > Footprint, calculator and capacity planning. > [card] [Set up an EKS cluster](https://docs.xpander.ai/self-hosted/eks-setup.md) > > Nodes and StorageClasses. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.