Sizing and requirements

A self-hosted xpander runtime has three footprints. The base services (web UI, control plane, data tier) are sized per deployment. The agent workspace pod is created on demand for an agent that uses a workspace. The executor fleet is where agent turns run: a shared pool of warm executor pods where an agent is a directory on an XFS volume and every turn runs Claude Code, Codex or OpenCode in its own throwaway sandbox. This page has all three, so one read sizes a Hybrid or Air-Gapped cluster. Every agent’s turns land on the fleet, whether the agent is created in Xpander Chat, over the API or the SDK, or built by Omni, the built-in agent. The one exception is an SDK agent whose loop you run in your own process. Plan fleet capacity for your whole agent population.

Base footprint

Hybrid

Minimum footprint (the installer shows the same numbers, published at charts.xpander.ai/requirements.json):

	Hybrid minimum
Nodes	3 × 8 vCPU / 32 GiB
Storage	~50 GiB PVC
Kubernetes	≥ 1.28
Architecture	amd64
Network	Egress to `deployment-manager.xpander.ai` (or PrivateLink) and to your LLM providers
Extras	Ingress controller + wildcard DNS optional

Agents are autonomous: tasks come from schedules, webhooks, the SDK and API, and other agents, not only chat (in xpander cloud production, only ~5% of executions are interactive chat). Size for the number of tasks running concurrently at peak, whatever triggers them.

Concurrent tasks	Agents	Cluster capacity	Example nodes	Persistent storage
3	10	10 vCPU / 19 GiB	3 × `m5.xlarge` (or 2 × `t3.xlarge` for evaluation)	~110 GiB
10	25	19 vCPU / 34 GiB	3 × `m5.2xlarge`	~180 GiB
20	50	29 vCPU / 49 GiB	4 × `m5.2xlarge`	~310 GiB
50	100	66 vCPU / 105 GiB	9 × `m5.2xlarge`, enable autoscaling	~570 GiB

Capacity figures include ~50% headroom over pod resource requests for burst load, Kubernetes system pods, and node upgrades.

Sizing calculator

Take your own numbers: peak concurrent tasks, agents, and how many of them use a workspace. The estimate follows the model described under it.

How the estimate works

The calculator applies the same model we use to size xpander cloud:

Concurrent tasks is the primary driver. A task is any agent execution, from any source: chat, API/SDK calls, schedules, webhooks, MCP clients, Slack, or another agent. If you don’t know your peak concurrency yet, assume one concurrent task per agent - that covers most production organizations we measured. In practice most teams peak at just a few concurrent tasks (typically under 5), and even the busiest organization peaks around 30.
Execution replicas (agent-worker) = concurrent tasks ÷ 4 (each replica processes 4 tasks by default, configurable with MAX_CONCURRENT_EXECUTIONS). Each replica requests 2 vCPU / 2.5 GiB.
Agent Controllers: 1 replica up to ~8 concurrent tasks, 2 up to ~20, 3 beyond. Each requests 1 vCPU / 2 GiB.
Active workspaces = concurrent tasks × workspace share, capped at the agent count. Budget ~0.5 vCPU / 1 GiB per active workspace (measured typical load; each can burst to its 2 vCPU / 3 GiB limit).
Fixed base (~2 vCPU / 6 GiB): AI Gateway, MCP, Chat, Code Runner, AWS Operator, API, Redis, PostgreSQL, container registry, metrics-server at one replica each.
Cluster capacity = requests × 1.5 for burst, DaemonSets, and safe node drains. Storage = 48 GiB of chart volumes + 5 GiB per agent (each agent that uses a workspace keeps a persistent volume) + image cache per node.

Per-service requirements

Per-replica values. Defaults are what the Helm chart applies out of the box; recommended raises the values we found low against measured production usage.

Service	CPU request	Memory request	CPU limit	Memory limit	Notes
Agent Controller	1000m	2 Gi	3000m	4 Gi	Orchestrator; observed steady-state memory is ~1.5-2 GiB, and the limit carries headroom for peaks
Execution tier (`agent-worker`)	2000m	2.25 Gi → 2.5 Gi	4000m	3 Gi	Scale replicas for task concurrency, not size
API	100m → 250m	256 Mi → 1 Gi	- → 2000m	- → 1.5 Gi	Chart sets no limits; observed usage ~0.9-1.4 GiB
MCP	200m	750 Mi	400m	1 Gi
Chat	300m	1 Gi	600m	1.5 Gi
AI Gateway	200m	400 Mi	400m	600 Mi
Code Runner	400m	800 Mi	800m	1229 Mi	Runs privileged (Docker-in-Docker); each concurrent code execution adds up to 1 vCPU / 2 GiB
AWS Operator	200m	400 Mi	400m	600 Mi	Optional component

Override any of these in your Helm values (keys are camelCase):

resources:
  agentController:
    requests:
      cpu: "1000m"
      memory: "2Gi"
    limits:
      cpu: "3000m"
      memory: "4Gi"

Horizontal Pod Autoscaling is available for Agent Controller, AI Gateway, the execution tier (agent-worker), and MCP via the chart’s autoscaling values. The chart installs metrics-server by default to support it.

Air-Gapped

	Air-Gapped minimum
Nodes	5 × 16 vCPU / 64 GiB, amd64
Storage	300–500 GiB PVC on an RWO StorageClass, encrypted at rest
Kubernetes	≥ 1.28, with a CSI driver and a default RWO StorageClass

The full prerequisite list (network, license files, registry, tooling) is on Air-Gapped: Prerequisites.

Sizing tiers

The Air-gapped deployment runs everything in-cluster: the execution tier (the agent-worker service) plus the control plane services, the web UI, the Supabase stack (its own PostgreSQL 15, auth, storage, edge functions), and MongoDB.

Tier	Nodes	Fits	Storage (PVC)
Initial deployment	3 × (8 vCPU / 32 GiB)	~10 users, 5–10 concurrent agent runs	~200 GiB
Production	5 × (16 vCPU / 64 GiB)	~50 users, 30–50 concurrent workspaces, headroom	300–500 GiB
+ GPU (only if needed)	+1 GPU node (L4/A10 class, 8 vCPU / 32 GiB)	Local voice STT; in-cluster LLM inference is your own pool, sized separately by model	+ model weights

Storage breakdown at the production tier: Supabase PostgreSQL 50 GiB, xpander’s PostgreSQL 50 GiB, Supabase Storage 50 GiB+, MongoDB 20 GiB, Redis 8 GiB, registry ~50 GiB, mirror caches 50–100 GiB. All RWO, one StorageClass (the chart’s preflight verifies it exists). External data tier sizing. You can bring managed stores instead (Data Tier). A reference customer install runs on these external stores:

RDS PostgreSQL 16 on db.t4g.small with 50 GiB gp3
MemoryDB on db.t4g.small with one shard and one replica
S3 as the Supabase Storage backend for blobs

With external stores, xpander’s PostgreSQL, Redis and storage PVCs listed above fall away. Scale with usage. Most services default to a single replica; deployment-manager, actions, and the Supabase edge runtime ship at ×2 replicas with soft anti-affinity out of the box. For production, run the rest of the hot path at ×2 as well: agent-controller, agent-worker, api, agents, functions.

Scaling with usage

The chart ships fixed replica counts and no autoscalers: scaling policy is yours. What actually grows with usage, and how to scale each:

What grows	Grows with	How to scale it
`agent-worker`: executes agent tasks	Concurrent agent runs. Each replica runs up to 4 tasks at once (`MAX_CONCURRENT_EXECUTIONS`); beyond `replicas × 4`, tasks queue	The primary scaler: raise `agent-worker.replicas`. The service is stateless: adding replicas is always safe. If you want this automated, don’t autoscale `agent-worker` on CPU: a replica waiting on a model reply looks idle even when it’s full. The honest signal is how many runs are waiting to start (xpander keeps that queue in its Redis), and KEDA (an open-source Kubernetes autoscaler) can watch that queue’s length and add or remove `agent-worker` replicas to keep pace
Agent workspace pods (`sb-*`): the agents’ computers	Number of agents with an active workspace: one pod + PVC each, created on demand by agent-controller and reaped when idle	Not a Deployment you own, so not HPA territory: this is node capacity. Size headroom for your expected concurrent workspaces, or run a node autoscaler
Executor fleet (the `fleet-executor` StatefulSet), where agent turns run	Peak concurrent turns. One executor per node, 12 turn slots (8 GiB / 4 vCPU ceiling each; the chart’s `global.fleet.turnsMax` and `global.fleet.turn` values, which override the image’s own default of 6 turns and 16 GiB per turn), 500 GiB XFS volume with a 5 GiB quota per agent; idle agents cost only disk	`global.fleet.replicas` (fixed capacity, the sealed-network posture), or KEDA with `global.fleet.keda.enabled` (mirror KEDA with `image_sync.sh --with-keda`). Prerequisites and capacity tables: the executor fleet
Code-execution Jobs: spawned per run by the actions service	Bursts of agent code execution	Node capacity, same as workspaces
Request path: `api`, `agent-controller`, `agents`, `functions`	Users and API traffic; `functions` also runs CPU-heavy local embedding during knowledge-base ingest	Raise `.replicas`; standard CPU-based HPA works here if you install metrics-server (not required otherwise)

Node capacity: workspaces and code-execution Jobs create pods on demand, so the real ceiling on concurrent agent activity is how many nodes you have. Two workable postures:

Static capacity: size the node group for your peak concurrent workspaces and leave it fixed. Simplest, fully offline, and the recommended posture for sealed networks.
Cluster Autoscaler: adds nodes automatically when pods can’t schedule. It works in a no-internet VPC, but it talks to your cloud’s scaling API to do its job, so that API must be reachable from inside the network (on AWS: the autoscaling and ec2 VPC endpoints).

Karpenter is not a good fit for sealed networks: it fetches instance pricing data from the internet.

Agent workspaces

Each agent that uses the agent workspace gets a dedicated pod, created on demand and reclaimed when idle. Size the cluster for the number of concurrently active workspaces, not total agents.

Resource	Value
CPU	250m request, 2000m limit
Memory	256 Mi request, 3 Gi limit
Storage	5 GiB persistent volume per agent
Idle shutdown	5 minutes without activity (configurable via `SANDBOX_IDLE_TIMEOUT` on the Agent Controller)
Maximum lifetime	3 hours, then the pod is recycled

Idle workspaces consume almost nothing (~50 Mi observed); an active workspace typically uses ~0.5-1 vCPU and up to 1 GiB, bursting to its limits for heavy work. The workspace volume persists after the pod is reclaimed, so a returning agent keeps its files - plan 5 GiB of storage per workspace-using agent, not per pod.

The workspace image is ~1 GiB. The first workspace start on each node pulls it; subsequent starts on that node are fast. Budget ~2 GiB of node disk for image cache.

The fleet is always rendered by both charts (there is no enable flag) and starts warm with one executor (fleet.replicas: 1 on Hybrid, global.fleet.replicas: 1 on Air-Gapped). The Agent Controller sends a turn to the fleet whenever a live executor is registered. Keep it warm: the first turn of any agent is then seconds away, and idle agents cost only their directory. Where a cluster cannot host the privileged executor, set replicas: 0; on a live cluster the chart also idles the executor by itself when its StorageClass is missing (see Idle guard).

The executor fleet

The fleet is a fleet-executor StatefulSet. Each replica is one pod with two privileged containers over one XFS volume: the executor, which runs turns, and a mounter sidecar, which binds each agent’s directory into the executor for the duration of a turn and enforces the per-agent disk quota. An agent’s directory lives on exactly one executor’s volume, and its turns always run there.

	Executor fleet
What an agent is	A directory (its home, CLI state, working tree) on the executor’s XFS volume, under a 5 GiB quota
What runs a turn	A free slot on the executor that holds the agent’s directory, inside a bubblewrap sandbox with its own cgroup and seccomp filter, as the agent’s own uid
Cost of an idle agent	Disk only; no pod, no requests
Cost of a new agent	`mkdir` on the first turn, seconds
Scaling axis	Concurrent turns, never the number of agents
Model credentials	None in the pod; every model call goes through the Agent Controller’s `/llm-proxy`

Per Executor

Resource	xpander cloud reference	Self-hosted chart default
Executor CPU request / limit	12 / 12 vCPU (Guaranteed)	2 / 8 vCPU
Executor memory request / limit	100 / 100 GiB (Guaranteed)	8 / 28 GiB
Mounter sidecar	200m / 256 Mi (requests = limits)	50m..500m / 64..256 Mi
Concurrent turn slots (`turnsMax`)	10	12
Volume	500 GiB, XFS, `prjquota`, gp3 encrypted, RWO	500 GiB, same class shape
Per-agent disk quota (`quotaPerAgent`)	5 GiB (XFS project quota, hard)	5 GiB
`/dev/shm` shared by the turns (memory-backed, for Chromium)	4 GiB	4 GiB
Package cache (`emptyDir`, node disk)	20 GiB	20 GiB
Termination grace	7500 s pod grace, a ceiling: the executor first drains running turns for up to 3600 s	7500 s (`fleet.executor.terminationGraceSeconds`), drain 3600 s (`fleet.executor.drainSeconds`)
Probes	readiness 10 s x 6, startup 5 s x 36 (3 min)	same
Placement	One executor per node (required anti-affinity), memory-rich `r6i.4xlarge` tier	One per node; `fleet.nodeSelector` / `fleet.tolerations`

Size the executor’s memory limit against turnsMax. Every turn gets its own cgroup with an 8 GiB ceiling by default, and the turn cgroups sit under the executor’s limit: if the sum exceeds it, the kernel OOM-kills one turn instead of the pod. The cloud reference keeps 10 x 8 GiB = 80 GiB under a 100 GiB limit. The chart default (12 slots under 28 GiB) is a conservative starting footprint that assumes typical turns stay well below their ceiling; for a production fleet either raise fleet.resources.limits.memory toward turnsMax x 8 GiB or lower turnsMax.

# Hybrid (xpander chart); on Air-Gapped the same keys live under global.fleet
fleet:
  replicas: 1
  turnsMax: 12
  orgMaxTurns: 12
  executor:
    drainSeconds: 3600              # a roll waits this long for running turns
    terminationGraceSeconds: 7500   # pod grace; keep drainSeconds at least 120 s below it
  resources:
    requests: { cpu: "2", memory: "8Gi" }
    limits:   { cpu: "8", memory: "28Gi" }
  storage:
    className: fleet-xfs
    createClass: true          # false on Air-Gapped: bring your own XFS + prjquota class
    provisioner: ebs.csi.aws.com
    size: 500Gi
    quotaPerAgent: 5Gi
  shm:   { size: 4Gi }
  cache: { size: 20Gi }

Per Turn

Each turn runs in its own cgroup. These are the ceilings the executor applies. An agent’s owner can raise them for that agent under Agent settings, up to xpander’s caps.

Runner	Memory	CPU	Processes	Notes
Agent turn (Claude Code / Codex / OpenCode)	8 GiB	4 vCPU	1024 pids	Sized for real `node` / `python` builds inside the turn
Built-in xpander runner	512 MiB	1 vCPU	64 pids	Lightweight; no CLI process
Per-agent override, hard cap	24 GiB	8 vCPU	8192 pids	Floor 128 MiB / 0.1 vCPU
Disk per agent	5 GiB			XFS project quota, hard limit; a `du`-based sweeper enforces it softly where `prjquota` is missing

The comment above the fleet block in both charts’ values.yaml still quotes older per-turn defaults (“2Gi / 2 vCPU / 256 pids for a CLI turn”). The values in the table are what the current agent-harness image applies; the chart comment is being corrected.

Concurrency Caps

Cap	Where	Default
Turns one executor accepts	`fleet.turnsMax` (`XPANDER_FLEET_TURNS_MAX`)	12 in the chart, 10 in cloud
Turns one organization may hold across the fleet	`fleet.orgMaxTurns` (`FLEET_ORG_MAX_TURNS`)	12 in the chart, 8 in cloud; on a sealed install the license’s agent cap wins when it is set
Turns above capacity	Queue in Redis and start as slots free	Nothing is dropped, only delayed

Reading the runtime logs

Two logs carry one turn: the executor that ran it and the Agent Controller that dispatched it. Follow both while you send a message:

kubectl -n xpander logs -f fleet-executor-0 -c executor
kubectl -n xpander logs -f deploy/agent-controller

Executor (fleet-executor-<n>, container executor). One turn logs these events, in this order:

Event	Carries
`harness.fleet.turn_start`	The turn was claimed
`harness.cli.selected`	The CLI and where the choice came from, for example `cli=claude-code source=task`
`execution.runtime_selected`
`harness.sandbox.open`	The sandbox’s `uid` and `cgroup`
`harness.skills.sync`	`synced=` and `exposed=`, the skills placed in the sandbox
`harness.mcp_remote.resolved`	`servers=` and `tools=`, the MCP endpoints resolved for the turn and how many actions they expose
`harness.run.start`, `harness.cli.spawn`, `harness.cli.init`	The CLI process is up
`harness.cli.tool_call`, `harness.mcp.call`, `harness.mcp.result`, `harness.cli.tool_result`	One group per skill call the agent makes
`harness.cli.exit`	The CLI finished
`harness.sandbox.close`	`cpu_usec`, the turn’s CPU time
`harness.run.end`, `harness.fleet.turn_end`	The `outcome`

Agent Controller (deploy/agent-controller). The same turn shows as:

direct turn conversation=<id> task=<id> status=... received_to_enqueued_ms=... enqueued_to_claimed_ms=...: the message became a task and was picked up by the fleet; enqueued_to_claimed_ms is where a cold executor shows. The /llm-proxy lines below name the model that served the turn (<host>:<port>/<model id> for a custom provider), the model the conversation picked; the agent record’s model is only the default for new conversations.
event=fleet.claimed executor=fleet-executor-0: which executor took it.
POST /llm-proxy/<provider>/model/<model>/invoke-with-response-stream: one line per model call the CLI made. The path names the provider and model that served the turn.
event=compute_usage.emitted cli=claude-code cpu_seconds=...: the compute record written when the turn ended.

Node Prerequisites

The installer (install.sh) and the Air-Gapped preflight hook check all of these and warn (non-blocking) when one is missing; the executor then fails to schedule and the fleet has no live executor. A cluster that fails any one of them cannot run agent turns until it is fixed, so treat the list as required.

Requirement	Why	How the installer checks
amd64 / x86_64 nodes	The `agent-harness` image is amd64 only	Same rule as the rest of xpander
cgroup v2	Per-turn `memory.max`, `cpu.max`, `pids.max` are written into a delegated cgroup subtree	Kubelet >= 1.25 on every node is the proxy check (xpander already requires Kubernetes >= 1.28)
Privileged pods allowed in the release namespace	The executor needs a writable cgroup filesystem and root for `bwrap` + `setpriv`; the mounter needs `CAP_SYS_ADMIN` for bind mounts. Agent code itself is confined by bubblewrap, its own uid, the cgroup and a seccomp allow-list. Not needed in the unprivileged mode	Pod Security Admission `enforce` label unset or `privileged` on the namespace
A StorageClass that formats XFS and mounts it with `prjquota`	Hard 5 GiB per-agent quotas	Hybrid: `fleet.storage.createClass: true` (default) renders `fleet-xfs` for the EBS CSI driver (`ebs.csi.aws.com`, gp3, encrypted, XFS, `prjquota`; on EKS Auto Mode set `provisioner: ebs.csi.eks.amazonaws.com`, see EKS Auto Mode). Air-Gapped: `global.fleet.storage.createClass: false` by default, so bring a class that mounts XFS with `prjquota`, or set it to `true` on EKS. Without `prjquota` the sweeper enforces quotas softly
RWO block storage, 500 GiB per executor	One volume per executor, via `volumeClaimTemplates`	The class exists and provisions
KEDA (optional)	Autoscales the StatefulSet on CPU	Only with `fleet.keda.enabled`; KEDA is not bundled. Air-Gapped mirrors its images with `image_sync.sh --with-keda` and installs it first
Pinned image	Both fleet containers run `agent-harness`	Air-Gapped render validation refuses an unpinned `global.images.agent-harness` tag or digest
Memory-rich nodes	Turn memory dominates; CPU is bursty	`r`-class (1:8 vCPU:GiB) fits the profile

Running the executor fleet without privileged containers

The executor fleet can run with no privileged container. Set fleet.isolation: userns on Hybrid, or global.fleet.isolation: userns on Air-Gapped. The default stays privileged, and nothing changes until you set it. In this mode the executor is a Kubernetes user-namespace pod (hostUsers: false): root inside the pod is an unprivileged user on the node. It is a single container, with no mounter. Each agent’s work still runs in its own sandbox, as the agent’s own user, and one agent cannot see another’s files. Agent traffic leaves the executor only through a filtering proxy. The proxy refuses the cloud metadata endpoint and private addresses. It allows the hosts the platform needs, such as the Agent Controller and your package mirrors. To let agents reach an internal host such as your git server, add it to fleet.egress.allowHosts. A corporate proxy set in HTTPS_PROXY on the executor is used for outbound traffic.

Security profile

Pick one of two pod profiles with fleet.seccompProfile.type:

Profile	Pod security context	Namespace Pod Security level
`RuntimeDefault` (default)	The runtime’s default seccomp profile, plus `SYS_ADMIN`, which only applies inside the pod’s own user namespace	`privileged`
`Localhost`	The seccomp profile shipped with the chart, and no added capabilities	`baseline`

The Localhost profile must be present on every node that runs an executor. Install it in one of two ways:

Security Profiles Operator: set fleet.seccompProfile.operator: true and the chart creates the profile for you.
Node file: place the chart’s profile at /var/lib/kubelet/seccomp/xpander-fleet-pod.json on each node.

# Hybrid (xpander chart); on Air-Gapped the same keys live under global.fleet
fleet:
  isolation: userns
  seccompProfile:
    type: Localhost          # or RuntimeDefault
    operator: false          # true with the Security Profiles Operator installed
  egress:
    allowHosts: []           # internal hosts agents may reach, for example git.corp.internal
  runtimeClassName: ""       # optional, see "Limits" below

Requirements

Requirement	Why
Kubernetes 1.33 or later	User namespaces are on by default from 1.33
User namespaces enabled on the nodes	`user.max_user_namespaces` must be above 0. Bottlerocket ships `0`: set `settings.kernel.sysctl."user.max_user_namespaces"` in the node’s user data
Any ReadWriteOnce StorageClass	No XFS class is needed: per-agent disk limits are enforced by the platform

The installer and the Air-Gapped preflight check these before the executor is rolled out.

Limits

Per-turn memory and process limits are enforced by the platform: a turn that goes over its ceiling is stopped on its own, and the rest of the executor keeps running. If your nodes offer a runtime class that makes the pod’s cgroup writable (containerd’s cgroup_writable option), set fleet.runtimeClassName to it. The kernel then enforces the limits exactly as in the default mode.

Autoscaling

Scaling is driven by concurrent turns, never by the number of agents. With fleet.keda.enabled: true the chart renders a KEDA ScaledObject on the StatefulSet:

Knob	Chart default	xpander cloud
`minReplicas`	1 (never scales to zero; the first turn is warm)	1
`maxReplicas`	12	3 (bounded by the node count of the executor tier)
Trigger	CPU utilization 70 %	CPU 70 % plus queue depth from the controller’s `/fleet/capacity` (pending + active turns, one executor per 10)
Cooldown	300 s	300 s, scale-down one executor per 5 minutes so in-flight turns drain

Because there is exactly one executor per node, maxReplicas can never usefully exceed the number of nodes that fit an executor. Grow the node group and raise maxReplicas together. Without KEDA, set fleet.replicas to the number of executors you want and scale it by hand; on a sealed network a fixed replica count sized for peak is the recommended posture.

Safe Rollout, Drain, and Upgrades

A roll never kills a running turn

A rolling update, a KEDA scale-in and a plain pod delete all end in the same SIGTERM to the executor pod, and the pod treats it as a drain, not a kill:

The executor stops claiming work, and its heartbeat says draining. The Agent Controller routes around it: nothing new lands on it, and whatever was queued for it goes back to the shared queue. It still counts as fleet presence, so new turns wait for the replacement instead of starting anywhere else.
The turns in flight keep running for up to fleet.executor.drainSeconds (3600 s by default). A turn still running when that window ends is stopped the way a user stop is: the CLI is interrupted, its session is kept, and the task ends stopped with “The executor is restarting; send the message again to continue where it left off.” Sending the message again resumes the session on the new pod. No task is left hanging as executing.
The mounter sidecar waits on the executor’s health endpoint until it reports draining with no turn in flight, and only then unbinds the agent directories. The pod grace (fleet.executor.terminationGraceSeconds, 7500 s: the CLI run budget plus five minutes) is a ceiling, never a wait; the pod leaves as soon as both containers exit.

A StatefulSet roll replaces one pod at a time, so the replacement for a busy executor appears only after its turns end: a roll during a long turn takes as long as that turn, up to the drain window. KEDA scale-in goes through the same drain and removes at most one executor per five minutes. Operator rules that follow from this:

Raise drainSeconds toward the run budget if your turns run long; it must stay at least 120 s under the pod grace (the Air-Gapped chart refuses to render otherwise).
Never shorten a termination with --grace-period=0 or --force on a pod that still has turns; that is the one way to kill a turn.
Plan image rolls for quiet hours when you can; they are safe at any time, only slower under load.
The drain needs an agent-harness image that carries it; an older executor ignores the setting and exits as before.

Idle guard when the StorageClass is missing

The executor becomes Ready only when its node can run the privileged pod and its volume binds on an XFS + prjquota StorageClass. So that an install or upgrade never waits on an executor that cannot bind, both charts guard themselves on a live cluster:

Situation	What the chart renders
`fleet.storage.createClass: false` and no StorageClass named `fleet.storage.className` exists	the executor StatefulSet with `replicas: 0`, annotated `xpander.ai/fleet-idle-reason`
`fleet.storage.createClass: true` but no `CSIDriver` named `fleet.storage.provisioner` is installed (the rendered class could never provision, the non-AWS cluster case)	same: idle, with the reason on the annotation
the class exists (or the driver is present for the rendered class)	the configured `replicas`

helm template and dry-runs always render the configured replicas; the guard reads the live cluster only. Create the class (or point provisioner at your cluster’s CSI driver, or bring your own class) and the next upgrade brings the executor up. Nothing else about the release changes while the executor idles: turns queue until an executor registers.

`--wait`, `--atomic` and the installer

helm upgrade --wait (and --atomic, which implies it) waits for the executor StatefulSet like any other workload. Two consequences:

A running turn holds the roll. With --wait, an upgrade during a long turn can take up to the drain window plus pod start. Give --timeout at least the drain window (3600 s by default) or upgrade in a quiet period.
An idle executor never stalls an upgrade. With the idle guard above, a cluster without the storage prerequisites renders the executor at zero replicas and the upgrade completes.

install.sh handles both for you: a first install runs without --wait or --atomic (the post-install smoke hook is the readiness gate) and its --fleet checks only warn; upgrades run helm upgrade --atomic with the installer’s timeout and then verify workloads, skipping zero-replica ones.

Capacity Planning

Use the cloud reference executor (12 vCPU / 100 GiB, 10 slots, one per 16 vCPU / 128 GiB node) as the unit. It keeps every turn’s 8 GiB ceiling fully backed, and the roughly 28 GiB of node headroom covers the kubelet, DaemonSets, the mounter, the 4 GiB /dev/shm and the 20 GiB package cache on node disk.

Peak concurrent turns	Executors (10 slots each)	Nodes (16 vCPU / 128 GiB each, e.g. `r6i.4xlarge`)	Fleet volumes	Agents the volumes hold at full 5 GiB quota	Also raise
10	1	1	1 x 500 GiB	~90	nothing; `orgMaxTurns` 12 already covers it
50	5	5	5 x 500 GiB = 2.5 TiB	~450	`fleet.keda.maxReplicas: 5` (or `replicas: 5`), `orgMaxTurns` >= 50 if one org drives the load
200	20	20	20 x 500 GiB = 10 TiB	~1800	`maxReplicas: 20`, `orgMaxTurns` >= 200, and a node autoscaler or a fixed 20-node group

How to read the table:

Slots, not agents. A fleet with one executor serves hundreds of idle agents; what it cannot do is run more than 10 (or turnsMax) turns at once. Extra turns queue and start as slots free.
Agents per volume is a ceiling, not a target. Most agent directories stay far below the 5 GiB quota, so a 500 GiB volume holds many more than 90 agents in practice; it also carries the shared read-only CLI, skills and runtime environment.
Memory headroom. Keep the executor’s memory limit at or below about 80 % of the node so the node never evicts it, and keep turnsMax x 8 GiB at or below that limit if you want every turn’s ceiling backed.
Smaller executors are fine. On the chart defaults (8 vCPU / 28 GiB limit, 12 slots) one executor fits on a 16 vCPU / 32 GiB node, at the price described in the warning above. Lower turnsMax to 3 or 4 there if your turns run heavy builds.
Workspace pods are separate. Agents that also use a workspace still add their sb-* pod while active (0.25 vCPU / 256 Mi request, up to 2 vCPU / 3 GiB, 5 GiB volume); size those from the Hybrid page’s workspace accordion.

Data services and storage

The chart deploys single-replica Redis and PostgreSQL StatefulSets and a container registry alongside xpander:

Component	Persistent volume (default)	Guidance
Redis	8 GiB	Queues, caches, and runtime state. 2-4 GiB of memory is sufficient for a single organization; give the pod a 512 Mi-1 GiB memory request
PostgreSQL	20 GiB	xpander state; grows slowly
Container registry	20 GiB	Stores agent container images built in your cluster

Baseline storage: 48 GiB of persistent volumes, plus 5 GiB per workspace-using agent (see above). All volumes use your cluster’s default StorageClass unless overridden. Production checklist:

Node types: x86_64 only. General-purpose instances with a 1:4 vCPU:GiB ratio (m5, m6i) fit the workload profile; t3 burstable instances are fine for evaluation, not for production.
Raise the low defaults: API memory (table above) before real load.
Enable autoscaling for the execution tier (agent-worker) once concurrency grows beyond a couple of replicas.
Keep headroom: pod requests should stay under ~65% of cluster capacity so node upgrades and bursts don’t evict workloads.
Storage class: SSD-backed volumes for Redis and PostgreSQL.

Image and Registry

Item	Value
Image	`agent-harness` (Docker Hub `xpanderaihub/agent-harness`, mirrored into your registry on Air-Gapped)
Compressed size	~1.5 GB (amd64)
On node disk	Plan ~4 GiB for the extracted image
Contents	Claude Code, Codex, OpenCode, Playwright Chromium, Node 22, Python 3.12, git, plus `kubectl`, `helm`, `aws`, `terraform`, `gh`, `docker` client and other developer utilities
Pull policy	Executors pull on every roll (`Always`); pin the tag or digest so a roll is deliberate

Both fleet containers use this one image. On a fresh node the first executor pays the full pull, so budget the ~1.5 GB registry transfer per executor node and, on Air-Gapped, make sure the mirror holds it at the pinned tag before the first executor starts.

Network Egress

Executors hold no LLM provider key. The only LLM path is the Agent Controller’s /llm-proxy, reached over the cluster network, and the controller is the only component that egresses to a model provider.

From	To	When
Executor	Agent Controller (`/llm-proxy`, registration, heartbeat, claim)	Always, in-cluster (`fleet.controllerUrl`, default the in-cluster Service)
Agent Controller	`api.anthropic.com`, `bedrock-runtime.<region>.amazonaws.com`, or your OpenAI-compatible endpoint	Every model call, on behalf of the CLIs
Executor	`registry.npmjs.org` (or your `XPANDER_NPM_REGISTRY` mirror)	Only when CLI auto-update is on (`cli_auto_update`, default on). The preinstalled CLIs need no egress; Air-Gapped uses a sealed CLI bundle instead
A turn’s own commands	npm / PyPI / apt / Homebrew, or their mirrors	Only when an agent’s runtime environment or the agent itself installs packages; follows the install’s package-mirror contract
Executor (NetworkPolicy)	DNS, the release namespace, the internet minus private ranges and the cloud metadata endpoint	The chart renders this policy; the internet leg is dropped with `global.network.internetEgress: false`

The executor also blackholes the instance metadata endpoint (169.254.169.254) at boot and refuses to register if it cannot, so a turn can never read node credentials.

Model providers

Executors hold no provider key; the Agent Controller does. Which key each provider needs on a self-hosted install, the Bedrock authentication methods, and which harness and model a defaulted agent gets are on AI vendors.

Air-Gapped vs Hybrid

	Hybrid	Air-Gapped
Fleet values	`fleet.*`, image `images.agentHarness`	`global.fleet.*`, image `global.images.agent-harness` from your mirror, tag or digest pinned
Starting executors	`fleet.replicas: 1`	`global.fleet.replicas: 1`
Fleet token	Generated into the chart’s Secret, or `fleet.tokenSecret.existing`	One more key in `xpander-generated-keys`
StorageClass	`createClass: true` renders the EBS gp3 XFS class (`ebs.csi.aws.com`; `ebs.csi.eks.amazonaws.com` on EKS Auto Mode)	`createClass: false`; bring a class that mounts XFS with `prjquota`, or set it to `true` on EKS
KEDA	Install `kedacore/keda` yourself, then `fleet.keda.enabled`	Mirror with `image_sync.sh --with-keda`, install, then `global.fleet.keda.enabled`
CLI updates	Executors update from the npm registry or your `XPANDER_NPM_REGISTRY` mirror	Sealed CLI bundle imported from a PVC (`global.fleet.cliBundle`)
LLM	Controller egress to Anthropic / Bedrock / your endpoint; defaulted agents follow the org default LLM, else Anthropic direct (no Bedrock relay; Bedrock needs an org key)	In-network endpoint only; VPC Bedrock via `global.llm.bedrock.baseUrl` (Pod Identity, assume role or bearer token); defaulted agents follow the org default LLM, then `global.harness.defaultModels` / `defaultProviders`
Upgrades	`install.sh upgrade` runs `--atomic`; a running turn holds the roll up to `fleet.executor.drainSeconds`; the idle guard covers a missing class	same, under `global.fleet.executor`; the first install runs without `--wait`
Seat cap	`orgMaxTurns`	The license’s agent cap when set, else `orgMaxTurns`
Recommended posture	`minReplicas: 1`, KEDA on	Fixed `replicas` sized for peak (static capacity, no autoscaler), KEDA optional

Recommended for both self-hosted deployments: keep the fleet warm (at least one executor always up). The first turn of any agent is then seconds, and idle agents cost only their directory. A cluster that cannot meet the prerequisites should set replicas: 0 (or let the idle guard do it) and fix the missing capability before agents run on it, since every agent’s turns land on the fleet.

Next Steps

Hybrid deployment

The installer, PrivateLink, LLM keys and upgrades

Air-gapped deployment

Prerequisites, registry mirroring, and the install pipeline

EKS Cluster Setup

Node groups, the EBS CSI driver, and StorageClasses

Two-File Licensing

How seat and agent caps are enforced on a sealed install

Govern

Deploy

Integrations

Licensing and identity

Security and compliance

Sizing and requirements

Base footprint

Hybrid

Sizing calculator

Air-Gapped

Sizing tiers

Scaling with usage

Agent workspaces

The executor fleet

Per Executor

Per Turn

Concurrency Caps

Reading the runtime logs

Node Prerequisites

Running the executor fleet without privileged containers

Security profile

Requirements

Limits

Autoscaling

Safe Rollout, Drain, and Upgrades

A roll never kills a running turn

Idle guard when the StorageClass is missing

`--wait`, `--atomic` and the installer

Capacity Planning

Data services and storage

Image and Registry

Network Egress

Model providers

Air-Gapped vs Hybrid

Next Steps

Hybrid deployment

Air-gapped deployment

EKS Cluster Setup

Two-File Licensing

​Base footprint

​Hybrid

​Sizing calculator

​Air-Gapped

​Sizing tiers

​Scaling with usage

​Agent workspaces

​The executor fleet

​Per Executor

​Per Turn

​Concurrency Caps

​Reading the runtime logs

​Node Prerequisites

​Running the executor fleet without privileged containers

​Security profile

​Requirements

​Limits

​Autoscaling

​Safe Rollout, Drain, and Upgrades

​A roll never kills a running turn

​Idle guard when the StorageClass is missing

​--wait, --atomic and the installer

​Capacity Planning

​Data services and storage

​Image and Registry

​Network Egress

​Model providers

​Air-Gapped vs Hybrid

​Next Steps

Hybrid deployment

Air-gapped deployment

EKS Cluster Setup

Two-File Licensing

Base footprint

Hybrid

Sizing calculator

Air-Gapped

Sizing tiers

Scaling with usage

Agent workspaces

The executor fleet

Per Executor

Per Turn

Concurrency Caps

Reading the runtime logs

Node Prerequisites

Running the executor fleet without privileged containers

Security profile

Requirements

Limits

Autoscaling

Safe Rollout, Drain, and Upgrades

A roll never kills a running turn

Idle guard when the StorageClass is missing

`--wait`, `--atomic` and the installer

Capacity Planning

Data services and storage

Image and Registry

Network Egress

Model providers

Air-Gapped vs Hybrid

Next Steps