> ## Documentation Index
> Fetch the complete documentation index at: https://docs.xpander.ai/llms.txt
> Use this file to discover all available pages before exploring further.
# Harness Agents: Sizing and Requirements
> Hardware and cluster requirements for harness agents (Claude Code, Codex, OpenCode) on a self-hosted install: the executor fleet, per-turn ceilings, volume and quota, node prerequisites, autoscaling, capacity planning, image and egress needs, and LLM provider setup
A **harness agent** is an agent whose framework is an external coding CLI: Claude Code, Codex, or OpenCode. Instead of running on the platform's `agent-worker`, its turns run the real CLI inside the `agent-harness` image, which carries all three CLIs, a headless Chromium, and a developer toolchain. Harness agents run on the **executor fleet**: a shared pool of warm executor pods where an agent is a directory on a large XFS volume and every turn runs in its own throwaway sandbox. That fleet has cluster prerequisites and a footprint of its own; this page gives the numbers so you can size a Hybrid or Air-Gapped cluster for it.
The base platform footprint is on the edition pages ([Hybrid](/self-hosted/hybrid#requirements--sizing), [Air-Gapped](/self-hosted/airgap#prerequisites)). Everything here is **on top of** those numbers.
The fleet is **always rendered** by both charts (there is no enable flag) and starts warm with one executor (`fleet.replicas: 1` on Hybrid, `global.fleet.replicas: 1` on Air-Gapped). The Agent Controller sends a harness turn to the fleet whenever a live executor is registered. Keep it warm: the first turn of any agent is then seconds away, and idle agents cost only their directory. Where a cluster cannot host the privileged executor, set `replicas: 0`; on a live cluster the chart also idles the executor by itself when its StorageClass is missing (see [Idle guard](#idle-guard-when-the-storageclass-is-missing)).
**One per-agent pod remains, and it is not a harness pod.** Agents that use workspace tools still get an on-demand `sb-*` workspace pod (`agent-sandbox` image); its sizing is unchanged and lives in the [Agent workspaces](/self-hosted/hybrid#requirements--sizing) accordion on the Hybrid page. The older always-on `ag-*` harness pod, one per Claude Code / Codex agent with its own 40 GiB volume, is the legacy path being removed; do not size new clusters for it.
**Every new agent is a harness agent.** A new conversation agent, whether created in the app, over the API or the SDK, or built by Omni, runs on a harness CLI unless the request names another framework, so plan fleet capacity for your whole agent population, not only for the agents you explicitly put on a CLI. Existing agents keep the runtime they have until someone moves them, and that move is one-way; what carries over and what does not is on [Apps and Agents](/guides/omni/manage-agents/agent-configuration#harness-runtime).
***
## What the Fleet Is
The fleet is a `fleet-executor` StatefulSet. Each replica is one pod with two privileged containers over one XFS volume: the **executor**, which runs turns, and a **mounter** sidecar, which binds each agent's directory into the executor for the duration of a turn and enforces the per-agent disk quota. An agent's directory lives on exactly one executor's volume, and its turns always run there.
| | Executor fleet |
| --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| What an agent is | A directory (its home, CLI state, working tree) on the executor's XFS volume, under a 5 GiB quota |
| What runs a turn | A free **slot** on the executor that holds the agent's directory, inside a bubblewrap sandbox with its own cgroup and seccomp filter, as the agent's own uid |
| Cost of an idle agent | Disk only; no pod, no requests |
| Cost of a new agent | `mkdir` on the first turn, seconds |
| Scaling axis | **Concurrent turns**, never the number of agents |
| Model credentials | None in the pod; every model call goes through the Agent Controller's `/llm-proxy` |
***
## Per Executor
| Resource | xpander cloud reference | Self-hosted chart default |
| ------------------------------------------------------------ | ----------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------- |
| Executor CPU request / limit | 12 / 12 vCPU (Guaranteed) | **2 / 8 vCPU** |
| Executor memory request / limit | 100 / 100 GiB (Guaranteed) | **8 / 28 GiB** |
| Mounter sidecar | 200m / 256 Mi (requests = limits) | 50m..500m / 64..256 Mi |
| Concurrent turn slots (`turnsMax`) | 10 | **12** |
| Volume | 500 GiB, XFS, `prjquota`, gp3 encrypted, RWO | **500 GiB**, same class shape |
| Per-agent disk quota (`quotaPerAgent`) | 5 GiB (XFS project quota, hard) | 5 GiB |
| `/dev/shm` shared by the turns (memory-backed, for Chromium) | 4 GiB | 4 GiB |
| Package cache (`emptyDir`, node disk) | 20 GiB | 20 GiB |
| Termination grace | 7500 s pod grace, a ceiling: the executor first **drains** running turns for up to 3600 s | 7500 s (`fleet.executor.terminationGraceSeconds`), drain 3600 s (`fleet.executor.drainSeconds`) |
| Probes | readiness 10 s x 6, startup 5 s x 36 (3 min) | same |
| Placement | One executor per node (required anti-affinity), memory-rich `r6i.4xlarge` tier | One per node; `fleet.nodeSelector` / `fleet.tolerations` |
Size the executor's **memory limit against `turnsMax`**. Every turn gets its own cgroup with an 8 GiB ceiling by default, and the turn cgroups sit under the executor's limit: if the sum exceeds it, the kernel OOM-kills one turn instead of the pod. The cloud reference keeps `10 x 8 GiB = 80 GiB` under a 100 GiB limit. The chart default (12 slots under 28 GiB) is a conservative starting footprint that assumes typical turns stay well below their ceiling; for a production fleet either raise `fleet.resources.limits.memory` toward `turnsMax x 8 GiB` or lower `turnsMax`.
```yaml theme={"dark"}
# Hybrid (xpander chart); on Air-Gapped the same keys live under global.fleet
fleet:
replicas: 1
turnsMax: 12
orgMaxTurns: 12
executor:
drainSeconds: 3600 # a roll waits this long for running turns
terminationGraceSeconds: 7500 # pod grace; keep drainSeconds at least 120 s below it
resources:
requests: { cpu: "2", memory: "8Gi" }
limits: { cpu: "8", memory: "28Gi" }
storage:
className: fleet-xfs
createClass: true # false on Air-Gapped: bring your own XFS + prjquota class
provisioner: ebs.csi.aws.com
size: 500Gi
quotaPerAgent: 5Gi
shm: { size: 4Gi }
cache: { size: 20Gi }
```
***
## Per Turn
Each turn runs in its own cgroup. These are the ceilings the executor applies; an agent can raise its own in its harness settings, clamped to the platform caps.
| Runner | Memory | CPU | Processes | Notes |
| ----------------------------------------- | --------- | ---------- | --------- | -------------------------------------------------------------------------------------------------- |
| CLI turn (Claude Code / Codex / OpenCode) | **8 GiB** | **4 vCPU** | 1024 pids | Sized for real `node` / `python` builds inside the turn |
| Built-in xpander runner | 512 MiB | 1 vCPU | 64 pids | Lightweight; no CLI process |
| Per-agent override, hard cap | 24 GiB | 8 vCPU | 8192 pids | Floor 128 MiB / 0.1 vCPU |
| Disk per agent | 5 GiB | | | XFS project quota, hard limit; a `du`-based sweeper enforces it softly where `prjquota` is missing |
The comment above the `fleet` block in both charts' `values.yaml` still quotes older per-turn defaults ("2Gi / 2 vCPU / 256 pids for a CLI turn"). The values in the table are what the current `agent-harness` image applies; the chart comment is being corrected.
### Concurrency Caps
| Cap | Where | Default |
| ------------------------------------------------ | -------------------------------------------- | -------------------------------------------------------------------------------------------- |
| Turns one executor accepts | `fleet.turnsMax` (`XPANDER_FLEET_TURNS_MAX`) | 12 in the chart, 10 in cloud |
| Turns one organization may hold across the fleet | `fleet.orgMaxTurns` (`FLEET_ORG_MAX_TURNS`) | 12 in the chart, 8 in cloud; on a sealed install the license's agent cap wins when it is set |
| Turns above capacity | Queue in Redis and start as slots free | Nothing is dropped, only delayed |
***
## Node Prerequisites
The installer (`install.sh`) and the Air-Gapped preflight hook check all of these and **warn** (non-blocking) when one is missing; the executor then fails to schedule and the fleet has no live executor. A cluster that fails any one of them cannot run harness agents until it is fixed, so treat the list as required.
| Requirement | Why | How the installer checks |
| ----------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **amd64 / x86\_64 nodes** | The `agent-harness` image is amd64 only | Same rule as the rest of the platform |
| **cgroup v2** | Per-turn `memory.max`, `cpu.max`, `pids.max` are written into a delegated cgroup subtree | Kubelet **>= 1.25** on every node is the proxy check (the platform already requires Kubernetes >= 1.28) |
| **Privileged pods allowed in the release namespace** | The executor needs a writable cgroup filesystem and root for `bwrap` + `setpriv`; the mounter needs `CAP_SYS_ADMIN` for bind mounts. Agent code itself is confined by bubblewrap, its own uid, the cgroup and a seccomp allow-list | Pod Security Admission `enforce` label unset or `privileged` on the namespace |
| **A StorageClass that formats XFS and mounts it with `prjquota`** | Hard 5 GiB per-agent quotas | Hybrid: `fleet.storage.createClass: true` (default) renders `fleet-xfs` for the EBS CSI driver (`ebs.csi.aws.com`, gp3, encrypted, XFS, `prjquota`). Air-Gapped: `global.fleet.storage.createClass: false` by default, so bring a class that mounts XFS with `prjquota`, or set it to `true` on EKS. Without `prjquota` the sweeper enforces quotas softly |
| **RWO block storage, 500 GiB per executor** | One volume per executor, via `volumeClaimTemplates` | The class exists and provisions |
| **KEDA** (optional) | Autoscales the StatefulSet on CPU | Only with `fleet.keda.enabled`; KEDA is not bundled. Air-Gapped mirrors its images with `image_sync.sh --with-keda` and installs it first |
| **Pinned image** | Both fleet containers run `agent-harness` | Air-Gapped render validation refuses an unpinned `global.images.agent-harness` tag or digest |
| **Memory-rich nodes** | Turn memory dominates; CPU is bursty | `r`-class (1:8 vCPU:GiB) fits the profile |
***
## Autoscaling
Scaling is driven by **concurrent turns**, never by the number of agents. With `fleet.keda.enabled: true` the chart renders a KEDA `ScaledObject` on the StatefulSet:
| Knob | Chart default | xpander cloud |
| ------------- | ------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------- |
| `minReplicas` | 1 (never scales to zero; the first turn is warm) | 1 |
| `maxReplicas` | 12 | 3 (bounded by the node count of the executor tier) |
| Trigger | CPU utilization 70 % | CPU 70 % **plus** queue depth from the controller's `/fleet/capacity` (pending + active turns, one executor per 10) |
| Cooldown | 300 s | 300 s, scale-down one executor per 5 minutes so in-flight turns drain |
Because there is exactly one executor per node, `maxReplicas` can never usefully exceed the number of nodes that fit an executor. Grow the node group and raise `maxReplicas` together. Without KEDA, set `fleet.replicas` to the number of executors you want and scale it by hand; on a sealed network a fixed replica count sized for peak is the recommended posture.
***
## Safe Rollout, Drain, and Upgrades
### A roll never kills a running turn
A rolling update, a KEDA scale-in and a plain pod delete all end in the same `SIGTERM` to the executor pod, and the pod treats it as a **drain**, not a kill:
1. The executor stops claiming work and its heartbeat says `draining`. The Agent Controller routes around it: nothing new lands on it, whatever was queued for it goes back to the shared queue, and it still counts as fleet presence, so new turns wait for the replacement instead of starting anywhere else.
2. The turns in flight keep running for up to `fleet.executor.drainSeconds` (3600 s by default). A turn still running when that window ends is stopped the way a user stop is: the CLI is interrupted, its session is kept, and the task ends `stopped` with "The executor is restarting; send the message again to continue where it left off." Sending the message again resumes the session on the new pod. No task is left hanging as executing.
3. The mounter sidecar waits on the executor's health endpoint until it reports draining with no turn in flight, and only then unbinds the agent directories. The pod grace (`fleet.executor.terminationGraceSeconds`, 7500 s: the CLI run budget plus five minutes) is a ceiling, never a wait; the pod leaves as soon as both containers exit.
A StatefulSet roll replaces one pod at a time, so the replacement for a busy executor appears only after its turns end: **a roll during a long turn takes as long as that turn, up to the drain window.** KEDA scale-in goes through the same drain and removes at most one executor per five minutes. Operator rules that follow from this:
* Raise `drainSeconds` toward the run budget if your turns run long; it must stay at least 120 s under the pod grace (the Air-Gapped chart refuses to render otherwise).
* Never shorten a termination with `--grace-period=0` or `--force` on a pod that still has turns; that is the one way to kill a turn.
* Plan image rolls for quiet hours when you can; they are safe at any time, only slower under load.
* The drain needs an `agent-harness` image that carries it; an older executor ignores the setting and exits as before.
### Idle guard when the StorageClass is missing
The executor becomes Ready only when its node can run the privileged pod and its volume binds on an XFS + `prjquota` StorageClass. So that an install or upgrade never waits on an executor that cannot bind, both charts guard themselves on a **live cluster**:
| Situation | What the chart renders |
| ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------- |
| `fleet.storage.createClass: false` and no StorageClass named `fleet.storage.className` exists | the executor StatefulSet with `replicas: 0`, annotated `xpander.ai/fleet-idle-reason` |
| `fleet.storage.createClass: true` but no `CSIDriver` named `fleet.storage.provisioner` is installed (the rendered class could never provision, the non-AWS cluster case) | same: idle, with the reason on the annotation |
| the class exists (or the driver is present for the rendered class) | the configured `replicas` |
`helm template` and dry-runs always render the configured replicas; the guard reads the live cluster only. Create the class (or point `provisioner` at your cluster's CSI driver, or bring your own class) and the next upgrade brings the executor up. Nothing else about the release changes while the executor idles: harness turns queue until an executor registers.
### `--wait`, `--atomic` and the installer
`helm upgrade --wait` (and `--atomic`, which implies it) waits for the executor StatefulSet like any other workload. Two consequences:
* **A running turn holds the roll.** With `--wait`, an upgrade during a long turn can take up to the drain window plus pod start. Give `--timeout` at least the drain window (3600 s by default) or upgrade in a quiet period.
* **An idle executor never stalls an upgrade.** With the idle guard above, a cluster without the storage prerequisites renders the executor at zero replicas and the upgrade completes.
`install.sh` handles both for you: a **first install** runs without `--wait` or `--atomic` (the post-install smoke hook is the readiness gate) and its `--fleet` checks only warn; **upgrades** run `helm upgrade --atomic` with the installer's timeout and then verify workloads, skipping zero-replica ones.
***
## Capacity Planning
Use the **cloud reference executor** (12 vCPU / 100 GiB, 10 slots, one per 16 vCPU / 128 GiB node) as the unit. It keeps every turn's 8 GiB ceiling fully backed, and the roughly 28 GiB of node headroom covers the kubelet, DaemonSets, the mounter, the 4 GiB `/dev/shm` and the 20 GiB package cache on node disk.
| Peak concurrent turns | Executors (10 slots each) | Nodes (16 vCPU / 128 GiB each, e.g. `r6i.4xlarge`) | Fleet volumes | Agents the volumes hold at full 5 GiB quota | Also raise |
| --------------------- | ------------------------- | -------------------------------------------------- | --------------------- | ------------------------------------------- | ---------------------------------------------------------------------------------------------- |
| 10 | 1 | 1 | 1 x 500 GiB | \~90 | nothing; `orgMaxTurns` 12 already covers it |
| 50 | 5 | 5 | 5 x 500 GiB = 2.5 TiB | \~450 | `fleet.keda.maxReplicas: 5` (or `replicas: 5`), `orgMaxTurns` >= 50 if one org drives the load |
| 200 | 20 | 20 | 20 x 500 GiB = 10 TiB | \~1800 | `maxReplicas: 20`, `orgMaxTurns` >= 200, and a node autoscaler or a fixed 20-node group |
How to read the table:
* **Slots, not agents.** A fleet with one executor serves hundreds of idle agents; what it cannot do is run more than 10 (or `turnsMax`) turns at once. Extra turns queue and start as slots free.
* **Agents per volume** is a ceiling, not a target. Most agent directories stay far below the 5 GiB quota, so a 500 GiB volume holds many more than 90 agents in practice; it also carries the shared read-only CLI, skills and runtime-environment layers.
* **Memory headroom.** Keep the executor's memory limit at or below about 80 % of the node so the node never evicts it, and keep `turnsMax x 8 GiB` at or below that limit if you want every turn's ceiling backed.
* **Smaller executors are fine.** On the chart defaults (8 vCPU / 28 GiB limit, 12 slots) one executor fits on a 16 vCPU / 32 GiB node, at the price described in the warning above. Lower `turnsMax` to 3 or 4 there if your turns run heavy builds.
* **Workspace pods are separate.** Agents that also use workspace tools still add their `sb-*` pod while active (0.25 vCPU / 256 Mi request, up to 2 vCPU / 3 GiB, 5 GiB volume); size those from the Hybrid page's workspace accordion.
***
## Image and Registry
| Item | Value |
| --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Image | `agent-harness` (Docker Hub `xpanderaihub/agent-harness`, mirrored into your registry on Air-Gapped) |
| Compressed size | **\~1.5 GB** (amd64) |
| On node disk | Plan **\~4 GiB** for the extracted layers |
| Contents | Claude Code, Codex, OpenCode, Playwright Chromium, Node 22, Python 3.12, git, plus `kubectl`, `helm`, `aws`, `terraform`, `gh`, `docker` client and other developer tools |
| Pull policy | Executors pull on every roll (`Always`); pin the tag or digest so a roll is deliberate |
Both fleet containers use this one image. On a fresh node the first executor pays the full pull, so budget the \~1.5 GB registry transfer per executor node and, on Air-Gapped, make sure the mirror holds it at the pinned tag before the first executor starts.
***
## Network Egress
Executors hold **no LLM provider key**. The only LLM path is the Agent Controller's `/llm-proxy`, reached over the cluster network, and the controller is the only component that egresses to a model provider.
| From | To | When |
| ------------------------ | ------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ |
| Executor | Agent Controller (`/llm-proxy`, registration, heartbeat, claim) | Always, in-cluster (`fleet.controllerUrl`, default the in-cluster Service) |
| Agent Controller | `api.anthropic.com`, `bedrock-runtime..amazonaws.com`, or your OpenAI-compatible endpoint | Every model call, on behalf of the CLIs |
| Executor | `registry.npmjs.org` (or your `XPANDER_NPM_REGISTRY` mirror) | Only when **CLI auto-update** is on (`cli_auto_update`, default on). The preinstalled CLIs need no egress; Air-Gapped uses a sealed CLI bundle instead |
| A turn's own commands | npm / PyPI / apt / Homebrew, or their mirrors | Only when an agent's runtime environment or the agent itself installs packages; follows the install's package-mirror contract |
| Executor (NetworkPolicy) | DNS, the release namespace, the internet minus private ranges and the cloud metadata endpoint | The chart renders this policy; the internet leg is dropped with `global.network.internetEgress: false` |
The executor also blackholes the instance metadata endpoint (`169.254.169.254`) at boot and refuses to register if it cannot, so a turn can never read node credentials.
***
## LLM Provider Requirements
Claude Code speaks to Anthropic; Codex speaks to OpenAI; OpenCode takes any `/` slot the proxy routes. Credentials live **only on the Agent Controller**.
### Anthropic direct
An Anthropic API key on the controller (the same LLM-key wiring as the rest of the platform). The controller egresses to `api.anthropic.com`; override the upstream with `LLM_PROXY_ANTHROPIC_BASE_URL` for a gateway of your own.
### Anthropic on Amazon Bedrock
Claude Code can run Anthropic models on Bedrock through the proxy's `amazon_bedrock` leg. The controller resolves AWS auth in this order, and no executor ever sees it:
| Method | Controller environment | Notes |
| ------------------------ | -------------------------------------------------------------------------------- | -------------------------------------------------------- |
| Bearer token | `AWS_BEARER_TOKEN_BEDROCK` (+ `BEDROCK_REGION` / `AWS_REGION`) | Relayed as-is, no signing. The simplest option |
| Assume role | `BEDROCK_ASSUME_ROLE_ARN` (+ `BEDROCK_ASSUME_ROLE_EXTERNAL_ID`, `_SESSION_NAME`) | STS credentials, self-refreshing, SigV4 signed |
| Static keys | `AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY` (/ `AWS_SESSION_TOKEN`) | SigV4 signed |
| Default credential chain | `AGENT_GATEWAY_BEDROCK_DEFAULT_CHAIN=true` (default) | IRSA / EKS Pod Identity / instance profile, SigV4 signed |
On Air-Gapped there is no public Bedrock: point `LLM_PROXY_AMAZON_BEDROCK_BASE_URL` at a VPC (PrivateLink) Bedrock runtime endpoint and set the matching region, or leave Bedrock off. The Air-Gapped chart exposes all of this under `global.llm.bedrock`: `region`, `bearerTokenSecret` (a Secret you create carrying a Bedrock API key), `assumeRoleArn` and `assumeRoleExternalId` (the STS path), and `baseUrl` (the VPC endpoint the `amazon_bedrock` leg dials on a sealed install). Leave the token and the role empty to use the pod's own credential chain (EKS Pod Identity or IRSA). On Hybrid, set the same variables on the Agent Controller through the chart's env values.
**Fable- and Mythos-class Anthropic models on Bedrock need the account's data-retention opt-in, per region.** Bedrock serves those models only to accounts whose data retention mode is one Anthropic accepts (`provider_data_share`, or `aws_review` where offered); with the default mode every call fails with `400 data retention mode 'default' is not available for this model`, even though the model id resolved. There is no console switch: set it once per region with the Bedrock control-plane API, `PUT https://bedrock..amazonaws.com/data-retention` with the body `{"mode": "provider_data_share"}` (SigV4-signed, IAM action `bedrock:PutAccountDataRetention`), for every region a `global.` inference profile can route to. The first call after the change can still fail while it propagates; retry before debugging. This applies to **whichever AWS account signs the call**: the proxy prefers an organization's own stored Bedrock key over the install's credential, so an org that brings its own Bedrock key must opt that account in too. Older Anthropic models on Bedrock need no opt-in, which is why the symptom looks model-specific.
**Hybrid has no Bedrock relay.** xpander cloud relays credentials for the vendor providers only, never Bedrock, so on a Hybrid install new Claude Code agents run on the relayed Anthropic key. Claude Code on Bedrock there requires an organization-level Bedrock key added in Settings > LLM (bring your own key), on an account with the opt-in above.
The keyless (role / Pod Identity) methods above apply to the **harness LLM proxy**. The platform's own agent-worker path has a separate constraint on keyless Bedrock; see [Amazon Bedrock: key vs IAM](/self-hosted/hybrid#amazon-bedrock-key-vs-iam) on the Hybrid page.
### Which CLI, provider and model a new agent gets
A create that names a framework, a provider, a model or harness settings is honored as written. A create that names none of them is **defaulted**, in the same order everywhere (cloud, Hybrid, Air-Gapped):
1. **The organization's default LLM** (Settings > LLM). Anthropic maps to Claude Code on that model; Amazon Bedrock with an Anthropic model id maps to Claude Code on Bedrock; OpenAI maps to Codex; any other provider the LLM proxy routes maps to OpenCode with that `/` slot. A provider no CLI can run, or no org default at all, falls through.
2. **The install's default knobs** (env on the `agents` and `agent-controller` services; on Air-Gapped the chart values below).
3. **The baked defaults**: Claude Code on Anthropic direct with Fable 5.1 (`claude-fable-5-1`), Codex on `gpt-6-astra`, OpenCode on `anthropic/claude-opus-5`.
So a fresh Air-Gapped install whose organization has not set a default LLM creates Claude Code agents on **Fable 5.1 over Anthropic direct**: the install needs an Anthropic key reachable from the controller, or Amazon Bedrock (with the data-retention opt-in above) declared as the org default or through the knobs. An existing agent keeps the provider and model it already runs on; each conversation can still pick another CLI or model in the composer.
| Edition | Knob | Effect |
| ---------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Air-Gapped | `global.harness.defaultModels.{claudeCode,codex,opencode,xpander}` (rendered as `XPANDER_HARNESS_DEFAULT_MODEL_` on both `agent-controller` and the `agents` Deployment) | The per-CLI model a defaulted agent gets when the org has no default LLM. The baked vendor ids are unreachable on a sealed network, so name the models your endpoint serves. `claudeCode` and `codex` take a bare model id; `opencode` and `xpander` take a `/` slot whose provider the proxy routes. A Bedrock inference-profile id in `claudeCode` (for example `us.anthropic.claude-sonnet-4-6`) is enough on its own: the provider is inferred from it. An explicit `agents.env` / `agent-controller.env` entry wins |
| Air-Gapped | `global.harness.defaultProviders.claudeCode` (rendered as `XPANDER_HARNESS_DEFAULT_PROVIDER_CLAUDE_CODE` on the `agents` Deployment) | Empty = Anthropic direct. `amazon_bedrock` routes defaulted Claude Code agents to Bedrock through `global.llm.bedrock`, keeping the default model |
| Hybrid | none in the chart | The `agents` service runs in xpander cloud, so step 2 is the cloud's own default: new Claude Code agents run on **Anthropic direct** over the relayed key (the cloud never relays Bedrock). Configure the org default LLM to steer defaulted agents elsewhere; the controller in your cluster needs credentials for whichever provider your agents use |
***
## Air-Gapped vs Hybrid
| | **Hybrid** | **Air-Gapped** |
| ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Fleet values | `fleet.*`, image `images.agentHarness` | `global.fleet.*`, image `global.images.agent-harness` from your mirror, tag or digest pinned |
| Starting executors | `fleet.replicas: 1` | `global.fleet.replicas: 1` |
| Fleet token | Generated into the chart's Secret, or `fleet.tokenSecret.existing` | One more key in `xpander-generated-keys` |
| StorageClass | `createClass: true` renders the EBS gp3 XFS class (`ebs.csi.aws.com`) | `createClass: false`; bring a class that mounts XFS with `prjquota`, or set it to `true` on EKS |
| KEDA | Install `kedacore/keda` yourself, then `fleet.keda.enabled` | Mirror with `image_sync.sh --with-keda`, install, then `global.fleet.keda.enabled` |
| CLI updates | Executors update from the npm registry or your `XPANDER_NPM_REGISTRY` mirror | Sealed CLI bundle imported from a PVC (`global.fleet.cliBundle`) |
| LLM | Controller egress to Anthropic / Bedrock / your endpoint; defaulted agents follow the org default LLM, else Anthropic direct (no Bedrock relay; Bedrock needs an org key) | In-network endpoint only; VPC Bedrock via `global.llm.bedrock.baseUrl` (Pod Identity, assume role or bearer token); defaulted agents follow the org default LLM, then `global.harness.defaultModels` / `defaultProviders` |
| Upgrades | `install.sh upgrade` runs `--atomic`; a running turn holds the roll up to `fleet.executor.drainSeconds`; the idle guard covers a missing class | same, under `global.fleet.executor`; the first install runs without `--wait` |
| Seat cap | `orgMaxTurns` | The license's agent cap when set, else `orgMaxTurns` |
| Recommended posture | `minReplicas: 1`, KEDA on | Fixed `replicas` sized for peak (static capacity, no autoscaler), KEDA optional |
**Recommended for both editions:** keep the fleet warm (at least one executor always up). The first turn of any agent is then seconds, and idle agents cost only their directory. A cluster that cannot meet the prerequisites should set `replicas: 0` (or let the idle guard do it) and fix the missing capability before opening harness agents to its users, since every new agent lands on the fleet.
***
## Next Steps
Base platform sizing, the workspace pod, the calculator, and the installer
Sizing tiers, registry mirroring, and scaling with usage
Node groups, the EBS CSI driver, and StorageClasses
How seat and agent caps are enforced on a sealed install