> ## Documentation Index
> Fetch the complete documentation index at: https://docs.xpander.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Hybrid Edition

> The Hybrid self-hosted edition end-to-end — architecture, requirements, the installer wizard, PrivateLink, LLM keys, upgrades, and troubleshooting

The **Hybrid** edition: your data and agents run in your cluster, control plane in xpander cloud. All execution, runtime data, credentials, and model keys stay inside your VPC; xpander cloud manages agent definitions, identity, and the connector catalog. This page is the complete guide — from architecture review to a running, connected location.

```bash theme={"dark"}
curl -fsSL https://charts.xpander.ai/install.sh | bash -s -- \
  --edition hybrid --org <ORG> --env <ENV> --dm-api-key <KEY>
```

<Note>
  Self-hosted locations are unlocked per account — [contact our team](https://cal.com/team/xpander-ai/activate) if the option isn't available on yours.
</Note>

Comparing editions first? See the [Self-Hosted overview](/self-hosted/index). The full-platform edition with no internet access is [Air-Gapped](/self-hosted/airgap).

***

## Architecture

Hybrid follows the **control plane / data plane** separation pattern — the same model used by AWS EKS, Datadog, and other infrastructure platforms:

* **Control plane (xpander Cloud)** — manages environment metadata, agent registration, connector definitions, and event logging. AI engineers use the Agent Workbench to define agents, configure connectors, and validate tasks. The control plane never touches runtime data.
* **Data plane (your VPC)** — runs all task execution, agent memory, connector credentials, model API keys, and user-facing services. The user's browser connects directly to the data plane. Chat threads, activity views, and task results are served from the cluster — nothing passes through the cloud.

The data plane connects **outbound only** to the control plane over port 443 for metadata sync and heartbeats. The control plane never initiates connections into your cluster. MCP and external-system calls run from your cluster, and agent workspaces — the agents' computers — run there too.

### What Gets Deployed

| Component             | Description                                                |
| --------------------- | ---------------------------------------------------------- |
| **Agent Controller**  | Task scheduling, queue management, and orchestration       |
| **Agent Worker**      | Task execution runtime                                     |
| **API Proxy**         | API endpoint and request routing                           |
| **API**               | REST API service                                           |
| **MCP**               | Accepts MCP client connections, generates tasks            |
| **Database**          | Agent memory, message history, and monitoring (PostgreSQL) |
| **Redis**             | Cache and fast lookups                                     |
| **Chat UI**           | Chat interface with per-thread subdomains                  |
| **Code Runner**       | Code execution runtime                                     |
| **AWS APIs Operator** | AWS resource management                                    |

### How It Works

Clients — Slack, the web UI, MCP clients, the SDK and API — connect directly to the cluster in your VPC. When using `app.xpander.ai`, the cloud UI handles authentication and tells the client which self-hosted cluster to connect to; all data then flows client-to-cluster. Inside the cluster, the **Agent Controller** schedules tasks and **Agent Workers** execute them — calling LLMs, persisting messages in PostgreSQL, and reaching external tools exclusively through the **AI Gateway**.

The only connection to xpander Cloud is the **deployment manager sync**: the Agent Controller authenticates with `organization_id` + `environment_id` + `api_key`, reports heartbeats, and pulls agent, connector, and MCP configuration.

### What Talks to What

The egress surface — the answers a security review needs. Inbound from xpander cloud: **none**; connections are outbound-only.

| From your cluster              | To                                                                                      | Port          | Purpose                                         |
| ------------------------------ | --------------------------------------------------------------------------------------- | ------------- | ----------------------------------------------- |
| agent-controller               | `deployment-manager.xpander.ai` (or the [PrivateLink](#privatelink) / staging endpoint) | 443           | Metadata sync, heartbeats, connector updates    |
| agent-controller, agent-worker | Your LLM providers (OpenAI, Anthropic, Google, Bedrock, ...)                            | 443           | Model calls                                     |
| Agent tools, MCP, connectors   | The external systems you connect                                                        | As configured | Run from your cluster, never from xpander cloud |

Outbound connectivity options to the control plane:

| Option                 | Target                                                    | Details                                            |
| ---------------------- | --------------------------------------------------------- | -------------------------------------------------- |
| **Global Accelerator** | `15.197.85.80`, `166.117.85.46`                           | Public internet, encrypted over TLS                |
| **PrivateLink**        | `com.amazonaws.vpce.us-west-2.vpce-svc-0101884b32f655197` | AWS private backbone — [setup below](#privatelink) |

### Data Residency

| Feature                                 |             Stays in your VPC?            |
| --------------------------------------- | :---------------------------------------: |
| Invoking agents (API to agent)          | Yes — client connects directly to cluster |
| Chat threads and activity view          |   Yes — fetched from cluster, not cloud   |
| Debugging agents (Monitor tab)          |     Yes — all data served from cluster    |
| Knowledge Base file uploads and queries |    Yes — stored and queried in cluster    |
| User identity and login                 |                     No                    |
| Agent metadata (agent list)             |                     No                    |

Data at rest stays in PostgreSQL and Redis inside your VPC; credentials and API keys are Kubernetes secrets; the control plane sync carries metadata and heartbeats only — user payloads never leave the VPC.

***

## Requirements & Sizing

Minimum footprint (the installer shows the same numbers, published at [charts.xpander.ai/requirements.json](https://charts.xpander.ai/requirements.json)):

|              | Hybrid minimum                                                                       |
| ------------ | ------------------------------------------------------------------------------------ |
| Nodes        | 3 × 8 vCPU / 32 GiB                                                                  |
| Storage      | \~50 GiB PVC                                                                         |
| Kubernetes   | ≥ 1.28                                                                               |
| Architecture | amd64                                                                                |
| Network      | Egress to `deployment-manager.xpander.ai` (or PrivateLink) and to your LLM providers |
| Extras       | Ingress controller + wildcard DNS optional                                           |

<Warning>
  xpander container images are **amd64 only**. Do not use ARM/Graviton instances (`t4g`, `m7g`, `c7g`, etc.).
</Warning>

Need a cluster? Follow the [EKS Cluster Setup](/self-hosted/eks-setup) guide.

### Example Estimates

Agents are autonomous — tasks come from schedules, webhooks, the SDK and API, and other agents, not only chat (in xpander cloud production, only \~5% of executions are interactive chat). Size for the number of tasks running **concurrently at peak**, whatever triggers them.

| Concurrent tasks | Agents | Cluster capacity  | Example nodes                                       | Persistent storage |
| ---------------- | ------ | ----------------- | --------------------------------------------------- | ------------------ |
| 3                | 10     | 10 vCPU / 19 GiB  | 3 × `m5.xlarge` (or 2 × `t3.xlarge` for evaluation) | \~110 GiB          |
| 10               | 25     | 19 vCPU / 34 GiB  | 3 × `m5.2xlarge`                                    | \~180 GiB          |
| 20               | 50     | 29 vCPU / 49 GiB  | 4 × `m5.2xlarge`                                    | \~310 GiB          |
| 50               | 100    | 66 vCPU / 105 GiB | 9 × `m5.2xlarge`, enable autoscaling                | \~570 GiB          |

Capacity figures include \~50% headroom over pod resource requests for burst load, Kubernetes system pods, and node upgrades. Use the calculator below to size for your exact numbers.

### Sizing Calculator

<iframe
  srcDoc={`<!doctype html><html><head><meta charset="utf-8"/><style>
:root {
  color-scheme: light dark;
  --xp-purple: #753CFF;
  --xp-purple-light: #9B70FF;
  --xp-violet-400: #a78bfa;
  --xp-mono: "JetBrains Mono", "Fira Code", "SF Mono", ui-monospace, Menlo, monospace;
  --xp-sans: "Helvetica Thin", Helvetica, Arial, sans-serif;
  --xp-bg: #ffffff;
  --xp-bg-card: #fafafa;
  --xp-bg-input: #ffffff;
  --xp-text: #18181b;
  --xp-text-muted: #71717a;
  --xp-text-faint: #a1a1aa;
  --xp-border: rgba(0,0,0,0.08);
  --xp-accent-bg: rgba(117,60,255,0.04);
  --xp-accent-border: rgba(117,60,255,0.25);
}
@media (prefers-color-scheme: dark) {
  :root {
    --xp-bg: #111113;
    --xp-bg-card: rgba(255,255,255,0.03);
    --xp-bg-input: rgba(255,255,255,0.02);
    --xp-text: #f4f4f5;
    --xp-text-muted: #a1a1aa;
    --xp-text-faint: #71717a;
    --xp-border: rgba(255,255,255,0.08);
    --xp-accent-bg: rgba(167,139,250,0.06);
    --xp-accent-border: rgba(167,139,250,0.35);
  }
}
* { box-sizing: border-box; }
html, body { margin: 0; padding: 0; background: var(--xp-bg); color: var(--xp-text); font-family: var(--xp-sans); font-weight: 100; overflow-x: hidden; }
body { padding: 20px; font-size: 15px; line-height: 1.7; word-break: break-word; }
.kicker { font-family: var(--xp-mono); font-size: 11px; font-weight: 500; letter-spacing: 0.14em; text-transform: uppercase; color: var(--xp-text-faint); margin: 0 0 10px; }
.sentence { display: flex; flex-wrap: wrap; align-items: center; gap: 8px 10px; background: var(--xp-bg-card); border: 1px solid var(--xp-border); border-radius: 12px; padding: 18px 22px; line-height: 2; }
.sentence .w { white-space: nowrap; }
input, select {
  background: var(--xp-bg-input);
  color: var(--xp-purple);
  border: 1px solid var(--xp-border);
  border-radius: 8px;
  padding: 6px 10px;
  font-family: var(--xp-mono);
  font-size: 13px;
  outline: none;
  transition: border-color 0.15s ease;
}
@media (prefers-color-scheme: dark) { input, select { color: var(--xp-violet-400); } }
input:hover, select:hover { border-color: var(--xp-accent-border); }
input:focus, select:focus { border-color: var(--xp-purple); }
input.num { width: 64px; text-align: center; }
.grid { display: grid; grid-template-columns: repeat(auto-fit, minmax(150px, 1fr)); gap: 10px; margin-top: 14px; }
.tile { background: var(--xp-bg-card); border: 1px solid var(--xp-border); border-radius: 12px; padding: 12px 16px; }
.tile .label { font-family: var(--xp-mono); font-size: 10px; font-weight: 500; letter-spacing: 0.12em; text-transform: uppercase; color: var(--xp-text-faint); margin: 0 0 4px; }
.tile .val { font-family: var(--xp-mono); font-size: 16px; color: var(--xp-purple); line-height: 1.4; }
.tile .sub { font-size: 12px; color: var(--xp-text-muted); margin-top: 2px; }
@media (prefers-color-scheme: dark) { .tile .val { color: var(--xp-violet-400); } }
.presets { display: flex; flex-wrap: wrap; gap: 8px; margin-top: 14px; }
.presets button {
  background: transparent; color: var(--xp-text-muted);
  border: 1px solid var(--xp-border); border-radius: 999px;
  padding: 6px 14px; font-family: var(--xp-sans); font-size: 12px; font-weight: 300;
  cursor: pointer; transition: border-color 0.15s ease, color 0.15s ease, background-color 0.15s ease;
}
.presets button:hover { border-color: var(--xp-accent-border); color: var(--xp-purple); background: var(--xp-accent-bg); }
@media (prefers-color-scheme: dark) { .presets button:hover { color: var(--xp-violet-400); } }
details { margin-top: 12px; }
summary { font-size: 12px; color: var(--xp-text-muted); cursor: pointer; }
.adv { display: flex; flex-wrap: wrap; gap: 8px 18px; align-items: center; padding: 10px 2px 0; font-size: 13px; color: var(--xp-text-muted); }
@media (max-width: 520px) { body { padding: 16px; font-size: 14px; } }
</style></head><body>
<p class="kicker">Size your cluster</p>
<div class="sentence">
<span class="w">At peak we run</span>
<input class="num" id="tasks" type="number" min="1" value="5" />
<span class="w">concurrent agent tasks across</span>
<input class="num" id="agents" type="number" min="1" value="15" />
<span class="w">agents, and</span>
<input class="num" id="wsshare" type="number" min="0" max="100" value="50" />
<span class="w">% of tasks use a workspace.</span>
</div>
<details>
<summary>Advanced assumptions</summary>
<div class="adv">
  <span>Concurrent tasks per worker replica: <input class="num" id="slots" type="number" min="1" max="20" value="4" /></span>
  <span>(default 4, raise via <span style="font-family:var(--xp-mono);font-size:12px">MAX_CONCURRENT_EXECUTIONS</span>)</span>
</div>
<div class="adv">
  <span>Not sure? Assume one concurrent task per agent. Most production teams peak at just a few concurrent tasks.</span>
</div>
</details>
<div class="grid">
<div class="tile"><p class="label">Pod requests</p><div class="val" id="req">-</div><div class="sub" id="reqsub">-</div></div>
<div class="tile"><p class="label">Recommended cluster</p><div class="val" id="cap">-</div><div class="sub">includes ~50% headroom</div></div>
<div class="tile"><p class="label">Example nodes</p><div class="val" id="nodes">-</div><div class="sub">amd64 / x86_64 only</div></div>
<div class="tile"><p class="label">Persistent storage</p><div class="val" id="disk">-</div><div class="sub">volumes + workspace data</div></div>
</div>
<div class="presets">
<button data-t="3" data-a="10">3 tasks / 10 agents</button>
<button data-t="10" data-a="25">10 tasks / 25 agents</button>
<button data-t="20" data-a="50">20 tasks / 50 agents</button>
<button data-t="50" data-a="100">50 tasks / 100 agents</button>
</div>
<script>
const el = id => document.getElementById(id);
function num(id, lo, hi, fb){ const v = parseInt(el(id).value, 10); return Number.isInteger(v) ? Math.min(hi, Math.max(lo, v)) : fb; }
function update(){
  const tasks = num("tasks", 1, 100000, 5);
  const agents = num("agents", 1, 100000, 15);
  const wsShare = num("wsshare", 0, 100, 50) / 100;
  const slots = num("slots", 1, 20, 4);
  const workers = Math.ceil(tasks / slots);
  const controllers = tasks <= 8 ? 1 : tasks <= 20 ? 2 : 3;
  const workspaces = Math.min(agents, Math.max(1, Math.round(tasks * wsShare)));
  const baseCpu = 2.05, baseMem = 6;
  const cpu = baseCpu + controllers * 1 + workers * 2 + workspaces * 0.5;
  const mem = baseMem + controllers * 2 + workers * 2.5 + workspaces * 1;
  const capCpu = Math.ceil(cpu * 1.5);
  const capMem = Math.ceil(mem * 1.5);
  const node = capCpu <= 16 ? { name: "m5.xlarge", cpu: 4, mem: 16 } : { name: "m5.2xlarge", cpu: 8, mem: 32 };
  const count = Math.max(2, Math.ceil(capCpu / node.cpu), Math.ceil(capMem / node.mem));
  const disk = Math.ceil((48 + agents * 5 + count * 2) / 10) * 10;
  el("req").textContent = cpu.toFixed(1) + " vCPU / " + Math.ceil(mem) + " GiB";
  el("reqsub").textContent = workers + " worker" + (workers > 1 ? "s" : "") + " · " + controllers + " controller" + (controllers > 1 ? "s" : "") + " · ~" + workspaces + " active workspace" + (workspaces > 1 ? "s" : "");
  el("cap").textContent = capCpu + " vCPU / " + capMem + " GiB";
  el("nodes").textContent = count + " \\u00d7 " + node.name;
  el("disk").textContent = "~" + disk + " GiB";
}
["tasks","agents","wsshare","slots"].forEach(id => { el(id).addEventListener("input", update); el(id).addEventListener("change", update); });
document.querySelectorAll(".presets button").forEach(b => b.addEventListener("click", () => { el("tasks").value = b.dataset.t; el("agents").value = b.dataset.a; update(); }));
update();
</script></body></html>`}
  style={{ width: "100%", height: "430px", border: "none", borderRadius: "12px", display: "block", colorScheme: "light dark" }}
/>

<Accordion title="How the estimate works">
  The calculator applies the same model we use to size xpander cloud:

  1. **Concurrent tasks** is the primary driver. A task is any agent execution, from any source: chat, API/SDK calls, schedules, webhooks, MCP clients, Slack, or another agent. If you don't know your peak concurrency yet, **assume one concurrent task per agent** - that covers most production organizations we measured. In practice most teams peak at just a few concurrent tasks (typically under 5), and even the busiest organization peaks around 30.
  2. **Agent Workers** = concurrent tasks ÷ 4 (each worker replica processes 4 tasks by default, configurable with `MAX_CONCURRENT_EXECUTIONS`). Each replica requests 2 vCPU / 2.5 GiB.
  3. **Agent Controllers**: 1 replica up to \~8 concurrent tasks, 2 up to \~20, 3 beyond. Each requests 1 vCPU / 2 GiB.
  4. **Active workspaces** = concurrent tasks × workspace share, capped at the agent count. Budget \~0.5 vCPU / 1 GiB per active workspace (measured typical load; each can burst to its 2 vCPU / 3 GiB limit).
  5. **Fixed base** (\~2 vCPU / 6 GiB): AI Gateway, MCP, Chat, Code Runner, AWS Operator, API, Redis, PostgreSQL, container registry, metrics-server at one replica each.
  6. **Cluster capacity** = requests × 1.5 for burst, DaemonSets, and safe node drains. **Storage** = 48 GiB of chart volumes + 5 GiB per agent (each agent that uses a workspace keeps a persistent volume) + image cache per node.
</Accordion>

<Accordion title="Per-service requirements">
  Per-replica values. **Defaults** are what the Helm chart applies out of the box; **recommended** raises the values we found low against measured production usage.

  | Service          | CPU request     | Memory request       | CPU limit     | Memory limit   | Notes                                                                                               |
  | ---------------- | --------------- | -------------------- | ------------- | -------------- | --------------------------------------------------------------------------------------------------- |
  | Agent Controller | 1000m           | 2 Gi                 | 3000m         | 4 Gi           | Orchestrator; observed steady-state memory is \~1.5-2 GiB, and the limit carries headroom for peaks |
  | Agent Worker     | 2000m           | 2.25 Gi → **2.5 Gi** | 4000m         | 3 Gi           | Scale replicas for task concurrency, not size                                                       |
  | API              | 100m → **250m** | 256 Mi → **1 Gi**    | - → **2000m** | - → **1.5 Gi** | Chart sets no limits; observed usage \~0.9-1.4 GiB                                                  |
  | MCP              | 200m            | 750 Mi               | 400m          | 1 Gi           |                                                                                                     |
  | Chat             | 300m            | 1 Gi                 | 600m          | 1.5 Gi         |                                                                                                     |
  | AI Gateway       | 200m            | 400 Mi               | 400m          | 600 Mi         |                                                                                                     |
  | Code Runner      | 400m            | 800 Mi               | 800m          | 1229 Mi        | Runs privileged (Docker-in-Docker); each concurrent code execution adds up to 1 vCPU / 2 GiB        |
  | AWS Operator     | 200m            | 400 Mi               | 400m          | 600 Mi         | Optional component                                                                                  |

  Override any of these in your Helm values (keys are camelCase):

  ```yaml theme={"dark"}
  resources:
    agentController:
      requests:
        cpu: "1000m"
        memory: "2Gi"
      limits:
        cpu: "3000m"
        memory: "4Gi"
  ```

  <Tip>
    Horizontal Pod Autoscaling is available for Agent Controller, AI Gateway, Agent Worker, and MCP via the chart's `autoscaling` values. The chart installs metrics-server by default to support it.
  </Tip>
</Accordion>

<Accordion title="Agent workspaces">
  Each agent that uses workspace tools gets a dedicated pod, created on demand and reclaimed when idle. Size the cluster for the number of **concurrently active** workspaces, not total agents.

  | Resource         | Value                                                                                        |
  | ---------------- | -------------------------------------------------------------------------------------------- |
  | CPU              | 250m request, 2000m limit                                                                    |
  | Memory           | 256 Mi request, 3 Gi limit                                                                   |
  | Storage          | 5 GiB persistent volume per agent                                                            |
  | Idle shutdown    | 5 minutes without activity (configurable via `SANDBOX_IDLE_TIMEOUT` on the Agent Controller) |
  | Maximum lifetime | 3 hours, then the pod is recycled                                                            |

  Idle workspaces consume almost nothing (\~50 Mi observed); an active workspace typically uses \~0.5-1 vCPU and up to 1 GiB, bursting to its limits for heavy work. The workspace volume **persists after the pod is reclaimed**, so a returning agent keeps its files - plan 5 GiB of storage per workspace-using agent, not per pod.

  <Info>
    The workspace image is \~1 GiB. The first workspace start on each node pulls it; subsequent starts on that node are fast. Budget \~2 GiB of node disk for image cache.
  </Info>
</Accordion>

<Accordion title="Data services, storage, and the production checklist">
  The chart deploys single-replica Redis and PostgreSQL StatefulSets and a container registry alongside the platform:

  | Component          | Persistent volume (default) | Guidance                                                                                                                                 |
  | ------------------ | --------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- |
  | Redis              | 8 GiB                       | Queues, caches, and runtime state. 2-4 GiB of memory is sufficient for a single organization; give the pod a 512 Mi-1 GiB memory request |
  | PostgreSQL         | 20 GiB                      | Platform state; grows slowly                                                                                                             |
  | Container registry | 20 GiB                      | Stores agent container images built in your cluster                                                                                      |

  **Baseline storage: 48 GiB** of persistent volumes, plus 5 GiB per workspace-using agent (see above). All volumes use your cluster's default StorageClass unless overridden.

  Production checklist:

  * **Node types**: x86\_64 only. General-purpose instances with a 1:4 vCPU:GiB ratio (`m5`, `m6i`) fit the workload profile; `t3` burstable instances are fine for evaluation, not for production.
  * **Raise the low defaults**: API memory (table above) before real load.
  * **Enable autoscaling** for Agent Worker once concurrency grows beyond a couple of replicas.
  * **Keep headroom**: pod requests should stay under \~65% of cluster capacity so node upgrades and bursts don't evict workloads.
  * **Storage class**: SSD-backed volumes for Redis and PostgreSQL.
</Accordion>

***

## Install

### 1. Get Your One-Liner from the Console

1. Go to [app.xpander.ai/admin\_settings#locations](https://app.xpander.ai/admin_settings#locations)
2. Click **"Add Location"**
3. Choose the **Hybrid** edition. The dialog shows the minimum footprint for the edition you picked.
4. Optionally set a **URL** (your server's FQDN, e.g. `xpander.my-company.com`) and the hostname style (dash-based subdomains if your DNS can't do wildcards)
5. Copy the generated one-liner

The one-liner carries your identity — `--org`, `--env`, and `--dm-api-key` (plus `--domain` when you set a URL). That is how the cluster authenticates to the xpander control plane:

```bash theme={"dark"}
curl -fsSL https://charts.xpander.ai/install.sh | bash -s -- \
  --edition hybrid \
  --org <ORGANIZATION_UUID> \
  --env <ENVIRONMENT_UUID> \
  --dm-api-key <DEPLOYMENT_MANAGER_API_KEY> \
  --domain xpander.my-company.com
```

<Note>
  The one-liner deliberately carries **no LLM provider credentials** — the wizard prompts for those interactively (input hidden), or you pass them as flags. Secrets are written to a temporary values file (mode 600, removed on exit) and never appear on the helm command line.
</Note>

### 2. Run the Installer

Run it from a terminal with `kubectl` access to the target cluster. The command downloads the installer from `charts.xpander.ai` and runs an interactive wizard — preflight checks, configuration, Helm install, then the connection back to xpander. Run bare (`curl -fsSL https://charts.xpander.ai/install.sh | bash`), it is the zero-knowledge entry point: the first question is which edition you are installing.

The wizard walks through, in order:

<Steps>
  <Step title="Edition & requirements">
    If `--edition` wasn't given, it asks. It then prints the minimum footprint for that edition (the same numbers as [charts.xpander.ai/requirements.json](https://charts.xpander.ai/requirements.json)) and asks you to confirm your cluster meets it — stopping cleanly if not.
  </Step>

  <Step title="Identity">
    Organization ID, Environment ID, and the Deployment Manager API key (input hidden) — all from the console's install card, all skipped if passed as flags.
  </Step>

  <Step title="Ingress domain (optional)">
    Press Enter to skip — services then keep cluster-internal Services only, which is fine for a first install (port-forward reaches them; add ingress later). With a domain, it asks how to build hostnames: **subdomain** style (`agent-controller.xpander.<DOMAIN>`, needs wildcard DNS) or **dash** style (`agent-controller-xpander.<DOMAIN>`, individual A records).
  </Step>

  <Step title="Control plane reachability">
    Public internet (`https://deployment-manager.xpander.ai`), AWS PrivateLink (`https://deployment-manager-privatelink.xpander.ai`), or Staging. The installer auto-detects staging/local environments (a `.stg.` or local domain, or a kind/k3d/minikube/docker-desktop kube context) and leads with that option; any of `--staging`, `--privatelink`, or `--dm-url` overrides detection.
  </Step>

  <Step title="LLM providers (all optional)">
    OpenAI, Anthropic, Google/Gemini, Amazon Bedrock, then optionally Fireworks, Nebius, Helicone, and OpenRouter. Press Enter to skip any — the platform installs and starts without keys, and model calls light up as soon as you add a provider key in the UI or with a helm upgrade.

    For **Bedrock** it asks how to authenticate: an API key / bearer token (`AWS_BEARER_TOKEN_BEDROCK`), or keyless IAM role / EKS pod identity. See the [caveat below](#amazon-bedrock-key-vs-iam) before choosing IAM.
  </Step>

  <Step title="Namespace, context confirmation, summary">
    Namespace (default `xpander`, created if missing), an explicit confirmation of the kubectl context it is about to install into, and a masked configuration summary before anything changes.
  </Step>
</Steps>

### Install Phases

The Hybrid install runs six phases, each announced as a stable `[N/6] name: ...` progress line (the same shape scripts can grep — see [Automation & CI](#automation--ci)):

```
[1/6] preflight       helm ≥ 3.13, kubectl, cluster reachable, Kubernetes ≥ 1.28
[2/6] configuration   context confirmation + masked summary
[3/6] chart-source    helm repo add xpander https://charts.xpander.ai (or --chart-path)
[4/6] values          secrets written to a mode-600 temp file, removed on exit
[5/6] install         helm upgrade --install xpander xpander/xpander
[6/6] verify          waits for the agent-controller rollout (5-minute timeout)
```

Every failure prints a `FIX:` line naming the action that repairs it and a `RERUN:` line with your personalized non-interactive one-liner (secrets typed at prompts are placeholdered, never echoed). Fix the condition and paste the rerun line — the installer is idempotent.

### Amazon Bedrock: key vs IAM

`--bedrock-auth key` (default when `--bedrock-key` is given) stores the bearer token; `--bedrock-auth iam` is keyless and reads the pod's own AWS credential chain.

<Warning>
  **Keyless IAM Bedrock requires the updated xpander SDK release.** On older SDK versions, chats work but **task executions fail** — the SDK sets an empty bearer token unconditionally. On EKS it also needs IRSA or pod identity wired to a role with Bedrock access. If in doubt, use the bearer-token mode.
</Warning>

### 3. Verify the Connection

When the install completes, a summary box prints the release, control-plane URL, ingress state, and configured providers. Your cluster registers with the control plane and the xpander console flips to **"Connection established"** in a minute or two.

```bash theme={"dark"}
kubectl get pods -n xpander
```

Expect one `Running`, `1/1`-ready pod per application service (`agent-controller`, `agent-worker`, `ai-gateway`, `api`, `aws-operator`, `chat`, `code-runner`, `mcp`) plus the `postgres` and `redis` StatefulSets — and, at defaults, `docker-registry` and `metrics-server`.

Test locally without ingress:

```bash theme={"dark"}
kubectl -n xpander port-forward service/xpander-agent-controller 9016:9016 &
kubectl -n xpander port-forward service/xpander-ai-gateway 9018:9018 &
kubectl -n xpander port-forward service/xpander-mcp 8081:8081 &

curl http://localhost:9016/health
curl http://localhost:9018/health
curl http://localhost:8081/health
```

In the console ([admin\_settings#locations](https://app.xpander.ai/admin_settings#locations)), the location shows as **"Self-deploy"**; the **Components** tab shows heartbeat status per service, and the **Configuration** panel toggles Cloud Controls (Chat, Streaming, Scheduler).

***

## PrivateLink

Your cluster must reach the xpander **deployment manager**. Two options:

| Option                    | URL                                                 | Security                                                                  |
| ------------------------- | --------------------------------------------------- | ------------------------------------------------------------------------- |
| **Public (TLS)**          | `https://deployment-manager.xpander.ai`             | Encrypted over public internet — no setup needed, the installer's default |
| **Private (PrivateLink)** | `https://deployment-manager-privatelink.xpander.ai` | Encrypted over AWS private backbone — this section                        |

If public works for you, skip this section entirely: your cluster nodes just need outbound HTTPS (via NAT Gateway if in private subnets). If your security policy requires traffic to stay within the AWS network, set up PrivateLink, then tell the installer with `--privatelink` (or pick **AWS PrivateLink** at the wizard's reachability question). `--privatelink`, `--staging`, and `--dm-url` are mutually exclusive — pass exactly one, or none for the public endpoint.

### How PrivateLink Works

AWS PrivateLink is a two-sided connection. xpander exposes the deployment manager as a **VPC Endpoint Service** (provider). You create a **VPC Endpoint** (consumer) in your VPC, which provisions a private ENI with a `10.x.x.x` IP address. Your pods connect to this private IP — traffic goes over the AWS backbone, never the internet.

| Property            | Value                                                               |
| ------------------- | ------------------------------------------------------------------- |
| Service Name        | `com.amazonaws.vpce.us-west-2.vpce-svc-0101884b32f655197`           |
| Service Region      | `us-west-2`                                                         |
| Supported Regions   | `us-west-2`, `us-west-1`, `eu-west-2`, `eu-central-1`, `ap-south-1` |
| Acceptance Required | No (auto-accepted for allowed accounts)                             |
| Private DNS Name    | `deployment-manager-privatelink.xpander.ai`                         |

<Warning>
  Contact xpander to have your AWS account ID added as an allowed principal before creating the VPC endpoint.
</Warning>

### Set It Up

<Steps>
  <Step title="Create the VPC endpoint">
    <Warning>
      The xpander endpoint service is in `us-west-2`. If your cluster is in a different region, you **must** include the `--service-region us-west-2` flag — this is the most common mistake (see [Troubleshooting](#troubleshooting)).
    </Warning>

    ```bash theme={"dark"}
    aws ec2 create-vpc-endpoint \
      --vpc-id <VPC_ID> \
      --vpc-endpoint-type Interface \
      --service-name com.amazonaws.vpce.us-west-2.vpce-svc-0101884b32f655197 \
      --service-region us-west-2 \
      --subnet-ids <PRIVATE_SUBNET_A_ID> <PRIVATE_SUBNET_B_ID> \
      --no-private-dns-enabled \
      --tag-specifications 'ResourceType=vpc-endpoint,Tags=[{Key=Name,Value=xpander-privatelink}]' \
      --region <REGION> --profile <PROFILE>
    ```

    Save from the output: `VpcEndpointId`, `Groups[0].GroupId` (the security group), `DnsEntries[0].DnsName`, and `DnsEntries[0].HostedZoneId`.

    <Tip>
      If your cluster is in `us-west-2` (same region as the service), you can omit `--service-region`.
    </Tip>
  </Step>

  <Step title="Allow HTTPS on the endpoint security group">
    The VPC endpoint is created with the VPC's default security group, which only allows inbound traffic from itself. Allow HTTPS from your VPC CIDR:

    ```bash theme={"dark"}
    aws ec2 authorize-security-group-ingress \
      --group-id <ENDPOINT_SG_ID> \
      --protocol tcp \
      --port 443 \
      --cidr <VPC_CIDR> \
      --region <REGION> --profile <PROFILE>
    ```
  </Step>

  <Step title="Wait for the endpoint to become available">
    The endpoint transitions from `pending` to `available` in 1-3 minutes (auto-accepted):

    ```bash theme={"dark"}
    aws ec2 describe-vpc-endpoints \
      --vpc-endpoint-ids <VPCE_ID> \
      --query 'VpcEndpoints[0].State' \
      --region <REGION> --profile <PROFILE>
    ```
  </Step>

  <Step title="Create private DNS">
    Create a private hosted zone so `deployment-manager-privatelink.xpander.ai` resolves to the endpoint ENI inside your VPC:

    ```bash theme={"dark"}
    aws route53 create-hosted-zone \
      --name deployment-manager-privatelink.xpander.ai \
      --caller-reference "xpander-privatelink-$(date +%s)" \
      --vpc VPCRegion=<REGION>,VPCId=<VPC_ID> \
      --hosted-zone-config PrivateZone=true \
      --profile <PROFILE>
    ```

    Save the `HostedZone.Id` from the output, then create the alias record:

    ```bash theme={"dark"}
    cat <<EOF > /tmp/privatelink-dns.json
    {
      "Changes": [{
        "Action": "UPSERT",
        "ResourceRecordSet": {
          "Name": "deployment-manager-privatelink.xpander.ai.",
          "Type": "A",
          "AliasTarget": {
            "DNSName": "<VPCE_DNS_NAME>",
            "HostedZoneId": "<VPCE_HOSTED_ZONE_ID>",
            "EvaluateTargetHealth": true
          }
        }
      }]
    }
    EOF

    aws route53 change-resource-record-sets \
      --hosted-zone-id <PRIVATE_ZONE_ID> \
      --change-batch file:///tmp/privatelink-dns.json \
      --profile <PROFILE>
    ```
  </Step>

  <Step title="Verify connectivity">
    ```bash theme={"dark"}
    kubectl run curl-test --restart=Never --image=curlimages/curl:latest -n default \
      --command -- sh -c "curl -sk -o /dev/null -w '%{http_code}' \
      --connect-timeout 10 https://deployment-manager-privatelink.xpander.ai/health; echo"

    sleep 15 && kubectl logs curl-test
    # Expected output: 200

    kubectl delete pod curl-test
    ```

    A timeout or `000`? See [Troubleshooting](#troubleshooting) below.
  </Step>
</Steps>

***

## External Access & DNS

The xpander services communicate with each other internally via Kubernetes Services — no ingress is needed for that. You only need external access so that **users** can reach the chat UI, API, and other endpoints from their browser or client.

<Info>
  **Already have an ingress controller?** Give the installer a `--domain` (or set `ingress.enabled=true` in Helm values) and skip to [DNS records](#dns-records).
</Info>

### Option A: VPN / Corporate Network Access

If your organization has a VPN that routes into the VPC, users can access xpander services directly — no ingress controller, NLB, ACM certificate, or public DNS needed.

Point internal DNS records to the Kubernetes services:

```
chat.<DOMAIN>              → xpander-chat.xpander.svc.cluster.local
agent-controller.<DOMAIN>  → xpander-agent-controller.xpander.svc.cluster.local
ai-gateway.<DOMAIN>        → xpander-ai-gateway.xpander.svc.cluster.local
```

When installing, skip the domain prompt (ingress stays disabled, the default).

### Option B: Public Access via Ingress + NLB

For public or internet-facing access, set up an ACM certificate, nginx ingress controller, and NLB.

#### SSL Certificate

Request an ACM certificate for your domain.

<Warning>
  The certificate **must** include a wildcard for `*.chat.<DOMAIN>` because the xpander chat UI generates per-thread subdomains (e.g., `moccasin-prawn.chat.<DOMAIN>`). Standard wildcards only match one level, so `*.<DOMAIN>` does not cover these. If you omit `*.chat.<DOMAIN>`, the chat UI will show SSL certificate errors for thread URLs.
</Warning>

```bash theme={"dark"}
aws acm request-certificate \
  --domain-name "<DOMAIN>" \
  --subject-alternative-names "*.<DOMAIN>" "*.chat.<DOMAIN>" \
  --validation-method DNS \
  --region <REGION> --profile <PROFILE>
```

Add the DNS validation CNAME records to your Route 53 hosted zone. There will be two unique validation records — one for `<DOMAIN>` / `*.<DOMAIN>` (shared) and one for `*.chat.<DOMAIN>`:

```bash theme={"dark"}
aws acm describe-certificate --certificate-arn <CERT_ARN> \
  --region <REGION> --profile <PROFILE> \
  --query 'Certificate.DomainValidationOptions[0].ResourceRecord'
```

Wait for validation:

```bash theme={"dark"}
aws acm wait certificate-validated --certificate-arn <CERT_ARN> \
  --region <REGION> --profile <PROFILE>
```

#### Ingress Controller

Install the nginx ingress controller with AWS NLB and ACM SSL termination:

```bash theme={"dark"}
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update

helm upgrade --install ingress-nginx ingress-nginx/ingress-nginx \
  --namespace ingress-nginx --create-namespace \
  --set controller.service.type=LoadBalancer \
  --set controller.service.annotations."service\.beta\.kubernetes\.io/aws-load-balancer-type"=nlb \
  --set controller.service.annotations."service\.beta\.kubernetes\.io/aws-load-balancer-scheme"=internet-facing \
  --set controller.service.annotations."service\.beta\.kubernetes\.io/aws-load-balancer-ssl-cert"="<ACM_CERT_ARN>" \
  --set controller.service.annotations."service\.beta\.kubernetes\.io/aws-load-balancer-ssl-ports"="443" \
  --set controller.service.annotations."service\.beta\.kubernetes\.io/aws-load-balancer-backend-protocol"="tcp" \
  --set controller.service.targetPorts.https=http
```

Get the NLB hostname (you'll need it for the DNS records):

```bash theme={"dark"}
kubectl get svc -n ingress-nginx ingress-nginx-controller \
  -o jsonpath='{.status.loadBalancer.ingress[0].hostname}'
```

### DNS Records

Create wildcard CNAME records pointing `*.<DOMAIN>` and `*.chat.<DOMAIN>` to the NLB hostname. Both are required — the chat UI generates per-thread subdomains under `chat.<DOMAIN>`.

```bash theme={"dark"}
cat <<EOF > /tmp/dns-record.json
{
  "Changes": [
    {
      "Action": "UPSERT",
      "ResourceRecordSet": {
        "Name": "*.<DOMAIN>.",
        "Type": "CNAME",
        "TTL": 300,
        "ResourceRecords": [{ "Value": "<NLB_HOSTNAME>" }]
      }
    },
    {
      "Action": "UPSERT",
      "ResourceRecordSet": {
        "Name": "*.chat.<DOMAIN>.",
        "Type": "CNAME",
        "TTL": 300,
        "ResourceRecords": [{ "Value": "<NLB_HOSTNAME>" }]
      }
    }
  ]
}
EOF

aws route53 change-resource-record-sets \
  --hosted-zone-id <HOSTED_ZONE_ID> \
  --change-batch file:///tmp/dns-record.json \
  --profile <PROFILE>
```

This creates the following endpoints:

| Service          | URL                                 |
| ---------------- | ----------------------------------- |
| API              | `https://api.<DOMAIN>`              |
| Chat             | `https://chat.<DOMAIN>`             |
| AI Gateway       | `https://ai-gateway.<DOMAIN>`       |
| Agent Controller | `https://agent-controller.<DOMAIN>` |
| Agent Worker     | `https://agent-worker.<DOMAIN>`     |
| MCP              | `https://mcp.<DOMAIN>`              |
| Code Runner      | `https://code-runner.<DOMAIN>`      |
| AWS Operator     | `https://aws-operator.<DOMAIN>`     |

Test it:

```bash theme={"dark"}
curl -sk https://api.<DOMAIN>/health
```

***

## Managing LLM API Keys

Use `helm upgrade` with `--reuse-values` to add or change API keys without affecting existing configuration:

<CodeGroup>
  ```bash Single Key theme={"dark"}
  helm upgrade xpander xpander/xpander \
    --namespace xpander --reuse-values \
    --set secrets.static.anthropicApiKey=<KEY> \
    --timeout 5m
  ```

  ```bash Multiple Keys theme={"dark"}
  helm upgrade xpander xpander/xpander \
    --namespace xpander --reuse-values \
    --set secrets.static.anthropicApiKey=<KEY> \
    --set secrets.static.agentsOpenaiApiKey=<KEY> \
    --set secrets.static.fireworksApiKey=<KEY> \
    --timeout 5m
  ```
</CodeGroup>

After the upgrade, pods that use the updated secrets will automatically restart. Verify with `kubectl get pods -n xpander`.

<Accordion title="Secret not updating after helm upgrade?">
  The `xpander-static` secret has a Helm resource keep policy — `helm upgrade` may not update it on subsequent installs. If your API key isn't being picked up after a helm upgrade, set it directly in the secret:

  ```bash theme={"dark"}
  # Set the key directly in the Kubernetes secret
  kubectl patch secret xpander-static -n xpander --type=merge \
    -p "{\"data\":{\"ANTHROPIC_API_KEY\":\"$(echo -n '<YOUR_KEY>' | base64)\"}}"

  # Restart the worker to pick up the new key
  kubectl rollout restart deployment xpander-agent-worker -n xpander
  ```

  **Secret field name mapping:**

  | Helm Value                               | Secret Field                 |
  | ---------------------------------------- | ---------------------------- |
  | `secrets.static.anthropicApiKey`         | `ANTHROPIC_API_KEY`          |
  | `secrets.static.agentsOpenaiApiKey`      | `AGENTS_OPENAI_API_KEY`      |
  | `secrets.static.fireworksApiKey`         | `FIREWORKS_API_KEY`          |
  | `secrets.static.awsBedrockApiKey`        | `AWS_BEARER_TOKEN_BEDROCK`   |
  | `secrets.static.googleApiKey`            | `GOOGLE_API_KEY`             |
  | `secrets.static.nebiusApiKey`            | `NEBIUS_API_KEY`             |
  | `secrets.static.heliconeApiKey`          | `HELICONE_API_KEY`           |
  | `secrets.static.openrouterApiKey`        | `OPENROUTER_API_KEY`         |
  | `secrets.static.deploymentManagerApiKey` | `DEPLOYMENT_MANAGER_API_KEY` |
</Accordion>

### Using Kubernetes Secrets (Recommended for Production)

```bash theme={"dark"}
kubectl create secret generic ai-service-keys \
  --namespace xpander \
  --from-literal=openai-api-key=sk-your-openai-key \
  --from-literal=anthropic-api-key=sk-ant-your-anthropic-key
```

```yaml theme={"dark"}
# In your values file
agent-worker:
  envFromSecretKeys:
    AGENTS_OPENAI_API_KEY:
      secretName: "ai-service-keys"
      key: "openai-api-key"
    ANTHROPIC_API_KEY:
      secretName: "ai-service-keys"
      key: "anthropic-api-key"
```

***

## Using the SDK with Self-Hosted

Configure the xpander SDK to point to your Agent Controller endpoint.

<Note>
  When using self-hosted deployment, use the **Agent Controller API key** generated during installation, not your xpander.ai cloud API key.
</Note>

```python theme={"dark"}
from xpander_sdk import Configuration

config = Configuration(
    api_key="your-agent-controller-api-key",  # From the installation
    organization_id="your-org-id",
    base_url="https://agent-controller.my-company.com"
)
```

### Using with Agno Framework

```python theme={"dark"}
from xpander_sdk import Backend, Configuration
from agno.agent import Agent

config = Configuration(
    api_key="your-agent-controller-api-key",  # From the installation
    organization_id="your-org-id",
    base_url="https://agent-controller.my-company.com"
)

backend = Backend(configuration=config)
agno_agent = Agent(**backend.get_args(agent_id="agent-123", task=task))

result = await agno_agent.arun(input="What can you help me with?")
```

<Note>
  Make sure your `base_url` points to the Agent Controller endpoint (e.g., `https://agent-controller.{your-domain}`), not the root domain.

  For more SDK examples, see the [Self-Hosted SDK Configuration](/api-reference/configuration/self-hosted).
</Note>

***

## Upgrading

Upgrades work like the Datadog agent flow: watch the [xpander releases feed](https://github.com/xpander-ai/releases) (Releases page / Atom feed), then run the `upgrade` subcommand:

```bash theme={"dark"}
curl -fsSL https://charts.xpander.ai/install.sh | bash -s -- upgrade
# or, pinned:
curl -fsSL https://charts.xpander.ai/install.sh | bash -s -- upgrade --chart-version <X.Y.Z>
```

`install.sh upgrade`:

* finds the release (default `xpander` in namespace `xpander`; override with `--release` / `--namespace`),
* **infers its edition** from the installed chart — you never pass `--edition` to an upgrade,
* **reuses the values the release was installed with** (`helm get values`), plus anything extra you pass (`--set`, `--values`),
* shows current → target version with a link to the release notes, and asks before changing anything,
* always runs with `--atomic`: a failed upgrade (including a failed hook) rolls back automatically and the previous revision keeps serving.

Manual-Helm equivalent:

```bash theme={"dark"}
helm repo update
helm upgrade xpander xpander/xpander --namespace xpander --reuse-values
```

***

## Installer Flags Reference

```bash theme={"dark"}
bash install.sh [flags]            # install (the wizard, then helm upgrade --install)
bash install.sh upgrade [flags]    # upgrade an existing release in place
bash install.sh status [flags]     # read-only health report (--json for one object)
```

**Identity** (Hybrid identifies with the three console values; required non-interactively):

| Flag                       | Description                                                                                                                    |
| -------------------------- | ------------------------------------------------------------------------------------------------------------------------------ |
| `--edition hybrid\|airgap` | Edition. `hybrid` (default): worker plane here, control plane in xpander cloud. `airgap`: the whole platform here, no internet |
| `--org UUID`               | Organization ID                                                                                                                |
| `--env UUID`               | Environment ID                                                                                                                 |
| `--dm-api-key KEY`         | Deployment manager API key                                                                                                     |

**LLM providers** (all optional — the wizard asks for these):

| Flag                      | Description                                                                                                                     |
| ------------------------- | ------------------------------------------------------------------------------------------------------------------------------- |
| `--openai-key KEY`        | OpenAI                                                                                                                          |
| `--anthropic-key KEY`     | Anthropic                                                                                                                       |
| `--google-key KEY`        | Google / Gemini                                                                                                                 |
| `--bedrock-key KEY`       | Bedrock bearer token (`AWS_BEARER_TOKEN_BEDROCK`)                                                                               |
| `--bedrock-auth key\|iam` | `key` (default when `--bedrock-key` is given) or keyless IAM role / EKS pod identity — [see caveat](#amazon-bedrock-key-vs-iam) |
| `--fireworks-key KEY`     | Fireworks AI                                                                                                                    |
| `--nebius-key KEY`        | Nebius                                                                                                                          |
| `--helicone-key KEY`      | Helicone observability                                                                                                          |
| `--openrouter-key KEY`    | OpenRouter                                                                                                                      |

**Hybrid connectivity & ingress:**

| Flag              | Description                                                                                                  |
| ----------------- | ------------------------------------------------------------------------------------------------------------ |
| `--domain DOMAIN` | Enable ingress for this domain                                                                               |
| `--dash-domains`  | Dash-separated hostnames (`agent-controller-xpander.<DOMAIN>`)                                               |
| `--privatelink`   | Reach the deployment manager over [AWS PrivateLink](#privatelink)                                            |
| `--staging`       | Point at the xpander staging control plane (auto-detected for local/staging domains and local kube contexts) |
| `--dm-url URL`    | Custom deployment manager URL. `--privatelink` / `--staging` / `--dm-url` are mutually exclusive             |

**Chart and release:**

| Flag                      | Description                                    |
| ------------------------- | ---------------------------------------------- |
| `--chart-path PATH`       | Install from a local chart directory or `.tgz` |
| `--chart-version VERSION` | Pin the chart version                          |
| `--set KEY=VALUE`         | Extra helm `--set` (repeatable)                |
| `--values FILE`           | Extra helm `-f` values file (repeatable)       |
| `--namespace NAME`        | Default: `xpander`                             |
| `--release NAME`          | Default: `xpander`                             |
| `--kube-context NAME`     | kubectl/helm context                           |

**Other:**

| Flag                   | Description                                                                                                                                                                                                  |
| ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `--manual`             | Print the whole install as copy-pasteable commands, personalized from the flags you passed, and do nothing. Both editions; secrets stay placeholders. See [Advanced](#advanced-review-first--manual-install) |
| `--values-out PATH`    | Write the generated values file to PATH and exit — no cluster needed. Useful for GitOps: review the file, then `helm upgrade --install` it yourself                                                          |
| `--dry-run`            | Render everything through helm, install nothing                                                                                                                                                              |
| `-y`, `--yes`          | Accept every confirmation                                                                                                                                                                                    |
| `--non-interactive`    | Never read the terminal, even if one exists. Implies `--yes`; missing required flags exit 2 instead of prompting                                                                                             |
| `-q`, `--quiet`        | Phase lines and the outcome only                                                                                                                                                                             |
| `--json`               | `status` only: emit one JSON object                                                                                                                                                                          |
| `--help` / `--version` | Usage text / installer version                                                                                                                                                                               |

Air-gap-only flags (license, mirroring, image tags, data tier, base domain, code execution): [Air-Gapped edition](/self-hosted/airgap).

***

## Automation & CI

The installer is built to be driven by scripts, CI pipelines, and coding agents:

* **Every prompt has a flag.** With `--non-interactive` nothing is read from the terminal, even when one exists: missing required values exit 2 with the usage and a completed rerun line rather than blocking. It implies `--yes`.
* **Stable progress output.** Progress is one line per transition in a grep-friendly shape — `[3/6] chart-source: ok` — and `--quiet` reduces output to phase lines and the outcome only. Color is emitted only when stdout is a terminal.
* **`status` is the poll target.** A read-only health report: edition, chart, release status, hook jobs, workload readiness, and URLs. `--json` emits one JSON object.

**Exit codes:**

| Code | Meaning                                                                                           |
| ---- | ------------------------------------------------------------------------------------------------- |
| `0`  | Success                                                                                           |
| `1`  | Install or upgrade failure (`status`: degraded)                                                   |
| `2`  | Usage error, or a required flag missing with no terminal                                          |
| `3`  | `status`: no such release installed                                                               |
| `4`  | Preflight failure (missing helm/kubectl, unreachable cluster, version too old)                    |
| `5`  | External data-tier probe failed and `--on-probe-failure` was `abort` (air-gap external data tier) |

**Hybrid, end to end, no terminal:**

```bash theme={"dark"}
bash install.sh --non-interactive --quiet \
  --edition hybrid --org <uuid> --env <uuid> --dm-api-key <key> \
  --domain xpander.example.com --openai-key <key>
```

**Poll until healthy:**

```bash theme={"dark"}
until bash install.sh status --quiet; do sleep 30; done
```

When a scripted run fails, the exit code plus the last `FIX:`/`RERUN:` lines tell you what to repair — see [Troubleshooting](#troubleshooting).

***

## Advanced: Review-First / Manual Install

For teams that review before they run — everything the installer does can be inspected first, reproduced by hand, or both.

### Read the installer before running it

```bash theme={"dark"}
curl -fsSL https://charts.xpander.ai/install.sh | less
curl -fsSL https://charts.xpander.ai/install-v<version>.sh | less   # this exact version
```

* The **versioned URL is immutable** — `install-v<version>.sh` is published once per release and never rewritten in place, so pin it in runbooks and CI (`bash install.sh --version` prints the version you're holding).
* The script is **shellcheck-clean**, enforced by CI on every push.
* **Secrets never touch the helm command line**: they are written to a temporary values file (mode 600, removed on exit), so they don't land in shell history or process listings.
* **Nothing runs against a cluster before the kube-context confirmation** — the installer names the context and cluster it is about to install into and waits for your yes.

### Do it yourself

| Path                | What you get                                                                                                                                                                                                          |
| ------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `--manual`          | The whole install as copy-pasteable commands — repo setup, secrets, a complete values file, the helm command, verification — personalized from the flags you passed, **executing nothing**. Secrets stay placeholders |
| `--values-out PATH` | Just the generated values file (mode 600), plus the printed `helm upgrade --install` command to run yourself. No cluster needed                                                                                       |
| `--dry-run`         | The full flow rendered through `helm --dry-run`, installing nothing                                                                                                                                                   |

```bash theme={"dark"}
bash install.sh --edition hybrid --org <ORG> --env <ENV> --dm-api-key <KEY> --manual
```

<Accordion title="Manual Helm install (no installer at all)">
  The installer is a wrapper around `helm upgrade --install`; everything it does can be done by hand. The cleanest middle path is `--values-out`:

  ```bash theme={"dark"}
  bash install.sh --edition hybrid --org <ORG> --env <ENV> --dm-api-key <KEY> \
    --domain <DOMAIN> --values-out xpander-values.yaml
  helm upgrade --install xpander xpander/xpander \
    --namespace xpander --create-namespace -f xpander-values.yaml
  ```

  Or fully by hand. Add the repository:

  ```bash theme={"dark"}
  helm repo add xpander https://charts.xpander.ai
  helm repo update
  ```

  Set the deployment manager URL based on your connectivity choice — public `https://deployment-manager.xpander.ai`, or `https://deployment-manager-privatelink.xpander.ai` with [PrivateLink](#privatelink) — and install:

  ```bash theme={"dark"}
  helm upgrade --install xpander xpander/xpander \
    --namespace xpander --create-namespace \
    --set ingress.enabled=true \
    --set domain=<DOMAIN> \
    --set global.organizationId=<ORGANIZATION_ID> \
    --set global.environmentId=<ENVIRONMENT_ID> \
    --set secrets.static.deploymentManagerApiKey=<DEPLOYMENT_MANAGER_API_KEY> \
    --set urls.deploymentManager=<DEPLOYMENT_MANAGER_URL> \
    --timeout 10m
  ```

  LLM provider keys are `secrets.static.*` values — the full list is in the [key mapping table](#managing-llm-api-keys). Common optional parameters:

  | Parameter                    | Default                 | Description                                                                           |
  | ---------------------------- | ----------------------- | ------------------------------------------------------------------------------------- |
  | `domain`                     | `""` (ingress disabled) | Base domain for ingress hostnames                                                     |
  | `ingress.enabled`            | `false`                 | Expose services via ingress                                                           |
  | `ingress.tls.enabled`        | `false`                 | Enable TLS on ingress                                                                 |
  | `ingress.tls.source`         | `self-signed`           | `self-signed`, `cert-manager`, or `external`                                          |
  | `redis.storage.size`         | `8Gi`                   | Redis PVC size                                                                        |
  | `redis.storage.storageClass` | cluster default         | Storage class for Redis PVC                                                           |
  | `dockerRegistry.enabled`     | `true`                  | In-cluster registry for agent container images — disable if your cluster provides one |
  | `metricsServer.enabled`      | `true`                  | metrics-server for autoscaling — disable if your cluster runs one                     |

  For production, use a values file instead of `--set` flags:

  ```yaml xpander-values.yaml theme={"dark"}
  domain: "xpander.production.com"

  global:
    organizationId: "your-org-id"
    environmentId: "your-env-id"
    env:
      LOG_LEVEL: "info"
      ENVIRONMENT: "production"

  urls:
    deploymentManager: "https://deployment-manager.xpander.ai"

  secrets:
    static:
      deploymentManagerApiKey: "your-key"
      agentsOpenaiApiKey: "sk-your-openai-key"
      anthropicApiKey: "sk-ant-your-anthropic-key"

  ingress:
    enabled: true
    tls:
      enabled: true
      source: "cert-manager"
    annotations:
      cert-manager.io/cluster-issuer: "letsencrypt-prod"
      nginx.ingress.kubernetes.io/ssl-redirect: "true"

  resources:
    agentController:
      requests:
        cpu: "1000m"
        memory: "2Gi"
      limits:
        cpu: "3000m"
        memory: "4Gi"

  redis:
    storage:
      size: "32Gi"
      storageClass: "fast-ssd"
  ```

  ```bash theme={"dark"}
  helm upgrade --install xpander xpander/xpander \
    --namespace xpander --create-namespace \
    --values xpander-values.yaml
  ```

  See [Requirements & Sizing](#requirements--sizing) for recommended resource values per service.
</Accordion>

***

## Troubleshooting

Every installer failure prints two lines: **`FIX:`** — the action that repairs the condition — and **`RERUN:`** — your personalized non-interactive one-liner (secrets typed at prompts are shown as placeholders). The installer is idempotent: fix the condition, paste the rerun line, and it picks up where the failure left off. A secret-scrubbed session log is kept under `~/.xpander/logs/` (`latest.log` points at the most recent run). For scripted runs, the exit code names the failure class ([Automation & CI](#automation--ci)), and `bash install.sh status` gives a read-only health verdict.

### Installer

<AccordionGroup>
  <Accordion title="Release stuck in pending-install / pending-upgrade / failed">
    `install.sh upgrade` requires the release to be in `deployed` status — operating on a stuck release (`pending-install`, `pending-upgrade`, `pending-rollback`, `failed`, `uninstalling`) would strand it. Recover first:

    ```bash theme={"dark"}
    helm history xpander -n xpander
    helm rollback xpander <last-good-revision> -n xpander   # if a good revision exists
    helm uninstall xpander -n xpander                        # first install that never converged
    ```

    Then paste your `RERUN:` line. A `pending-install` status usually means a first install was interrupted (Ctrl-C, lost connection) or timed out — `helm uninstall` is the right recovery for it.
  </Accordion>

  <Accordion title="Release already runs the other edition's chart">
    The two editions share nothing and cannot be upgraded into each other. If the release name in the namespace already runs the other chart, the installer stops with the collision. Install under a different release or namespace (`--release` / `--namespace`), or remove the existing release first (`helm uninstall xpander -n xpander`).
  </Accordion>

  <Accordion title="Piped run exits with 'these required flags are missing'">
    Piping the installer into `bash` leaves no stdin to prompt on, so a piped run is non-interactive (the same contract as `--non-interactive`): Hybrid requires `--org`, `--env`, and `--dm-api-key`. Missing flags exit with code 2 and a completed rerun line. Copy the personalized one-liner from the xpander console — it carries the identity flags — or download the script and run `bash install.sh` for the interactive wizard.
  </Accordion>
</AccordionGroup>

### Cluster and Pods

<AccordionGroup>
  <Accordion title="Pods stuck in Init">
    The init containers wait for `agent-controller` to be ready. Check its logs:

    ```bash theme={"dark"}
    kubectl logs -n xpander deployment/xpander-agent-controller
    ```

    **Common cause:** Cannot reach the deployment manager. If using PrivateLink, verify the DNS and security groups are configured correctly — see [PrivateLink](#privatelink).
  </Accordion>

  <Accordion title="Pods stuck in Pending">
    **Cause:** Storage class or PVC issues.

    ```bash theme={"dark"}
    kubectl -n xpander get pvc
    kubectl -n xpander describe pvc
    ```

    Check that a default StorageClass exists and the EBS CSI driver is running:

    ```bash theme={"dark"}
    kubectl get storageclass  # Should show a default (gp3)
    kubectl get pods -n kube-system -l app.kubernetes.io/name=aws-ebs-csi-driver  # Should show Running
    ```

    If using EKS, ensure the node role has `AmazonEBSCSIDriverPolicy` attached.
  </Accordion>

  <Accordion title="exec format error">
    You are running on ARM/Graviton nodes. xpander images are **amd64 only**. Switch to x86 instance types (`t3`, `m5`, `c5`, etc.).
  </Accordion>

  <Accordion title="Insufficient CPU">
    The `agent-worker` pod requests 2 CPU by default. Options:

    * Add more nodes or use larger instances
    * For non-production environments only:

    ```bash theme={"dark"}
    kubectl patch deployment xpander-agent-worker -n xpander \
      --type='json' \
      -p='[{"op":"replace","path":"/spec/template/spec/containers/0/resources/requests/cpu","value":"500m"}]'
    ```
  </Accordion>

  <Accordion title="Health checks failing">
    Check application logs:

    ```bash theme={"dark"}
    kubectl -n xpander logs deployment/xpander-agent-controller
    kubectl -n xpander logs deployment/xpander-ai-gateway
    kubectl -n xpander logs deployment/xpander-agent-worker
    kubectl -n xpander logs deployment/xpander-mcp
    ```
  </Accordion>
</AccordionGroup>

### PrivateLink

<AccordionGroup>
  <Accordion title="PrivateLink InvalidServiceName (cross-region)">
    When creating a VPC endpoint to the xpander service from any region other than `us-west-2`, you **must** include `--service-region us-west-2`:

    ```bash theme={"dark"}
    aws ec2 create-vpc-endpoint \
      --service-name com.amazonaws.vpce.us-west-2.vpce-svc-0101884b32f655197 \
      --service-region us-west-2 \
      ...
    ```

    Without `--service-region`, AWS looks for the service in your local region and fails with `InvalidServiceName`.
  </Accordion>

  <Accordion title="PrivateLink connection timeout (HTTP 000)">
    Check the security group on the VPC endpoint allows inbound TCP 443 from your VPC CIDR:

    ```bash theme={"dark"}
    aws ec2 authorize-security-group-ingress \
      --group-id <ENDPOINT_SG_ID> \
      --protocol tcp --port 443 --cidr <VPC_CIDR> \
      --region <REGION> --profile <PROFILE>
    ```

    Also verify the private DNS hosted zone and alias record were created correctly — see [PrivateLink → Create private DNS](#privatelink).
  </Accordion>
</AccordionGroup>

### Ingress and Networking

<AccordionGroup>
  <Accordion title="Ingress not accessible">
    Verify ingress configuration:

    ```bash theme={"dark"}
    kubectl -n xpander describe ingress
    kubectl -n xpander get ingress
    ```

    Check that the NLB was provisioned and DNS CNAME records point to it:

    ```bash theme={"dark"}
    kubectl get svc -n ingress-nginx ingress-nginx-controller \
      -o jsonpath='{.status.loadBalancer.ingress[0].hostname}'
    ```
  </Accordion>

  <Accordion title="Load Balancer not provisioning">
    **Symptom:** the ingress controller's Service stays `<pending>`. Check its events (`kubectl -n ingress-nginx describe svc ingress-nginx-controller`):

    * `could not find any suitable subnets for creating the ELB` — the VPC's subnets are missing the ELB role tags. Tag public subnets `kubernetes.io/role/elb=1` for an internet-facing NLB; for an internal one, tag the private subnets `kubernetes.io/role/internal-elb=1` and annotate the Service `service.beta.kubernetes.io/aws-load-balancer-internal: "true"`.
    * On EKS Auto Mode, ensure the cluster role trust policy includes `sts:TagSession`:

    ```json theme={"dark"}
    {
      "Effect": "Allow",
      "Principal": { "Service": "eks.amazonaws.com" },
      "Action": ["sts:AssumeRole", "sts:TagSession"]
    }
    ```
  </Accordion>

  <Accordion title="SSL certificate errors on chat URLs">
    The ACM certificate must include `*.chat.<DOMAIN>` as a subject alternative name. The chat UI generates per-thread subdomains (e.g., `moccasin-prawn.chat.<DOMAIN>`) that are not covered by `*.<DOMAIN>`.

    Request a new certificate with:

    ```bash theme={"dark"}
    aws acm request-certificate \
      --domain-name "<DOMAIN>" \
      --subject-alternative-names "*.<DOMAIN>" "*.chat.<DOMAIN>" \
      --validation-method DNS \
      --region <REGION> --profile <PROFILE>
    ```
  </Accordion>
</AccordionGroup>

### API Keys

<AccordionGroup>
  <Accordion title="API key not being picked up after helm upgrade">
    The `xpander-static` secret has a Helm resource keep policy. Set the key directly:

    ```bash theme={"dark"}
    kubectl patch secret xpander-static -n xpander --type=merge \
      -p "{\"data\":{\"ANTHROPIC_API_KEY\":\"$(echo -n '<YOUR_KEY>' | base64)\"}}"

    kubectl rollout restart deployment xpander-agent-worker -n xpander
    ```

    See the [secret field name mapping](#managing-llm-api-keys) for all key names.
  </Accordion>

  <Accordion title="LLM calls fail with a missing API key">
    Verify the keys are reaching the agent-worker pod:

    ```bash theme={"dark"}
    kubectl -n xpander exec deployment/xpander-agent-worker -- env | grep API_KEY
    ```

    If missing, re-run the upgrade with the key set (`--set secrets.static.<key>=...`), or check that your secret is correctly mounted through `envFromSecretKeys` — see [Managing LLM API Keys](#managing-llm-api-keys).
  </Accordion>
</AccordionGroup>

### Debug Commands

```bash theme={"dark"}
# Get all resources
kubectl -n xpander get all

# Check events
kubectl -n xpander get events --sort-by=.metadata.creationTimestamp

# Describe problematic pods
kubectl -n xpander describe pod <pod-name>

# Check service endpoints
kubectl -n xpander get endpoints
```

***

## Uninstalling

```bash theme={"dark"}
# Remove the Helm release
helm uninstall xpander --namespace xpander

# Clean up the namespace (optional)
kubectl delete namespace xpander
```