> ## Documentation Index
> Fetch the complete documentation index at: https://docs.xpander.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Air-Gapped Edition

> The full xpander.ai platform in your environment with no internet access — license, registry mirroring, the installer flow, data tier options, and upgrades

## Overview

The Air-Gapped edition runs **the full platform in your environment, no internet access required**\*: the web UI, the control plane services, a self-hosted Supabase stack (auth, storage, edge functions), and the data tier (PostgreSQL, Redis Stack, MongoDB) — one Helm release, one namespace, zero connection to xpander cloud. As in every self-hosted edition, MCP and external-system calls run from your cluster, and agent workspaces — the agents' computers — run there too. It installs anywhere without an internet connection: a cloud VPC in private mode (no internet or NAT gateways), or any disconnected compute environment.

<sub>\*Except outbound traffic you choose to allow: LLM providers, plus MCP and connector calls to external systems.</sub>

Three things define an air-gapped install:

1. **The license file is the identity.** It carries your organization id — there are **no console IDs and no API key**. The installer generates the environment id, and the first install's seed pipeline creates everything else in-cluster, including your organization. Single-tenant: the license binds one organization.
2. **Images come from your own registry mirror.** xpander delivers the artifact set through the gated `xpanderaihub` Docker Hub org with a per-customer read token; the installer mirrors it into a registry your cluster can reach — or you run the mirroring yourself on a sealed network.
3. **Image tags are pinned per release.** The chart refuses to render against unpinned images, and upgrades require mirroring the new bundle first.

The install runs through xpander's interactive installer wizard, invoked with `--edition airgap` — the flag selects the air-gap flow: a pre-install checklist, license validation, registry mirroring when needed, and a live watch over the chart's install pipeline.

<Note>
  "Air-gapped" means air-gapped **from xpander's cloud** — not necessarily from the internet. Internet-backed agent tools (web search, docs search, TTS) can stay enabled through the chart's internet-posture setting, or be sealed off entirely. Ask your xpander contact about the sealed hardening profile.
</Note>

### What Talks to What

The network surface in both directions — the answers a security review needs.

**Outbound (from your cluster).** Nothing is required; there is no connection to xpander at all.

| From your cluster            | To                                                                                                                                                | Purpose                                                                                                         |
| ---------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------- |
| LLM-serving components       | Your LLM endpoint — in-network vLLM/Ollama/NIM, or Amazon Bedrock over a PrivateLink VPC endpoint, or a hosted provider if you allow it           | Model calls                                                                                                     |
| Agent tools, MCP, connectors | Only the external systems you choose to allow                                                                                                     | Egress is your policy; the chart's internet-posture setting shapes which internet-backed tools are even offered |
| (install/upgrade time only)  | xpander's registry — only when you use the [in-wizard mirroring](#when-the-mirror-is-empty-the-wizard-offers-to-fill-it) from a connected machine | Image delivery; a sealed network uses the [two-hop archive flow](#sealed--manual-mirroring) instead             |

**Inbound (from your users to your cluster).** These are your network requirements for reaching the platform — all of them land on your ingress controller, over whatever path your network provides (internal DNS, VPN, private load balancer — see [Domains, Ingress & DNS](#domains-ingress--dns)):

| From                                   | To                                          | Purpose                                                           |
| -------------------------------------- | ------------------------------------------- | ----------------------------------------------------------------- |
| Users' browsers                        | `chat.<domain>`, `supabase.<domain>`        | Using the web UI: invoking agents, chat, sign-in and file storage |
| SDK and API clients, CI pipelines      | `api.<domain>`, `agent-controller.<domain>` | Invoking agents programmatically (REST, streaming, SDK)           |
| In-network systems that trigger agents | `webhook.<domain>`                          | Webhook-triggered agent runs                                      |
| OAuth redirect flows (browser)         | `client-auth.<domain>`                      | Authorizing connectors                                            |

***

## Prerequisites

| Requirement        | Value                                                                                                                                                                                                                                                                                                                                                         |
| ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Nodes              | 5 × 16 vCPU / 64 GiB, amd64                                                                                                                                                                                                                                                                                                                                   |
| Kubernetes         | ≥ 1.28, with a working CNI and cluster DNS (CoreDNS), and a **CSI driver with a default RWO StorageClass** — the platform's stateful services provision PVCs through it. On EKS that means the `vpc-cni`, `coredns`, `kube-proxy`, and `aws-ebs-csi-driver` add-ons ([EKS Cluster Setup](/self-hosted/eks-setup#eks-add-ons)); metrics-server is not required |
| Storage            | 300–500 GiB PVC on an RWO StorageClass — **encrypted at rest** (the PVCs hold agent workspaces and customer data; on EKS: `encrypted: "true"` in the [StorageClass](/self-hosted/eks-setup#create-default-storageclass))                                                                                                                                      |
| Network            | **No internet required** (except outbound traffic you choose to allow: LLM providers, plus MCP and connector calls to external systems). An LLM endpoint reachable in-network                                                                                                                                                                                 |
| License file       | Issued by the xpander.ai team and delivered over email                                                                                                                                                                                                                                                                                                        |
| Container registry | Reachable by the cluster — the installer can fill it with the mirrored images for you                                                                                                                                                                                                                                                                         |
| Optional           | One GPU node (L4/A10 class) for voice / local inference                                                                                                                                                                                                                                                                                                       |
| Tooling            | `helm` ≥ 3.13 and `kubectl` on the machine you install from; `skopeo` + `python3` on whichever machine mirrors the images (Docker alone also works); `crane` or `skopeo` recommended for the installer's mirror verification                                                                                                                                  |

To compare against the currently published requirements, see the chart's [requirements.json](https://charts.xpander.ai/requirements.json).

### Sizing Tiers

The Air-Gapped edition runs the whole platform in-cluster — the worker plane **plus** the control plane services, the web UI, the Supabase stack (its own PostgreSQL 15, auth, storage, edge functions), and MongoDB.

| Tier                   | Nodes                                       | Fits                                                                                  | Storage (PVC)   |
| ---------------------- | ------------------------------------------- | ------------------------------------------------------------------------------------- | --------------- |
| Initial deployment     | 3 × (8 vCPU / 32 GiB)                       | \~10 users, 5–10 concurrent agent runs                                                | \~200 GiB       |
| Production             | 5 × (16 vCPU / 64 GiB)                      | \~50 users, 30–50 concurrent workspaces, headroom                                     | 300–500 GiB     |
| + GPU (only if needed) | +1 GPU node (L4/A10 class, 8 vCPU / 32 GiB) | Local voice STT; in-cluster LLM inference is your own pool, sized separately by model | + model weights |

Storage breakdown at the production tier: Supabase PostgreSQL 50 GiB, platform PostgreSQL 50 GiB, Supabase Storage 50 GiB+, MongoDB 20 GiB, Redis 8 GiB, registry \~50 GiB, mirror caches 50–100 GiB — all RWO, one StorageClass (the chart's preflight verifies it exists).

Chart defaults are single-replica. For production, run the hot path at ×2 replicas: agent-controller, agent-worker, api, deployment-manager, agents, functions.

### Scaling with Usage

The chart ships **fixed replica counts and no autoscalers** — scaling policy is yours. What actually grows with usage, and how to scale each:

| What grows                                                          | Grows with                                                                                                                       | How to scale it                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| ------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **`agent-worker`** — executes agent tasks                           | Concurrent agent runs. Each replica runs up to 4 tasks at once (`MAX_CONCURRENT_EXECUTIONS`); beyond `replicas × 4`, tasks queue | The primary scaler: raise `agent-worker.replicas`. Workers are stateless — adding replicas is always safe. If you want this automated, don't autoscale workers on CPU: a worker waiting on a model reply looks idle even when it's full. The honest signal is **how many runs are waiting to start** — the platform keeps that queue in its Redis — and [KEDA](https://keda.sh) (an open-source Kubernetes autoscaler) can watch that queue's length and add or remove worker replicas to keep pace |
| **Agent workspace pods** (`sb-*`) — the agents' computers           | Number of agents with an active workspace — one pod + PVC each, created on demand by agent-controller and reaped when idle       | Not a Deployment you own, so not HPA territory — this is **node capacity**. Size headroom for your expected concurrent workspaces, or run a node autoscaler                                                                                                                                                                                                                                                                                                                                         |
| **Code-execution Jobs** — spawned per run by the actions service    | Bursts of agent code execution                                                                                                   | Node capacity, same as workspaces                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| **Request path** — `api`, `agent-controller`, `agents`, `functions` | Users and API traffic; `functions` also runs CPU-heavy local embedding during knowledge-base ingest                              | Raise `<name>.replicas`; standard CPU-based HPA works here **if you install metrics-server** (not required otherwise)                                                                                                                                                                                                                                                                                                                                                                               |

**Node capacity**: workspaces and code-execution Jobs create pods on demand, so the real ceiling on concurrent agent activity is how many nodes you have. Two workable postures:

* **Static capacity** — size the node group for your peak concurrent workspaces and leave it fixed. Simplest, fully offline, and the recommended posture for sealed networks.
* **Cluster Autoscaler** — adds nodes automatically when pods can't schedule. It works in a no-internet VPC, but it talks to your cloud's scaling API to do its job, so that API must be reachable from inside the network (on AWS: the `autoscaling` and `ec2` VPC endpoints).

Karpenter is not a good fit for sealed networks — it fetches instance pricing data from the internet.

***

## Install in Three Steps

<Steps>
  <Step title="Receive the license and pull token by email">
    The xpander.ai team issues and emails you two things: **`license.json`** — the signed `xpander-license/v1` envelope that is both your entitlement and your identity (it carries your organization id; preflight validates its shape, expiry, and Ed25519 signature) — and **a registry pull token** for the gated `xpanderaihub` Docker Hub org (read-only, scoped to your account, individually revocable; it gates *delivery* only, the license is the enforcement).
  </Step>

  <Step title="Run the one-liner">
    The license file is the identity — nothing else needs to be filled in up front:

    ```bash theme={"dark"}
    curl -fsSL https://charts.xpander.ai/install.sh | bash -s -- \
      --edition airgap --license-file ./license.json
    ```

    The command downloads the installer from `charts.xpander.ai` and runs an interactive wizard — preflight checks, configuration, Helm install, and a seeded admin login to sign in with when it finishes. Run it from a terminal with `kubectl` access to the target cluster.

    The wizard prompts for the rest — registry, chart source (OCI pull or local path), data tier, admin email, organization display name, StorageClass, whether to enable agent code execution (`--code-execution` / `--no-code-execution` — the runner embeds dockerd, so its pod must be privileged; a restricted PodSecurity policy refuses it), and a [base domain](#domains-ingress--dns) for the public hosts — then the LLM provider keys, for any hosted providers your egress policy allows. Image tags are not typed in: they are [resolved from your mirror](#image-tags-resolved-from-your-mirror) and confirmed with one Enter.

    **If your registry has no xpander images yet, the wizard detects it and offers to [mirror them right there](#when-the-mirror-is-empty-the-wizard-offers-to-fill-it)** — it asks for the pull token from step 1, copies every image with per-image progress, and creates ECR repositories automatically. On a sealed network, choose the runbook option instead and see [Sealed / manual mirroring](#sealed--manual-mirroring).
  </Step>

  <Step title="Sign in as the seeded admin">
    The first install's seed pipeline creates your organization and a bootstrap admin user in your own in-cluster Supabase. The closing summary prints the sign-in commands — see [Signing In](#signing-in).
  </Step>
</Steps>

A complete non-interactive invocation (image tags resolve from the mirror unless pinned):

```bash theme={"dark"}
bash install.sh --edition airgap \
  --license-file ./license.json \
  --registry <your-registry>/xpander \
  --chart-path xpander-airgap-<X.Y.Z>.tgz \
  --domain xpander.acme.internal \
  --admin-email <you@company.example> --org-name "<Your Company>" --yes
```

Prefer to see every command before anything runs? `bash install.sh --edition airgap --manual` prints the whole install as a personalized, copy-pasteable runbook and executes nothing — see [Advanced: review-first / manual install](/self-hosted/hybrid#advanced-review-first--manual-install).

***

## What Happens When You Run the Installer

The full anatomy of an install, in order. Steps 1–6 take a few minutes altogether; step 7 — the cluster actually coming up — is the long one, typically **10–20 minutes**. Each phase is announced as a stable `[N/8] name: ...` progress line (the shape scripts grep — see [Automation & CI](#automation--ci)).

<Note>
  Wherever a step can fail, the failure is specific: every hook failure **names the Job**, prints its **last 30 log lines**, and closes with a `FIX:` line (the repair) and a `RERUN:` line (your personalized command to resume). Nothing needs diagnosing from scratch.
</Note>

<Steps>
  <Step title="Resolve — every image pinned at its own version">
    Each xpander image is versioned independently — there is no single fleet tag. The installer queries **your** registry per repository, resolves each image's own latest release version, and writes them into a pinned manifest values file, path shown:

    ```
    ✓ pinned 20 images at their own latest (~/.xpander/logs/mirror-manifest-<timestamp>.yaml)
    ```

    That manifest rides into the Helm install, so exactly what was resolved is what runs — and the file is yours to keep, diff, or replay. [More on resolution](#image-tags-resolved-from-your-mirror), including what happens when the mirror is still empty.
  </Step>

  <Step title="Checklist — preconditions verified before anything changes">
    Every requirement is checked and printed with the reason it matters: the registry answers `/v2/` (a 401 counts as up), representative images (`agent-controller`, `deployment-manager`) exist in the mirror — **each probed at its own pinned tag**, a usable RWO StorageClass exists, the image pins are in place (`N images pinned, each at its own version`), the org seed has its inputs (`--org-name` + `--admin-email`), and a chart source exists. Any failed item stops the run here, with nothing to clean up.
  </Step>

  <Step title="License — validated locally, applied as a Secret">
    The license file is checked (JSON shape, the `xpander-license/v1` envelope, expiry, the org id it is bound to), then applied idempotently as the `xpander-license` Secret. The chart wires it to every license-consuming service and verifies the **Ed25519 signature** against xpander's issuer public key — in the preflight hook, and again at deployment-manager boot.
  </Step>

  <Step title="Chart — pulled over OCI">
    The `xpander-airgap` chart is pulled from the gated OCI registry with your read token — or from your own mirror if you mirrored the chart next to the images, or from a hand-carried `.tgz` via `--chart-path`.
  </Step>

  <Step title="Values — secrets never touch the command line">
    Your configuration is written to a temporary values file (mode 600, removed on exit) together with the pinned image manifest. Secrets never appear on the helm command line, in shell history, or in process listings.
  </Step>

  <Step title="Install — helm, without --atomic on purpose">
    `helm upgrade --install` runs in the background (its log path is printed). The **first** install deliberately runs without `--atomic`: `--atomic` implies `--wait`, which would hold the post-install seed hooks until every pod is Ready — while the api and agent-controller pods are themselves waiting on that seed. The smoke hook (step 7) is the real readiness gate. **Upgrades always run with `--atomic`**: a failed upgrade rolls back automatically and the previous revision keeps serving.
  </Step>

  <Step title="Pipeline — the cluster comes up, hook by hook (10–20 min)">
    The chart installs through an ordered hook pipeline; the installer watches it live and reports each stage as it starts and completes:

    ```
    preflight → keygen → data-checks → workloads → migrations → org-seed → catalog-seed → smoke
    ```

    | Stage            | What it does — and where things fail                                                                                                                                                                                                                                                                   |
    | ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
    | **preflight**    | In-cluster checks: StorageClass, node architecture, and the license's shape, expiry, and **Ed25519 signature**. A bad license fails here in one readable message instead of crash-looping every consumer                                                                                               |
    | **keygen**       | Every platform key is **minted in your cluster**: 15+ inter-service API keys, the Supabase JWTs, and the bootstrap admin password, all into the `xpander-generated-keys` Secret. **Lookup-preserve**: a key already in the Secret is never regenerated, on any upgrade — your keys outlive the release |
    | **data-checks**  | External data tier only: `pg_isready` and `redis-cli ping` run **from inside the cluster**, exactly where the platform will connect from — security-group and network-policy problems surface here, by name, before any workload starts                                                                |
    | **workloads**    | The data tier, Supabase stack, control plane services, and UI roll out                                                                                                                                                                                                                                 |
    | **migrations**   | The **full database schema is replayed** into your in-cluster Supabase — a versioned Job per revision, kept as an audit trail. Forward-only: a schema rollback means restore-from-backup                                                                                                               |
    | **org-seed**     | **Your organization is created in your own cluster** — the org the license is bound to, the bootstrap admin user, and the platform vault. Idempotent: it re-runs safely on every upgrade                                                                                                               |
    | **catalog-seed** | The connector library loads from the mirrored artifacts image                                                                                                                                                                                                                                          |
    | **smoke**        | The end-to-end gate: every enabled component is probed through its Service, and the install only reports success when they all answer                                                                                                                                                                  |
  </Step>

  <Step title="Summary — URLs, password, logs">
    A closing box prints the release facts, the **derived URLs** (with a [base domain](#domains-ingress--dns) set: the UI at `https://chat.<domain>`, the API at `https://api.<domain>`, Supabase at `https://supabase.<domain>`), and the **bootstrap admin password itself** — plus the command to read it again later:

    ```bash theme={"dark"}
    kubectl -n xpander get secret xpander-generated-keys \
      -o jsonpath='{.data.XPANDER_ADMIN_PASSWORD}' | base64 -d
    ```

    The full session log (secret-scrubbed, mode 600) is saved under `~/.xpander/logs/` — `latest.log` always points at the most recent run, ready to review and share with support if you need help.
  </Step>
</Steps>

***

## Image Tags: Resolved from Your Mirror

You do not type image tags in — transcribing release tags from a manifest is exactly the step that goes wrong quietly. Every xpander image is **versioned independently** (there is no single fleet tag), so the installer reads your mirror over the **Docker Registry v2 API** (Docker Hub, Harbor, ECR, GitLab, and anonymous mirrors all work, including 401 bearer-token challenges) and resolves **each repository's own latest** release version — `X.Y.Z` first (a leading `v` is fine), then `X.Y`, then `latest`; pre-release tags are skipped.

The full set is written to a pinned manifest values file (path printed: `~/.xpander/logs/mirror-manifest-<timestamp>.yaml`) that rides into the Helm install. `migrations-runner` and `edge-functions` are optional: a missing repo just means that piece stays off, as documented below. The result is summarized and confirmed with a single Enter.

* **Explicit flags always win**: `--image-tag` pins deliberately — the installer honours it and does not go looking for anything newer; same for `--migrations-tag` / `--edge-functions-tag`, and a digest-pinned `--values` file stays in charge when given.
* **An unresolvable mirror is treated as an empty one**: the wizard first offers to [fill it](#when-the-mirror-is-empty-the-wizard-offers-to-fill-it), and only then falls back to asking for the tags from your bundle manifest; a non-interactive run stops with a `FIX:` line instead.
* The resolution is **reproducible**: the pinned manifest is a file you keep, and the `RERUN:` line replays the same install.

### When the Mirror Is Empty, the Wizard Offers to Fill It

A registry that answers but carries no xpander images is a thing to fix, not a thing to ask tag questions about. When resolution finds nothing, the wizard checks whether this machine has outbound internet and offers to mirror the images right there:

* **Mirror them now** — it asks for the xpander pull token from your delivery email (username defaults to `xpanderaihub`; input stays hidden) and, optionally, credentials for *your* registry (Enter reuses what `docker login` already holds). It resolves the current release from xpander's registry, then copies every image the install pulls — platform, UI, data tier, Supabase stack, artifact images — with per-image progress (`[12/47] mirroring agent-controller:0.3.2 ... ok`). **ECR destinations get their repositories created automatically** (idempotently, per repo). The copied versions are pinned into the same [per-image manifest](#image-tags-resolved-from-your-mirror) the install uses, tags re-resolve from your now-populated mirror, and the install continues. Re-running after a partial failure is safe — copies that already landed are skipped fast.
* **Print the sealed-network runbook** — for machines with no outbound internet; see [Sealed / manual mirroring](#sealed--manual-mirroring).
* **Stop here** — nothing has changed.

Non-interactively, `--mirror now|guide|skip` preselects that answer (with `now`, pass `--xpander-registry-token`, and `--xpander-registry-user` if it isn't `xpanderaihub`; `--dest-registry-user`/`--dest-registry-token` cover a destination registry your docker login doesn't). The mirroring choice and credentials ride the `RERUN:` line like everything else.

However a tag arrives — resolved or explicit — this is what each pin controls:

| Tag                        | Without it                                                                                                                                                                                                                                     |
| -------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `--image-tag TAG`          | **The install refuses to render.** The chart has a placeholder-image render guard: unpinned platform images are an error, not a default. A digest-pinned values file from `image_sync.sh --manifest` (passed via `--values`) also satisfies it |
| `--migrations-tag TAG`     | The in-cluster Supabase stack stays **off**: no schema replay, no seeded org, no admin user to sign in as — every control-plane service runs degraded. The installer says so and the release NOTES repeat it                                   |
| `--edge-functions-tag TAG` | The platform installs and runs, but **edge functions stay unavailable** until an edge-functions image is pinned                                                                                                                                |
| `--no-supabase`            | Keeps the Supabase stack off even with a migrations tag — for data-tier-only and staged installs                                                                                                                                               |

Turning the Supabase stack on later is one upgrade:

```bash theme={"dark"}
bash install.sh upgrade --release xpander --namespace xpander \
  --set supabase.enabled=true --set migrations.image.tag=<tag>
```

***

## Sealed / Manual Mirroring

When the install host has no outbound internet — or you want to run the mirroring yourself — use `image_sync.sh`, the same tool the wizard's in-place mirroring runs. It is published standalone and resolves the current release tags itself:

```bash theme={"dark"}
curl -fsSLO https://charts.xpander.ai/image_sync.sh
docker login -u xpanderaihub          # the pull token from your delivery email
bash image_sync.sh --source docker.io/xpanderaihub --dest <your-registry>/xpander
```

`--tag` pins one release instead of the newest, `--dry-run` prints the copy plan without copying, and `--manifest bundle-manifest.json` mirrors platform images by digest when you have a release bundle manifest. It needs `skopeo` and `python3` (with only Docker installed it falls back to pull/tag/push), prints per-image progress, creates ECR repositories automatically, and re-running after a failure is safe — copies that already landed are skipped fast.

**Fully sealed network?** Do it in two hops — archive on the connected side, carry the directory in, load on the inside:

```bash theme={"dark"}
# on a connected workstation:
bash image_sync.sh --source docker.io/xpanderaihub --archive-dir ./xpander-images
# carry ./xpander-images across the gap, then, on the inside:
bash image_sync.sh --from-archive ./xpander-images --dest <your-registry>/xpander
```

The chart itself is an OCI artifact. Either mirror it next to the images (`--chart-version <X.Y.Z> --chart-source registry-1.docker.io/xpanderaihub/xpander-airgap` — it lands at `oci://<your-registry>/xpander/charts/xpander-airgap`), or pull and hand-carry the `.tgz`:

```bash theme={"dark"}
helm registry login registry-1.docker.io --username <user> --password-stdin
helm pull oci://registry-1.docker.io/xpanderaihub/xpander-airgap --version <X.Y.Z>
# copy xpander-airgap-<X.Y.Z>.tgz onto the sealed-network host, then --chart-path it
```

<Accordion title="Using Nexus, Artifactory, or Harbor">
  A registry manager fits either of two patterns:

  **Hosted repository — mirror into it.** The registry manager is the `--registry` target: create a hosted Docker repository and fill it with the wizard's mirror-now step, or with `image_sync.sh` directly — `--dest-user` / `--dest-token` cover repositories behind Basic auth:

  ```bash theme={"dark"}
  bash image_sync.sh --source docker.io/xpanderaihub \
    --dest nexus.corp.example:5000/xpander \
    --dest-user <user> --dest-token <token>
  ```

  Then install with `--registry nexus.corp.example:5000/xpander`.

  **Pull-through proxy — no mirror step.** Create a Docker *proxy* repository with Docker Hub (`https://registry-1.docker.io`) as the upstream, and set the xpander-issued read token as the proxy's **upstream** credentials. Install with the org path kept inside the registry value:

  ```bash theme={"dark"}
  --registry <nexus-host>/<proxy-repo>/xpanderaihub
  ```

  Images cache lazily in the proxy on their first pull — nothing to mirror. The `xpanderaihub` path segment must remain: the chart composes every image reference as `<registry>/<name>`, so the cluster asks the proxy for `<proxy-repo>/xpanderaihub/agent-controller`, which maps to `xpanderaihub/agent-controller` upstream. The installer's tag resolution works against the proxy the same way (Docker Registry v2 API, Basic or bearer auth).

  A **fully sealed** network can't use a proxy — it needs upstream reachability. Keep the two-hop archive flow above, landing in a hosted repository.
</Accordion>

<Tip>
  The installer prints this entire runbook — personalized with your registry and namespace — with `bash install.sh --edition airgap --guide`, and prints it automatically when it detects no reachable cluster. `--manual` goes further: the [whole install as copy-pasteable commands](/self-hosted/hybrid#advanced-review-first--manual-install), executing nothing.
</Tip>

***

## Data Tier: In-Cluster or External

The data tier is the one choice that is expensive to change later — the chart-managed StatefulSets own their PVCs, and moving to a managed service afterwards is a migration. The wizard asks per store; the default is both in-cluster.

<Tabs>
  <Tab title="In-cluster (default)">
    Chart-managed StatefulSets on your StorageClass:

    * **PostgreSQL 16** — durable history/memory tier and per-agent database provisioning
    * **Redis Stack** (`redis-stack-server`) — queues, cache, and knowledge-base vectors. Deliberately not plain Redis: knowledge bases are RediSearch (`FT.*`) indexes
    * **MongoDB** — catalog and monitoring stores
    * The Supabase stack ships its **own** pinned PostgreSQL 15 (it requires extensions managed services don't offer)

    No flags needed. Set `--storage-class <name>` if you don't want the cluster default.
  </Tab>

  <Tab title="External (RDS / MemoryDB)">
    Bring your own Postgres and/or Redis:

    ```bash theme={"dark"}
    bash install.sh --edition airgap ... \
      --pg-mode external --pg-dsn 'postgresql://xpander:password@db.internal:5432/xpander' \
      --redis-mode external --redis-endpoint my-redis.internal \
      --redis-tls --redis-cluster --redis-username <acl-user> --redis-password <pass>
    ```

    * **Postgres**: pass the full DSN. The role needs **`CREATEDB`** — the platform provisions per-agent databases on this server.
    * **Redis**: the endpoint must ship the **RediSearch (`FT.*`) commands**, or knowledge bases accept writes and return nothing. MemoryDB and ElastiCache with in-transit encryption answer TLS only — pass `--redis-tls`. Pass `--redis-cluster` for cluster-mode endpoints, `--redis-username` / `--redis-password` for ACL auth, and `--redis-port` if the endpoint is not on 6379.
    * **MongoDB has no external mode** — it always runs in-cluster as a chart-managed StatefulSet.
    * Supabase's own Postgres is **never external** — its migrations hard-require extensions RDS does not ship; it always runs in-cluster.

    **Live connectivity probes**: before installing, the installer launches a probe pod inside your namespace — the same images and commands the chart's own preflight hooks use (`pg_isready`, `redis-cli ping`) — and reports reachability from where the platform will actually connect. A failed probe is a decision, not an abort: retry after fixing the network, continue anyway (the chart's pre-install hook re-checks and fails the install if the store is really unreachable), or abort with nothing changed. `--on-probe-failure retry|continue|abort` preselects that answer; non-interactive runs default to `abort` and exit with code 5.
  </Tab>
</Tabs>

***

## Local LLM Inference

An air-gapped install needs an LLM endpoint reachable in-network. The platform speaks the OpenAI-compatible API, so any local runtime that exposes it plugs in directly:

| Runtime               | Models                         | Notes                                                                         |
| --------------------- | ------------------------------ | ----------------------------------------------------------------------------- |
| vLLM                  | Llama, Mistral, Qwen, DeepSeek | High-throughput inference; OpenAI-compatible API                              |
| Ollama                | Llama, Mistral, Phi, Qwen      | Easier to operate; good for smaller deployments                               |
| NVIDIA NIM            | Llama, Mistral, others         | Optimized GPU runtime; enterprise support                                     |
| AWS Bedrock (private) | Anthropic, Mistral, Llama      | Only for VPC deployments with PrivateLink to Bedrock — *not* fully air-gapped |

Size your inference pool separately by model; it is your own infrastructure, not part of the chart's footprint. Hosted providers (OpenAI, Anthropic, Google, Bedrock) also work when your egress policy allows them — the same [LLM provider keys](/self-hosted/hybrid#managing-llm-api-keys) apply.

<Warning>
  **Amazon Bedrock currently requires a Bedrock API key (bearer token)** — IAM-role / EKS-pod-identity auth alone is not enough yet: without the token, gateway chat turns work but **agent task executions fail**. Create a Bedrock API key (an IAM service-specific credential whose policy includes `bedrock:CallWithBearerToken` and `bedrock:InvokeModel*`/`bedrock:Converse*`), store it in a Secret, and point `global.llm.bedrock.bearerTokenSecret` at it — with `global.llm.bedrock.region` set to the region of your (PrivateLink) endpoint. The chart's values file documents the same requirement; a future release will lift it in favor of the AWS default credential chain.
</Warning>

***

## The License

The installer validates the license file locally (JSON shape, the `xpander-license/v1` envelope, expiry) and creates the Secret for you, idempotently:

```bash theme={"dark"}
kubectl -n xpander create secret generic xpander-license \
  --from-file=license.json=<path/to/license.json> \
  --dry-run=client -o yaml | kubectl -n xpander apply -f -
```

The chart's preflight hook then verifies the **Ed25519 signature**, and deployment-manager re-verifies at boot — it refuses to start on a bad signature.

* **Expired license**: the installer warns at the license phase and the chart's preflight refuses the install. A running install whose license expires needs a renewed license file — the xpander.ai team issues one and delivers it over email.
* **Renewing / replacing**: recreate the `xpander-license` Secret with the new file (the command above), then run `helm upgrade` (or `install.sh upgrade`) — license consumers carry a checksum annotation over the Secret, so the upgrade rolls them onto the new file.
* **Single-tenant**: the license binds one organization; the org seed creates exactly that org.

***

## Domains, Ingress & DNS

One `--domain` names every public host. Give the installer a base domain — `xpander.acme.internal` — and the chart derives the whole browser-facing set from it:

| Host                        | Serves                                                          |
| --------------------------- | --------------------------------------------------------------- |
| `chat.<domain>`             | The web UI — the address your users open                        |
| `api.<domain>`              | The REST API                                                    |
| `agent-controller.<domain>` | SDK calls and the UI's chat backend                             |
| `supabase.<domain>`         | Auth, storage, and edge functions (the `supabase-kong` gateway) |
| `webhook.<domain>`          | Webhook triggers                                                |
| `client-auth.<domain>`      | OAuth flows for connectors                                      |
| `actions.<domain>`          | The actions service                                             |

Setting the domain **is** the publish decision: an Ingress renders for each of these services (an explicit `<name>.ingress.enabled: false` keeps one private; `global.ingress.enabled: false` suppresses all of them for clusters that manage their own ingress objects). The base domain also derives the **UI's runtime configuration** — the app needs to know its Supabase, API, and chat-backend origins at boot, and without a domain (or explicit `ui.env` overrides) a browser on a sealed network would be pointed at origins it cannot reach.

Overrides and knobs:

* `--ui-host <host>` replaces the derived `chat.<domain>` only; any other host is overridable per service via `global.hosts.<name>` in values.
* `--ingress-scheme https|http` (default `https`) sets how the browser reaches those hosts — it feeds every derived URL. Use `http` only for rigs without certificates.
* No ingress controller is assumed. Set the class and annotations per component (`<name>.ingress.className` / `annotations`) or once for all of them via `global.ingress.className` / `global.ingress.annotations`.

**TLS** is the standard Kubernetes Ingress `tls:` block, passed through verbatim — per component (`<name>.ingress.tls`) or once via `global.ingress.tls`. Terminate at your controller with certificates that cover the seven hosts (a wildcard `*.xpander.acme.internal` covers them all):

```yaml theme={"dark"}
global:
  ingress:
    className: traefik
    tls:
      - hosts: ["*.xpander.acme.internal"]
        secretName: xpander-wildcard-tls
```

### Making the Hosts Resolve

<Tabs>
  <Tab title="Internal DNS zone">
    Create records in the DNS zone your users' machines resolve — a wildcard is simplest:

    ```
    *.xpander.acme.internal.  CNAME  <your-ingress-controller's address>
    ```

    Or seven explicit records (the table above) when a wildcard is not allowed.
  </Tab>

  <Tab title="VPN + internal load balancer">
    Expose the ingress controller on an **internal** load balancer reachable over the VPN, then point the records at it. In a sealed VPC the private subnets need the `kubernetes.io/role/internal-elb=1` tag and the Service the `aws-load-balancer-internal` annotation — see [Troubleshooting](#troubleshooting) for the exact commands.
  </Tab>

  <Tab title="Local test: port-forward + /etc/hosts">
    No DNS yet? One port-forward to the ingress controller serves all seven hosts — the controller routes by `Host` header. **Forward to the real scheme port** (443 for https, 80 for http): the app's derived origins carry no port number, so a forward on a non-standard local port serves the login page but breaks every call the app makes afterwards. Binding 443/80 locally needs admin/root on macOS and Linux:

    ```bash theme={"dark"}
    sudo kubectl -n <ingress-namespace> port-forward svc/<controller-service> 443:443
    # installed with --ingress-scheme http, forward port 80 instead:
    sudo kubectl -n <ingress-namespace> port-forward svc/<controller-service> 80:80
    ```

    ```
    # /etc/hosts
    127.0.0.1 chat.xpander.acme.internal api.xpander.acme.internal agent-controller.xpander.acme.internal supabase.xpander.acme.internal webhook.xpander.acme.internal client-auth.xpander.acme.internal actions.xpander.acme.internal
    ```

    Then open `https://chat.xpander.acme.internal` (or `http://…` for an `--ingress-scheme http` install) — no port in the URL.
  </Tab>
</Tabs>

<Note>
  **Sealed networks:** public DNS records for these hosts are pointless without a path in — resolution has to happen where your users are (the internal zone, or VPN-side DNS). The domain itself never needs to exist on the public internet.
</Note>

***

## Signing In

The install ends with the summary box printing the derived URLs **and the bootstrap admin password**; the release NOTES (`helm get notes xpander -n xpander`) repeat the commands. With no external dependency configured, sign-in still works:

1. **Bootstrap password** (default) — open the UI (`https://chat.<domain>`) at `/login-password` with your `--admin-email` and the password from the summary; read it again any time:

   ```bash theme={"dark"}
   kubectl -n xpander get secret xpander-generated-keys \
     -o jsonpath='{.data.XPANDER_ADMIN_PASSWORD}' | base64 -d
   ```

2. **Email OTP without SMTP** — the login page's email flow works with no mailer: the code is generated and stored, just not sent. Print it with the chart's `scripts/get_login_code.sh <email> <namespace>`. With SMTP configured (`global.smtp.*` in values), the same codes arrive by email.

3. **OIDC / OAuth / SAML** — configure your provider through the chart's Supabase auth settings (see the chart README, section "Signing in").

No domain set yet? Port-forward to the UI first:

```bash theme={"dark"}
kubectl -n xpander port-forward svc/ui 8080:80   # then http://localhost:8080
```

***

## Upgrading an Air-Gapped Install

Watch the [xpander releases feed](https://github.com/xpander-ai/releases) for new versions. An air-gap upgrade is two steps, **in this order**:

<Steps>
  <Step title="Mirror the new bundle FIRST">
    The chart pins image tags per release, so an un-mirrored upgrade leaves pods in `ImagePullBackOff`. Re-run the mirroring for the target release — `curl -fsSLO https://charts.xpander.ai/image_sync.sh` and run it against your registry as in [Sealed / manual mirroring](#sealed--manual-mirroring) (`bash install.sh --edition airgap --guide` prints the exact invocation) — and hand-carry the new chart `.tgz` if your install host can't pull it over OCI.
  </Step>

  <Step title="Run the upgrade">
    ```bash theme={"dark"}
    bash install.sh upgrade --chart-path xpander-airgap-<X.Y.Z>.tgz
    ```

    The upgrade subcommand infers the edition from the installed chart, reuses the values the release was installed with (`helm get values`), **re-resolves the image tags from the mirror this release already pulls from** and pins the fresh tags over the reused values (printing them, like the install does — pass `--image-tag` / `--migrations-tag` / `--edge-functions-tag` to pin explicitly; when the mirror cannot be queried it keeps the tags the release already runs), and always runs with `--atomic` — a failed upgrade (including a failed smoke hook) auto-rolls-back and the previous revision keeps serving.
  </Step>
</Steps>

What survives what:

* **Generated keys survive every upgrade** — the keygen hook is lookup-preserve, the `xpander-generated-keys` Secret is Job-created (never Helm-rendered), and rollback never rewinds it.
* **`helm rollback xpander <N>` restores any prior revision** — everything the chart owns is release-versioned.
* **Except the database schema.** Migrations run as a versioned Job per revision (kept as the audit trail) and are forward-only: a schema rollback means restore-from-backup, never an automatic down-migration.

<Warning>
  **Avoid `--reuse-values` on air-gap upgrades** (the installer's upgrade flow does the right thing — this applies to hand-driven `helm upgrade`). `--reuse-values` replays the previous release's *computed* values, which can resurrect settings the new chart version deliberately removed — the known case is a stale edge-runtime placeholder image pin that leaves a crash-looping Deployment failing every `--atomic` upgrade. Re-supply your values files instead, or clear the stale pin once with `--set supabase.edgeRuntime.image.repository=null --set supabase.edgeRuntime.image.tag=null`.
</Warning>

***

## Automation & CI

The installer is built to be driven by scripts, CI pipelines, and coding agents:

* **Every prompt has a flag.** With `--non-interactive` nothing is read from the terminal, even when one exists: missing required values exit 2 with the usage and a completed rerun line rather than blocking. It implies `--yes` and `--on-probe-failure abort`.
* **Stable progress output.** Progress is one line per transition in a grep-friendly shape — `[3/8] license: ok` — and `--quiet` reduces output to phase lines and the outcome only. Color is emitted only when stdout is a terminal.
* **`status` is the poll target.** A read-only health report: edition, chart, release status, hook jobs, workload readiness, and URLs. `--json` emits one JSON object.

**Exit codes:**

| Code | Meaning                                                                        |
| ---- | ------------------------------------------------------------------------------ |
| `0`  | Success                                                                        |
| `1`  | Install or upgrade failure (`status`: degraded)                                |
| `2`  | Usage error, or a required flag missing with no terminal                       |
| `3`  | `status`: no such release installed                                            |
| `4`  | Preflight failure (missing helm/kubectl, unreachable cluster, version too old) |
| `5`  | External data-tier probe failed and `--on-probe-failure` was `abort`           |

**Air-gap, end to end, no terminal** (image tags resolve from the mirror unless pinned):

```bash theme={"dark"}
bash install.sh --non-interactive --quiet \
  --edition airgap \
  --license-file ./license.json --registry registry.example:5000/xpander \
  --org-name "Example Inc" --admin-email ops@example.com \
  --domain xpander.example.internal --storage-class gp3 \
  --on-probe-failure abort
```

If the registry might not be mirrored yet, add `--mirror now --xpander-registry-token <token>` — without a `--mirror` answer, a non-interactive run against an empty mirror exits 2 with the options spelled out.

**Poll until healthy:**

```bash theme={"dark"}
until bash install.sh status --quiet; do sleep 30; done
```

### CI/CD & GitOps

<Accordion title="Jenkins (or any CI): watch the releases feed, mirror, upgrade">
  The pattern: subscribe to the [releases feed](https://github.com/xpander-ai/releases) (`releases.atom`), mirror the new bundle with credentials from the CI secret store, then upgrade non-interactively and gate on `status`. Run it on an agent with `kubectl` access to the cluster and `helm` ≥ 3.13.

  ```groovy theme={"dark"}
  pipeline {
    agent { label 'xpander-admin' }        // has kubectl access to the cluster
    triggers { cron('H 6 * * 1') }         // or trigger from releases.atom
    environment {
      XPANDER_PULL = credentials('xpander-pull-token')  // username: xpanderaihub, password: the read token
    }
    stages {
      stage('Mirror the release') {
        steps {
          sh '''
            curl -fsSLO https://charts.xpander.ai/image_sync.sh
            bash image_sync.sh --source docker.io/xpanderaihub \
              --dest registry.corp.example:5000/xpander \
              --source-user "$XPANDER_PULL_USR" --source-token "$XPANDER_PULL_PSW"
          '''
        }
      }
      stage('Upgrade and verify') {
        steps {
          sh '''
            curl -fsSL https://charts.xpander.ai/install.sh -o install.sh
            bash install.sh upgrade --non-interactive --quiet
            bash install.sh status --quiet
          '''
        }
      }
    }
  }
  ```

  Exit codes do the gating (the [table above](#automation--ci)): the upgrade step fails the build on `1` (already rolled back — upgrades run `--atomic`), `2` (usage), or `4` (preflight); `status` fails on `1` (degraded, with reasons) or `3` (not installed). `status --json` emits one JSON object if you'd rather parse than gate on the exit code.
</Accordion>

<Accordion title="ArgoCD: the chart as Helm OCI from your registry">
  The air-gap chart is standard Helm OCI. [Mirror it next to the images](#sealed--manual-mirroring), keep your values in git, and point an Application at both:

  ```yaml theme={"dark"}
  apiVersion: argoproj.io/v1alpha1
  kind: Application
  metadata:
    name: xpander
    namespace: argocd
  spec:
    project: default
    destination:
      server: https://kubernetes.default.svc
      namespace: xpander
    sources:
      - repoURL: registry.corp.example:5000/xpander/charts   # OCI repo credential with "Enable OCI"
        chart: xpander-airgap
        targetRevision: 0.3.2                                # the chart version you mirrored
        helm:
          valueFiles:
            - $values/xpander/values.yaml           # generated once: install.sh --values-out
            - $values/xpander/mirror-manifest.yaml  # the pinned per-image manifest, committed
      - repoURL: https://git.corp.example/platform/xpander-config.git
        targetRevision: main
        ref: values
    syncPolicy:
      syncOptions:
        - CreateNamespace=true
  ```

  * **Values**: generate the base file once with `install.sh --edition airgap ... --values-out xpander/values.yaml` and commit it together with the pinned mirror manifest. The generated file can carry secrets (LLM keys, external DSNs) — strip those into your secret manager before committing.
  * **Bootstrap secrets**: the `xpander-license` Secret (and the registry pull secret, when one is needed) are created once out-of-band — `kubectl create secret ...` as shown in [The License](#the-license) — or managed via SealedSecrets / External Secrets Operator.
  * **Hooks**: Argo CD maps Helm hooks (the chart's preflight → keygen → migrations → org-seed → smoke pipeline) onto its own sync phases, and hook weights onto sync waves. This mapping is supported by Argo, but we have not certified the chart under Argo CD — **verify the hook ordering on your Argo version** before relying on it for upgrades; `install.sh` remains the certified path.
</Accordion>

***

## Troubleshooting

Every installer failure prints two lines: **`FIX:`** — the action that repairs the condition — and **`RERUN:`** — your personalized non-interactive one-liner (secrets typed at prompts are shown as placeholders). The installer is idempotent: fix the condition, paste the rerun line, and it picks up where the failure left off. The air-gap checklist verifies preconditions before anything changes, so most problems stop the run with nothing to clean up. For scripted runs, the exit code names the failure class ([Automation & CI](#automation--ci)), and `bash install.sh status` gives a read-only health verdict.

### Mirroring and Registries

<AccordionGroup>
  <Accordion title="Mirror push fails: name unknown — The repository ... does not exist (ECR)">
    **Symptom:**

    ```
    writing blob: initiating layer upload to /v2/xpander/ai-gateway/blobs/uploads/ in
    123456789012.dkr.ecr.eu-west-2.amazonaws.com: name unknown: The repository with
    name 'xpander/ai-gateway' does not exist in the registry with id '123456789012'
    ```

    **Cause:** ECR refuses a push to a repository that does not exist yet, and there is no push-time flag to create one.

    **Fix:** the current `image_sync.sh` creates each ECR repository automatically before pushing (it needs AWS CLI credentials on the machine running it) — refresh your copy:

    ```bash theme={"dark"}
    curl -fsSLO https://charts.xpander.ai/image_sync.sh
    ```

    On an older copy, pre-create the repository and re-run (already-landed copies are skipped fast):

    ```bash theme={"dark"}
    aws ecr create-repository --repository-name xpander/ai-gateway --region <region>
    ```
  </Accordion>

  <Accordion title="Mirror push fails: authentication required at the destination">
    **Symptom:** `authentication required` writing to your registry — even though `docker login` to it succeeds.

    **Cause:** `skopeo` cannot read logins kept in Docker Desktop's credential store, so the docker login is invisible to the copy.

    **Fix:** the current `image_sync.sh` logs in itself for ECR destinations (`aws ecr get-login-password | skopeo login --username AWS --password-stdin`). For other registries, pass the credentials explicitly:

    ```bash theme={"dark"}
    bash image_sync.sh --source docker.io/xpanderaihub --dest <registry>/<path> \
      --dest-user <user> --dest-token <token>
    ```
  </Accordion>

  <Accordion title="401 from Docker Hub when resolving or mirroring">
    **Symptom:** `401 from auth.docker.io (incorrect username or password)` — the installer surfaces the registry's own reason.

    **Cause:** the pull credential is the **organization username** `xpanderaihub` plus the read token from your delivery email — a personal Docker Hub username returns 401, and so does a token missing its read scopes.

    **Fix:** re-run with the exact pair from the email (`--xpander-registry-user xpanderaihub --xpander-registry-token <token>`). If it still refuses, ask xpander to re-issue the token — it needs repository-read and image-pull access to the gated repos.
  </Accordion>

  <Accordion title="Pods can't pull: manifest unknown at your pinned tag">
    **Symptom:** pod events show `manifest unknown`, or the installer's checklist reports:

    ```
    ✗ deployment-manager:0.0.417 not found in registry.example:5000/xpander
    ```

    **Cause:** the mirror does not hold that repository at the pinned tag — a partial mirror, or an upgrade that pinned newer versions than were copied. Tags differ **per image** (each service versions independently), so one stale repo is enough.

    **Fix:** re-run the mirroring for the release ([in-wizard](#when-the-mirror-is-empty-the-wizard-offers-to-fill-it), or `image_sync.sh` — safe to re-run), then verify the exact coordinate before retrying:

    ```bash theme={"dark"}
    crane digest registry.example:5000/xpander/deployment-manager:0.0.417
    ```
  </Accordion>
</AccordionGroup>

### Cluster and Network

<AccordionGroup>
  <Accordion title="Nodes time out pulling from docker.io (dial tcp ... i/o timeout)">
    **Symptom:**

    ```
    failed to pull image "docker.io/...": dial tcp 44.208.254.194:443: i/o timeout
    ```

    **Cause:** a sealed cluster has no route to Docker Hub — expected. Some image reference is still pointing at `docker.io` instead of your mirror.

    **Fix:** install with `--registry <your-mirror>/<path>` so every platform image resolves in-network, and make sure the mirror itself is reachable from the **nodes** (on an ECR-only sealed VPC that means the `ecr.api` + `ecr.dkr` interface endpoints plus the S3 gateway endpoint, and ECR read on the node role). Find the offending reference with:

    ```bash theme={"dark"}
    kubectl -n xpander get pods -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.containers[*].image}{"\n"}{end}' | grep docker.io
    ```
  </Accordion>

  <Accordion title="Preflight: no node with a supported architecture (Graviton/ARM)">
    **Symptom:**

    ```
    ✗ no node with a supported architecture: cluster has [arm64], the images need [amd64]
    ```

    (Or, past preflight on a mixed cluster: pods crash with `exec format error`.)

    **Cause:** xpander images are amd64-only; ARM/Graviton nodes (`t4g`, `m7g`, `c7g`, ...) cannot run them. The installer's preflight and the chart's preflight hook both check before anything deploys.

    **Fix:** add an amd64 node group (x86 instance types) and re-run. Only if you have genuinely arm64 images should you override the chart's check: `--set 'preflight.allowedArchitectures[0]=arm64'`.
  </Accordion>

  <Accordion title="LoadBalancer Service stuck <pending>: could not find any suitable subnets">
    **Symptom:** the ingress controller's Service never gets an address; its events show:

    ```
    could not find any suitable subnets for creating the ELB
    ```

    **Cause:** in a sealed VPC there are no public subnets, and the private subnets are missing the ELB role tags — AWS has nowhere to put the load balancer.

    **Fix:** for an internal load balancer, tag the private subnets and annotate the Service:

    ```bash theme={"dark"}
    aws ec2 create-tags --resources <subnet-ids> \
      --tags Key=kubernetes.io/role/internal-elb,Value=1
    ```

    ```yaml theme={"dark"}
    metadata:
      annotations:
        service.beta.kubernetes.io/aws-load-balancer-internal: "true"
    ```

    Or skip the load balancer entirely — reach the UI over your VPN, or with `kubectl -n xpander port-forward svc/ui 8080:80`.
  </Accordion>

  <Accordion title="External data-tier probe fails: namespaces 'xpander' not found">
    **Symptom:**

    ```
    Error from server (NotFound): namespaces "xpander" not found
    ```

    **Cause:** the connectivity probe pod was launched before the install had created the namespace. Fixed in current installer versions.

    **Fix:** refresh your installer copy (`curl -fsSL https://charts.xpander.ai/install.sh`), or create the namespace and re-run:

    ```bash theme={"dark"}
    kubectl create namespace xpander
    ```
  </Accordion>

  <Accordion title="Pods stuck in Pending">
    **Symptom:** data-tier pods stay `Pending`; `kubectl -n xpander describe pvc` shows unbound claims.

    **Cause:** no usable RWO StorageClass (or the wrong one), or the cluster is under capacity for the [footprint](#prerequisites).

    **Fix:** `kubectl get storageclass` — pass `--storage-class <name>` if the default isn't right; check `kubectl -n xpander get events --sort-by=.lastTimestamp` for `FailedScheduling` and add nodes if capacity is the constraint.
  </Accordion>

  <Accordion title="UI loads blank: Missing or invalid environment variables">
    **Symptom:** the UI serves but renders a blank page; the browser console shows:

    ```
    Missing or invalid environment variables
    ```

    **Cause:** the app validates its runtime origins (Supabase, chat backend) at boot and refuses to start without them. Chart versions before **0.6.0** did not derive the UI's runtime config from the base domain.

    **Fix:** upgrade — `bash install.sh upgrade` (chart ≥ 0.6.0 injects the config from `--domain`). Staying on an older chart, set the origins yourself via `ui.env` (`VITE_SUPABASE_URL`, `VITE_CHAT_BACKEND_URL`, ...) in your values.
  </Accordion>

  <Accordion title="Only the UI host resolves; other subdomains 404 or NXDOMAIN">
    **Symptom:** `chat.<domain>` opens, but sign-in or API calls fail — `supabase.<domain>` / `api.<domain>` don't resolve, or land on a default-backend 404.

    **Cause:** the install publishes [seven hosts](#domains-ingress--dns), and only one got a DNS record (or an `/etc/hosts` entry).

    **Fix:** create a wildcard record (`*.<domain>`) pointing at the ingress controller, or one record per host: `chat`, `api`, `agent-controller`, `supabase`, `webhook`, `client-auth`, `actions`. Verify what the cluster actually publishes:

    ```bash theme={"dark"}
    kubectl -n xpander get ingress
    ```
  </Accordion>
</AccordionGroup>

### Install and Upgrade

<AccordionGroup>
  <Accordion title="Release stuck in pending-install / pending-upgrade / failed">
    The installer refuses to install over a release in a stuck status, and `install.sh upgrade` requires `deployed` — operating on a stuck release would strand it. Recover first:

    ```bash theme={"dark"}
    helm history xpander -n xpander
    helm rollback xpander <last-good-revision> -n xpander   # if a good revision exists
    helm uninstall xpander -n xpander                        # first install that never converged
    ```

    Then paste your `RERUN:` line. `pending-install` usually means an interrupted or timed-out first install — `helm uninstall` is the right recovery. The `xpander-generated-keys` Secret survives the uninstall (Job-created, not Helm-rendered), so a reinstall keeps the same keys.
  </Accordion>

  <Accordion title="'images are unpinned' / chart refuses to render">
    The chart refuses to render against unpinned platform images — in a sealed network an unpinned tag is a pod that can never pull. Normally the [tag auto-resolution](#image-tags-resolved-from-your-mirror) supplies the pins; an unresolvable registry usually means an empty mirror, and the wizard offers to fill it (`--mirror now` non-interactively). Otherwise pass `--image-tag <release-tag>` explicitly, or `--values <file>` with the digest-pinned manifest from `image_sync.sh --manifest`.
  </Accordion>

  <Accordion title="License expired or rejected">
    The installer validates the file locally (JSON shape, `xpander-license/v1` envelope, expiry, org id); the chart's preflight hook re-verifies including the Ed25519 signature, and deployment-manager verifies again at boot.

    **Expired:** the installer warns `License expired on <date> - preflight will refuse the install`. Get a renewed `license.json` from the xpander.ai team (delivered over email), recreate the secret, and re-run:

    ```bash theme={"dark"}
    kubectl -n xpander create secret generic xpander-license \
      --from-file=license.json=<new-file> \
      --dry-run=client -o yaml | kubectl -n xpander apply -f -
    ```

    On a running install, follow with `install.sh upgrade` — license consumers carry a checksum annotation over the Secret, so the upgrade rolls them onto the new file.
  </Accordion>

  <Accordion title="A pipeline stage failed (preflight / keygen / migrations / org-seed / smoke)">
    The installer already printed the failing Job's last 30 log lines. Dig deeper:

    ```bash theme={"dark"}
    kubectl -n xpander logs job/<failed-job> --tail=100
    kubectl -n xpander describe job/<failed-job>
    kubectl -n xpander get pods,jobs,events --sort-by=.lastTimestamp
    ```

    The org-seed stage requires `--org-name` and `--admin-email` whenever the Supabase stack is on — the checklist catches this before install; a hand-driven `helm upgrade` that drops them fails at render time.
  </Accordion>

  <Accordion title="Pods in ImagePullBackOff after an upgrade">
    The chart pins image tags per release, so upgrading before mirroring the new bundle leaves pods unable to pull (typically with [`manifest unknown`](#troubleshooting)). Re-run the mirroring for the target release, then re-run the upgrade — it runs with `--atomic`, so the failed attempt already rolled back and the previous revision kept serving.
  </Accordion>

  <Accordion title="Edge-runtime crash-looping after upgrading with --reuse-values">
    `helm upgrade --reuse-values` replays the previous release's computed values, which can resurrect a stale edge-runtime placeholder image pin that newer chart versions removed. Upgrade with your real values files instead (what `install.sh upgrade` does), or clear the stale pin once:

    ```bash theme={"dark"}
    --set supabase.edgeRuntime.image.repository=null --set supabase.edgeRuntime.image.tag=null
    ```

    Better: pin the real mirrored edge-functions build (`--edge-functions-tag`).
  </Accordion>

  <Accordion title="Release already runs the other edition's chart">
    The two editions share nothing and cannot be upgraded into each other. Install under a different release or namespace (`--release` / `--namespace`), or remove the existing release first (`helm uninstall xpander -n xpander`).
  </Accordion>

  <Accordion title="Piped run exits with 'these required flags are missing'">
    Piping into `bash` leaves no stdin to prompt on, so the run is non-interactive: Air-Gapped requires only `--license-file`. Missing flags exit with code 2 and a completed rerun line. Download the script and run `bash install.sh --edition airgap` from a terminal for the interactive wizard.
  </Accordion>
</AccordionGroup>

***

## Next Steps

* [Advanced: review-first / manual install](/self-hosted/hybrid#advanced-review-first--manual-install) — read the installer before running it, plus `--manual`, `--values-out`, and `--dry-run`
* The chart's own README (inside the `.tgz`) covers advanced topics: bring-your-own ingress for the UI/API/Supabase gateway hosts, private CA and corporate proxy support, SMTP, internet posture and tool offering, builtin-skills seeding, and Bedrock region pinning
