> ## Documentation Index
> Fetch the complete documentation index at: https://docs.xpander.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Operate and troubleshoot an air-gapped install

> Back up an air-gapped install, check its health, and fix install, upgrade, registry and network problems by symptom.

Back up the volumes below yourself, check health with `bash install.sh status`, and find fixes by symptom under [Troubleshooting](#troubleshooting).

## Backups

The chart creates no backup job. Back up these volumes yourself, for example with AWS Backup or EBS snapshots:

| Volume | What it holds | Size on a reference install |
| - | - | - |
| `data-supabase-db-0` | Supabase PostgreSQL: organizations, users, agents, schema | 50Gi |
| `data-mongodb-0` | MongoDB data | 20Gi |
| `supabase-storage-data` | Supabase storage files | 50Gi |
| `fleet-volume-fleet-executor-<n>` | Executor fleet workspaces | 500Gi |
| `ag-<agent-id>-data` | The workspace of each agent that uses one (harness agents keep their files on the executor fleet volume) | 5Gi |
| `data-postgres-0` | Platform PostgreSQL, when `postgres.mode: internal` (the default) | 50Gi |
| `data-redis-0` | Redis Stack, when `redis.mode: internal` (the default) | 8Gi |

External PostgreSQL or Redis use the managed service's backups. Migrations are forward-only, so a schema rollback needs a restore.

## Monitoring

No metrics stack or `ServiceMonitor` ships. The `xpander-smoke` Job logs `smoke passed: every required component serves` when all components answer, and `bash install.sh status` gives a read-only verdict. Scrape the services with your own stack for metrics.

## Troubleshooting

Every failure prints **`FIX:`** (the repair) and **`RERUN:`** (your non-interactive one-liner, secrets as placeholders). The installer is idempotent: fix, paste the rerun line, and it resumes. The checklist stops most problems before anything changes. Scripts read the exit code ([Automation & CI](https://docs.xpander.ai/self-hosted/airgap-upgrade.md#automation-and-ci)), and `bash install.sh status` gives a read-only verdict.

### Mirroring and registries

> [accordion] Mirror push fails: name unknown, 'The repository ... does not exist' (ECR)
>
> ```
> writing blob: initiating layer upload to /v2/xpander/ai-gateway/blobs/uploads/ in
> 123456789012.dkr.ecr.eu-west-2.amazonaws.com: name unknown: The repository with
> name 'xpander/ai-gateway' does not exist in the registry with id '123456789012'
> ```
>
> ECR refuses pushes to repositories that do not exist. The current `image_sync.sh` creates them (with AWS CLI credentials). Refresh your copy:
>
> ```bash theme={"dark"}
> curl -fsSLO https://charts.xpander.ai/image_sync.sh
> ```
>
> On an older copy, pre-create the repository and re-run (copied images are skipped):
>
> ```bash theme={"dark"}
> aws ecr create-repository --repository-name xpander/ai-gateway --region <region>
> ```

> [accordion] Mirror push fails: authentication required at the destination
>
> `authentication required` even though `docker login` works: `skopeo` cannot read Docker Desktop's credential store. The current `image_sync.sh` logs in to ECR itself (`aws ecr get-login-password | skopeo login --username AWS --password-stdin`). For other registries, pass credentials:
>
> ```bash theme={"dark"}
> bash image_sync.sh --source docker.io/xpanderaihub --dest <registry>/<path> \
>   --dest-user <user> --dest-token <token>
> ```

> [accordion] 401 from Docker Hub when resolving or mirroring
>
> `401 from auth.docker.io (incorrect username or password)`: the credential is the **organization username** `xpanderaihub` plus the token from your email, not a personal username. Re-run with `--xpander-registry-user xpanderaihub --xpander-registry-token <token>`. Still refused? Ask xpander to re-issue the token with repository-read and image-pull access.

> [accordion] Pods can't pull: manifest unknown at your pinned tag
>
> Pod events show `manifest unknown`, or the checklist reports:
>
> ```
> ✗ deployment-manager:0.0.417 not found in registry.example:5000/xpander
> ```
>
> The mirror lacks that repository at the pinned tag. Tags differ **per image**, so one stale repo is enough. Re-run the mirroring ([in-wizard](https://docs.xpander.ai/self-hosted/airgap-install.md#when-the-mirror-is-empty-the-wizard-offers-to-fill-it) or `image_sync.sh`), then check the exact coordinate:
>
> ```bash theme={"dark"}
> crane digest registry.example:5000/xpander/deployment-manager:0.0.417
> ```

### Cluster and network

> [accordion] Nodes time out pulling from docker.io (dial tcp ... i/o timeout)
>
> ```
> failed to pull image "docker.io/...": dial tcp 44.208.254.194:443: i/o timeout
> ```
>
> An image reference still points at `docker.io` instead of your mirror. Install with `--registry <your-mirror>/<path>`, and make sure the **nodes** reach the mirror (on an ECR-only sealed VPC: `ecr.api` + `ecr.dkr` endpoints, the S3 gateway endpoint, ECR read on the node role). Find the reference:
>
> ```bash theme={"dark"}
> kubectl -n xpander get pods -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.containers[*].image}{"\n"}{end}' | grep docker.io
> ```

> [accordion] Preflight: no node with a supported architecture (Graviton/ARM)
>
> ```
> ✗ no node with a supported architecture: cluster has [arm64], the images need [amd64]
> ```
>
> (On a mixed cluster: `exec format error`.) xpander images are amd64-only, so ARM/Graviton (`t4g`, `m7g`, `c7g`, ...) cannot run them. Add an amd64 node group. Only for genuinely arm64 images: `--set 'preflight.allowedArchitectures[0]=arm64'`.

> [accordion] LoadBalancer Service stuck \<pending>: could not find any suitable subnets
>
> The ingress Service's events show:
>
> ```
> could not find any suitable subnets for creating the ELB
> ```
>
> A sealed VPC has no public subnets and the private ones lack ELB role tags. Tag them and annotate the Service:
>
> ```bash theme={"dark"}
> aws ec2 create-tags --resources <subnet-ids> \
>   --tags Key=kubernetes.io/role/internal-elb,Value=1
> ```
>
> ```yaml theme={"dark"}
> metadata:
>   annotations:
>     service.beta.kubernetes.io/aws-load-balancer-internal: "true"
> ```
>
> Or skip the load balancer: use the VPN, or `kubectl -n xpander port-forward svc/ui 8080:80`.

> [accordion] External data-tier probe fails: namespaces 'xpander' not found
>
> ```
> Error from server (NotFound): namespaces "xpander" not found
> ```
>
> The probe ran before the namespace existed (fixed in current installers). Refresh (`curl -fsSL https://charts.xpander.ai/install.sh`) or create the namespace and re-run:
>
> ```bash theme={"dark"}
> kubectl create namespace xpander
> ```

> [accordion] Pods stuck in Pending
>
> Unbound PVCs (`kubectl -n xpander describe pvc`) mean no usable RWO StorageClass or too little capacity for the [footprint](https://docs.xpander.ai/self-hosted/airgap.md#prerequisites). Check `kubectl get storageclass`, pass `--storage-class <name>`, and look for `FailedScheduling` in `kubectl -n xpander get events --sort-by=.lastTimestamp`.

> [accordion] UI loads blank: Missing or invalid environment variables
>
> The browser console shows:
>
> ```
> Missing or invalid environment variables
> ```
>
> Charts before **0.6.0** did not derive the UI's origins from the domain. Upgrade with `bash install.sh upgrade`, or set `ui.env` (`VITE_SUPABASE_URL`, `VITE_CHAT_BACKEND_URL`, ...) yourself.

> [accordion] Only the UI host resolves. Other subdomains 404 or NXDOMAIN
>
> `chat.<domain>` opens but sign-in or API calls fail: only one of the [published hosts](https://docs.xpander.ai/self-hosted/airgap-access.md#domains-ingress-and-dns) has a record. Add a wildcard `*.<domain>`, or records for `chat`, `api`, `agent-controller`, `supabase`, `webhook`, `client-auth` and `actions` (or your `<name>.ingress.host` values). Check what is published:
>
> ```bash theme={"dark"}
> kubectl -n xpander get ingress
> ```

### Install and upgrade

> [accordion] Release stuck in pending-install / pending-upgrade / failed
>
> The installer refuses a stuck release, and `install.sh upgrade` needs `deployed`. Recover first:
>
> ```bash theme={"dark"}
> helm history xpander -n xpander
> helm rollback xpander <last-good-revision> -n xpander   # if a good revision exists
> helm uninstall xpander -n xpander                        # first install that never converged
> ```
>
> Then paste your `RERUN:` line. `pending-install` (an interrupted first install) is fixed with `helm uninstall`. `xpander-generated-keys` survives it, so the reinstall keeps the same keys.

> [accordion] 'images are unpinned' / chart refuses to render
>
> Load a complete [latest release](https://docs.xpander.ai/self-hosted/airgap-upgrade.md#releases), mirror its images, and retry with that manifest. For direct Helm, pass both generated values files, release values last. Never guess a shared image tag.

> [accordion] License expired or rejected
>
> The installer checks the two `xpander-license/v2` files (or the legacy v1 file) and the `--org` binding. Preflight repeats it with the Ed25519 signature, and deployment-manager repeats it at boot. `GET /v1/license/status` answers `valid`, `expiring`, `grace` or `expired`.
>
> **Expired:** the installer warns `License expired on <date> - preflight will refuse the install`. Replace only `activation.json` in the Secret with the renewed file and redeploy:
>
> ```bash theme={"dark"}
> kubectl -n xpander create secret generic xpander-license \
>   --from-file=install.json=./install.json \
>   --from-file=activation.json=./activation-new.json \
>   --dry-run=client -o yaml | kubectl -n xpander apply -f -
> ```
>
> Then run `install.sh upgrade` or your pipeline's re-install (the Secret checksum rolls the services). Keep the `--set global.license.installKey/activationKey/publicKey` flags if you pass your own `--values` ([The License](https://docs.xpander.ai/self-hosted/airgap-install.md#the-license)). A legacy v1 install swaps its `license.json` the same way.

> [accordion] An agent answers 500 Failed to build agent details
>
> Usually the agent holds a connection to an API connector that never finished (status `faulty`, no spec). The `agents` pod log shows a pydantic error on `ConnectedConnector.open_api_spec_id`. Recover from the in-cluster Supabase database:
>
> ```sql theme={"dark"}
> -- the connections of the faulty connector
> update public.connector_organization set deleted_at = now()
>   where connector_id = '<connector id>' and deleted_at is null;
> -- take the connection id out of the agent's skill list
> update public.ai_agents
>   set attached_tools = (select coalesce(jsonb_agg(t), '[]'::jsonb)
>     from jsonb_array_elements(attached_tools::jsonb) t
>     where t->>'id' <> '<connection id>')::json
>   where id = '<agent id>';
> ```
>
> Reload the agent in every open tab before saving it again: a stale tab writes the dead connection back.

> [accordion] A pipeline stage failed (preflight / keygen / migrations / org-seed / smoke)
>
> The installer printed the Job's last 30 lines. Dig deeper:
>
> ```bash theme={"dark"}
> kubectl -n xpander logs job/<failed-job> --tail=100
> kubectl -n xpander describe job/<failed-job>
> kubectl -n xpander get pods,jobs
> kubectl -n xpander get events --sort-by=.lastTimestamp
> ```
>
> org-seed needs `--org-name` and `--admin-email` whenever Supabase is on. A hand-driven `helm upgrade` without them fails at render time.

> [accordion] Services fail auth after an upgrade with a custom values file
>
> A username with `password: ""` beside it **overrides the generated password with an empty string**. Remove the empty `password:` key: an absent key is preserved, `""` wins.

> [accordion] Creating a skill from an OpenAPI spec stops at "Generating the enriched spec", but the skill appears anyway
>
> On releases before the single-provider generator, a missing OpenAI key made the dialog show *"The AI enrichment pipeline reported a failure"* although the skill finished seconds later. Check the skills list before retrying. Upgrading fixes it.

> [accordion] Pods in ImagePullBackOff after an upgrade
>
> You upgraded before mirroring (typically [`manifest unknown`](#mirroring-and-registries)). Mirror the target release and re-run the upgrade. `--atomic` already rolled back the failed attempt.

> [accordion] Edge-runtime crash-looping after upgrading with --reuse-values
>
> `--reuse-values` resurrected a stale edge-runtime image pin. Upgrade with your real values files (as `install.sh upgrade` does), or clear it once:
>
> ```bash theme={"dark"}
> --set supabase.edgeRuntime.image.repository=null --set supabase.edgeRuntime.image.tag=null
> ```
>
> Better: pin the mirrored edge-functions build (`--edge-functions-tag`).

> [accordion] Release already runs the other deployment's chart
>
> Hybrid and Air-Gapped cannot be upgraded into each other. Use another `--release` / `--namespace`, or `helm uninstall xpander -n xpander` first.

> [accordion] Piped run exits with 'these required flags are missing'
>
> Piped into `bash`, the run is non-interactive and needs `--org`, `--install-license-file` and `--activation-license-file` (or `--license-file` for v1). It exits with code 2 and a rerun line. For the wizard, download the script and run `bash install.sh --edition airgap`.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.