> ## Documentation Index > Fetch the complete documentation index at: https://docs.xpander.ai/llms.txt > Use this file to discover all available pages before exploring further. # Upgrade an air-gapped install > How xpander releases reach an air-gapped install: mirror the release first, then upgrade by hand, from CI or with GitOps. Mirror the new release into your registry first, then upgrade. **Publishing a release does not upgrade clusters.** Your Helm, GitOps or upgrade schedule decides when. ## Upgrading an Air-Gapped install ### Upgrade procedure Check the [latest release](https://github.com/xpander-ai/releases/releases/tag/latest). Two steps, **in this order**: **1. Mirror the new bundle FIRST** Pinned digests mean upgrading before mirroring leaves pods in `ImagePullBackOff`. Re-run `image_sync.sh` (`curl -fsSLO https://charts.xpander.ai/image_sync.sh`) for the target release as in [Sealed / manual mirroring](https://docs.xpander.ai/self-hosted/airgap-install.md#sealed-or-manual-mirroring), and carry the new chart `.tgz` if needed. **2. Run the upgrade** ```bash theme={"dark"} bash install.sh upgrade --chart-path xpander-airgap-.tgz \ --release-manifest ./release-manifest.json ``` It reuses the installed values (`helm get values`), loads the release pins over them, and stops if the manifest is invalid. It runs `--atomic`. Migrations stay forward-only. Pipelines should use the [declarative re-install](#declarative-re-install-for-ci) instead. Before a release with breaking changes, read its notes: > [accordion] Release v2026.10.22 (chart xpander-airgap 0.17.2) > > Carries the changes of Hybrid chart 0.22.5 ([Hybrid: Upgrading to the current chart](https://docs.xpander.ai/self-hosted/hybrid-upgrade.md#upgrading-to-the-current-chart)). Before you upgrade: > > * **The LLM Gateway is always on** and `llm-gw.enabled` is ignored. If you had it off, mirror `llm-gw`, give every provider your agents use a key, and open its network paths ([LLM Gateway on a self-hosted install](https://docs.xpander.ai/run/llm-gateway/self-hosted.md)). > * **In-cluster Postgres moves to the pgvector build** of PostgreSQL 16.15. Mirror `pgvector/pgvector`, and snapshot the Postgres volume or set `postgres.upgradeGuard.backup.pvcName`. Postgres restarts once. A pinned `postgres.image` must stay on a pgvector build of the same major version. > * **Per-agent MCP URLs are gone.** Point clients at the install's MCP address, `global.mcp.publicUrl`, or at `https://api./mcp/`. Both list and run your agents and serve each person the skills the Skills registry gives them ([MCP gateway](https://docs.xpander.ai/run/integrations/mcp-gateway.md)). Omni over MCP needs `global.mcp.publicUrl`. > * **`BRAIN_INGEST_API_KEY`** is added to the generated secret automatically. > > Mirror and upgrade (replace ``, `` and ``, Helm 3.14+): > > ```bash theme={"dark"} > curl -fsSL https://charts.xpander.ai/image_sync.sh -o image_sync.sh > curl -fsSL https://github.com/xpander-ai/releases/releases/download/v2026.10.22/release-manifest.json -o release-manifest.json > > bash image_sync.sh --source-user xpanderaihub --source-token \ > --dest --release-manifest release-manifest.json --manifest-out pins.yaml > > helm registry login registry-1.docker.io -u xpanderaihub -p > helm upgrade xpander oci://registry-1.docker.io/xpanderaihub/xpander-airgap --version 0.17.2 \ > -n --reset-then-reuse-values -f pins.yaml --wait --timeout 30m > ``` > > Then run the [verification guide](https://pages.xpander.ai/airgap-verification-guide). > [accordion] Release v2026.10.15 (chart xpander-airgap 0.16.2) > > Carries the changes of Hybrid chart 0.22.0 ([Hybrid: Upgrading to the current chart](https://docs.xpander.ai/self-hosted/hybrid-upgrade.md#upgrading-to-the-current-chart)): > > * **Mirror with the current `image_sync.sh`** (adds `llm-gw` and `tool-repo-seed`, drops `--icon-pack-tag`). > * **The LLM Gateway is on by default.** Its generated keys are kept on every upgrade. Never rotate the content key. Every provider your agents use needs a key, or calls are refused naming the missing key. Check `kubectl -n get pods -l app=llm-gw`: two `llm-gw` pods (one on a single-node `install.sh` cluster), `1/1` `Running`. See [LLM Gateway on a self-hosted install](https://docs.xpander.ai/run/llm-gateway/self-hosted.md). > * **Key names and endpoints:** env-var key names matching the install's own credentials read as no key, and metadata-address endpoints are refused ([details](https://docs.xpander.ai/self-hosted/hybrid-llm-keys.md#key-names-read-from-an-environment-variable)). > * **Helm 3.14+** for `--reset-then-reuse-values`. Migrations run automatically. > [accordion] Slack handle management compatibility > > For releases with administrator-authorized [Slack agent handles](https://docs.xpander.ai/run/integrations/slack.md#authorize-handle-management), mirror the matching Supabase migrations and bots images before upgrading the chart. The migrations image must include `20260910231402` and `20260911192635`, and the chart's post-migration check requires `get_slack_handle_authorization`, `save_slack_handle_authorization` and `complete_slack_handle_authorization`. Never upgrade the chart alone before those images are in your mirror. > > Add `usergroups:write` to your Slack app's **User Token Scopes**, then complete administrator consent in **Platform > Agents > Channels**, on the **Slack** row. The token is encrypted in the local Supabase Vault. Slack needs explicitly allowed connectivity, and this upgrade does not turn it on. ### Declarative re-install for CI A pipeline should inherit nothing from the release it upgrades. Run `bash install.sh --edition airgap ...` on every deploy with all values files, the release manifest and mirror pins, the license flags and `--org`, as on day one. Against a deployed release the installer adds `--atomic` itself. (`install.sh upgrade` once let a stored digest ride along and ran a stale image on staging for four days.) * **`--helm-timeout 20m`** (default `30m`). The `--atomic` rollback waits the same time again, so budget twice the value plus hooks within your job's credential lifetime. A one-hour OIDC session cannot host a 30-minute timeout. * **Heal first.** The installer refuses a release in `pending-install`, `pending-upgrade` or `failed`. Check `helm history xpander -n xpander`, then `helm rollback xpander -n xpander`, or `helm uninstall xpander -n xpander` when none deployed (`xpander-generated-keys` survives). Delete Failed pods and failed hook Jobs first. * **Pin the environment id.** After the first install, pass `--env `: the default location's runtime environment cannot move to a new id. Read it first: ```bash theme={"dark"} kubectl -n xpander exec supabase-db-0 -c postgres -- psql -U supabase_admin -d postgres -tAc \ "SELECT id FROM public.organization_environments WHERE is_default LIMIT 1" ``` * **Wait for Helm's verdict.** The stage watcher exits on the first failed hook pod, even an evicted one whose Job then succeeds. If `install.sh` exits non-zero while `helm upgrade --install xpander` still runs, do not kill Helm (it strands the release in `pending-upgrade`). Wait and read the release status. * **`global.upgradeStrategy: replace`** on clusters that provision nodes on demand, so a surge upgrade does not trigger provisioning waves that outlive the timeout. A running harness turn holds the roll for up to the fleet drain window. > [accordion] The executor fleet and --atomic > > A **running harness turn holds the roll**: on `SIGTERM` an executor drains for up to `global.fleet.executor.drainSeconds` (3600 s). The installer's timeout covers it. On a hand-driven `helm upgrade`, set `--timeout` to at least the drain window, or upgrade when quiet. A turn still running at the deadline stops with "The executor is restarting; send the message again to continue where it left off." and resumes when the user sends again. An **idle executor** (zero replicas, `xpander.ai/fleet-idle-reason`) is skipped and never stalls an upgrade. > [accordion] What survives an upgrade or rollback > > What survives what: > > * **Generated keys survive every upgrade and rollback** (lookup-preserve, and the Secret is Job-created, never Helm-rendered). > * **`helm rollback xpander ` restores any prior revision**, **except the database schema**: migrations are forward-only, so a schema rollback means restore-from-backup. > * **The one-day token lifetime** (`supabase.db.jwtExpiry`) is re-applied by the migrations hook on every upgrade. > [!WARNING] > > **Avoid `--reuse-values` on hand-driven air-gap upgrades.** It replays the old computed values, keeping changed defaults (from v2026.10.15 the LLM Gateway would stay off) and resurrecting removed settings, such as a stale edge-runtime image pin that crash-loops and fails every `--atomic` upgrade. Re-supply your values files, or clear it once with `--set supabase.edgeRuntime.image.repository=null --set supabase.edgeRuntime.image.tag=null`. ## Releases The release manifest supplies xpander image versions and digests to Helm. Helm still deploys. By default, the installer and mirror script use the [latest release](https://github.com/xpander-ai/releases/releases/tag/latest). A manifest names every xpander image: the services (including `billing`, `openapi-spec-generator` and `llm-gw`), the agent runtime images, and the artifacts (UI, migrations, edge functions, skills, catalog specs and `tool-repo-seed`). Logos ship in the UI image. * **Latest release:** omit `--release-manifest` on a connected machine. * **Repeat a release:** pass the same `--release-manifest FILE` to both scripts, so a new release between mirroring and installing changes nothing. * **Invalid release:** stops the run. It never falls back to `latest` or guesses tags. * **Consistency:** tag flags must match the manifest, and its pins override conflicting image values. Registry, infrastructure and secrets stay yours. Third-party image versions come from the chart, so keep the matching chart package with each release. **Keep mirror tags longer than the releases you may re-install.** A "keep the last twenty images" rule spans only days, and a re-install then fails on an expired image. Keep hundreds of tags, or exempt pinned ones. If repositories must exist before a push, create one per release image, including `agent-harness`, `openapi-spec-generator`, the third-party data-tier and Supabase images, and `charts/xpander-airgap`. `image_sync.sh` creates missing ECR repositories itself. > [accordion] Per-artifact tag flags, credentials and pre-release builds > > | Artifact | Mirror script (`image_sync.sh`) | Installer | > | - | - | - | > | UI | `--ui-tag` | not available | > | Skills | `--skills-tag` | `--skills-tag` | > | Catalog specs | `--catalog-specs-tag` | `--catalog-tag` | > | Skills repository seed (`tool-repo-seed`) | `--tool-repo-seed-tag` | read from the release manifest | > | Migrations | `--migrations-tag` | `--migrations-tag` | > | Edge functions | `--edge-functions-tag` | `--edge-functions-tag` | > > Other flags: `--manifest-out ` writes the copied tags as a values file (pass it last with `--values`). `--release-candidate` selects a candidate. Credentials: `image_sync.sh --source-user xpanderaihub --source-token ` (and `--dest-user` / `--dest-token`), installer `--xpander-registry-token `. `--icon-pack-tag` no longer exists, and `image_sync.sh` refuses unknown flags. > > **Tracking pre-release builds** (staging only): set `XPANDER_IMAGE_STREAM=dev` for the installer and pass `--stream dev` to `image_sync.sh`. Otherwise an air-gapped install refuses development overrides. > [!WARNING] > > Installer 0.17.1 cannot parse the JSON release manifest, so `--release-manifest` alone fails. Until the fix ships, mirror with `--manifest-out mirror-pins.yaml` and pass `--values mirror-pins.yaml` as the last values file, alongside `--release-manifest`. ### Existing Helm and GitOps pipelines Refresh the installer and mirror scripts together, and keep chart and image values in step. Per flow: > [accordion] Release handling per deployment flow > > | Deployment flow | Release handling | > | - | - | > | Installer invokes Helm | Refresh the installer and mirror scripts together. | > | Direct `helm upgrade`, Argo CD, or Flux | Supply the generated release values with your configuration, and update chart and image values together. Helm does not read the JSON manifest. | > | Fully offline installation | Transfer the image bundle, manifest, scripts, validator and chart package before upgrading. | Example pipeline: [CI/CD & GitOps](#pipelines-and-gitops). ## Automation and CI * **Every prompt has a flag.** `--non-interactive` reads nothing from the terminal, exits 2 with the usage and a rerun line when a value is missing, and implies `--yes` and `--on-probe-failure abort`. * **Stable progress lines** such as `[3/8] license: ok`. `--quiet` prints phases and the outcome only. Color only on a terminal. * **`status`** is a read-only health report (edition, chart, release, hooks, readiness, URLs). `--json` emits one object. > [accordion] Exit codes > > | Code | Meaning | > | - | - | > | `0` | Success | > | `1` | Install or upgrade failure (`status`: degraded) | > | `2` | Usage error, or a required flag missing with no terminal | > | `3` | `status`: no such release installed | > | `4` | Preflight failure (missing helm/kubectl, unreachable cluster, version too old) | > | `5` | External data-tier probe failed and `--on-probe-failure` was `abort` | **Air-gap, end to end, no terminal** (release already mirrored): ```bash theme={"dark"} bash install.sh --non-interactive --quiet \ --edition airgap --release-manifest ./release-manifest.json \ --org --install-license-file ./install.json --activation-license-file ./activation.json \ --registry registry.example:5000/xpander \ --org-name "Example Inc" --admin-email ops@example.com \ --domain xpander.example.internal --storage-class gp3 \ --on-probe-failure abort ``` If the mirror may be empty, add `--mirror now --xpander-registry-token `. Without a `--mirror` answer, an empty mirror exits 2. **Poll until healthy:** ```bash theme={"dark"} until bash install.sh status --quiet; do sleep 30; done ``` ### Reference values file A complete production values file. Hosts, buckets and endpoints are placeholders from your infrastructure outputs. Pass passwords with `--set`, and the mirror manifest last for image pins. > [accordion] Reference values file (YAML) > > ```yaml theme={"dark"} > global: > upgradeStrategy: replace # in-place recreation; surge upgrades cascade into node waves on autoscaled clusters > imageRegistry: / # every image is pulled from the mirror > license: > secretName: xpander-license > smtp: # one block feeds sign-in mail, invitations and xpander's notifications > host: email-smtp..amazonaws.com > port: 587 > starttls: true > user: > passwordSecret: { name: xpander-smtp, key: password } > adminEmail: admin@ > senderName: xpander > assetStore: { publicUrl: https://auth. } > shortLinks: { publicUrl: https://links. } > # mcp.publicUrl unset: Platform > Agents > Channels shows the MCP row as "Not offered on this install" until you set it > llm: > defaults: { provider: amazon_bedrock, model: global.anthropic.claude-sonnet-5, apiBase: "", credentialsKey: "" } > bedrock: { region: us-west-2 } # may differ from the cluster's region > gateway: { provider: amazon_bedrock, model: global.anthropic.claude-sonnet-4-6 } > harness: > defaultProviders: { claudeCode: amazon_bedrock } > defaultModels: { claudeCode: global.anthropic.claude-sonnet-5 } > fleet: > replicas: 1 > storage: { createClass: true, provisioner: ebs.csi.eks.amazonaws.com } > network: > internalFetchAllowlist: [".internal."] # intranet hosts agents may fetch for knowledge ingest > redis: { endpoint: , clusterMode: true, auth: { username: } } > redis: { mode: external } > postgres: { mode: external, external: { host: , port: 5432 } } > supabase: > ingress: { enabled: true, className: alb, host: auth. } > auth: { apiExternalUrl: https://auth., siteUrl: https://app. } > storage: > persistence: { size: 100Gi } > publicUrl: https://auth. > edgeRuntime: > publicUrl: https://auth. > functionsEnv: > APP_URL: https://app. > INBOUND_GW_URL: https://hooks. > AGENT_CONTAINER_INGRESS_DOMAIN: > DEPLOYMENT_MANAGER_PUBLIC_URL: https://dm.internal. > ui: > ingress: { host: app., className: alb } > authProviders: [sso] > env: > VITE_WEBHOOK_URL: https://hooks. # the app's baked default is the cloud host > VITE_EMAIL_DOMAIN: agents. > VITE_PLATFORM_URL: https://app. > VITE_SUPABASE_URL: https://auth. > VITE_AGENTS_API_URL: https://api. > VITE_CHAT_BACKEND_URL: https://chat./chat-backend/{agent_id} > VITE_ACTIONS_SERVICE_URL: https://actions. > VITE_CLIENT_AUTH_URL: https://client-auth. > VITE_LICENSE_MODE: "true" > VITE_OMNI_MCP_URL: "" > extraEnv: > - name: VITE_SUPABASE_ANON_KEY > valueFrom: { secretKeyRef: { name: xpander-generated-keys, key: SUPABASE_ANON_KEY } } > client-auth: { ingress: { host: client-auth. }, env: { APP_URL: https://app., CLIENT_AUTH_EXTERNAL_URL: https://client-auth. } } # the second turns on MCP OAuth > api: > ingress: { host: api. } > env: { API_URL: https://api., INBOUND_EMAIL_REPLY_DOMAIN: agents., SMTP_HOST: , SMTP_PORT: "587", SMTP_STARTTLS: "true", SMTP_USERNAME: } > extraEnv: > - { name: SMTP_PASSWORD, valueFrom: { secretKeyRef: { name: xpander-smtp, key: password } } } > - { name: INBOUND_EMAIL_TOKEN, valueFrom: { secretKeyRef: { name: xpander-inbound-mail, key: token } } } > - { name: MAILGUN_SIGNING_KEY, valueFrom: { secretKeyRef: { name: xpander-inbound-mail, key: signing-key } } } > agent-controller: { ingress: { host: chat. } } # API clients on this host add / to the base URL > webhook: { ingress: { host: hooks. } } > functions: { ingress: { host: links. } } > deployment-manager: { ingress: { host: dm.internal., className: alb-internal } } > actions: > ingress: { host: actions. } > env: { XPANDER_SANDBOX_DNS: } > catalog: { env: { OAS_GEN_BUCKET_NAME: , AWS_DEFAULT_REGION: } } > openapi-spec-generator: { env: { OAS_GEN_BUCKET_NAME: , AWS_DEFAULT_REGION: } } > ``` Never repeat the installer's own keys (`orgSeed`, generated passwords, image pins): extra values files merge last and override them. ### Pipelines and GitOps > [accordion] Jenkins (or any CI): watch the releases feed, mirror, upgrade > > Fetch the release manifest once, mirror it with CI-stored credentials, re-run the install with that file, your values and the license flags ([declarative re-install](#declarative-re-install-for-ci)), then gate on `status`. The agent needs cluster access, Helm 3.13+, skopeo and Python 3. > > ```groovy theme={"dark"} > pipeline { > agent { label 'xpander-admin' } // has kubectl access to the cluster > triggers { cron('H 6 * * 1') } // or trigger from releases.atom > environment { > XPANDER_PULL = credentials('xpander-pull-token') // username: xpanderaihub, password: the read token > XPANDER_ORG = credentials('xpander-org-id') // the organization id from the license handoff > } > stages { > stage('Mirror the release') { > steps { > sh ''' > curl -fsSLO https://charts.xpander.ai/image_sync.sh > curl -fsSL https://github.com/xpander-ai/releases/releases/download/latest/release-manifest.json -o release-manifest.json > bash image_sync.sh --source docker.io/xpanderaihub \ > --dest registry.corp.example:5000/xpander \ > --release-manifest ./release-manifest.json \ > --source-user "$XPANDER_PULL_USR" --source-token "$XPANDER_PULL_PSW" > ''' > } > } > stage('Upgrade and verify') { > steps { > sh ''' > curl -fsSL https://charts.xpander.ai/install.sh -o install.sh > bash install.sh --non-interactive --quiet --edition airgap \ > --org "$XPANDER_ORG" --install-license-file install.json --activation-license-file activation.json \ > --registry registry.corp.example:5000/xpander --mirror skip \ > --release-manifest ./release-manifest.json --values values.yaml \ > --helm-timeout 20m > bash install.sh status --quiet > ''' > } > } > } > } > ``` > > The install step fails on `1` (upgrade failed, already rolled back by `--atomic`), `2` (usage) and `4` (preflight). `status` fails on `1` (degraded, with reasons) and `3` (not installed). `status --json` emits one object if you prefer to parse. > [accordion] ArgoCD: the chart as Helm OCI from your registry > > [Mirror the chart with the images](https://docs.xpander.ai/self-hosted/airgap-install.md#sealed-or-manual-mirroring) and keep configuration and generated release values in Git. A newer chart alone does not fetch the release manifest. Generate values with your install arguments plus `--release-manifest ./release-manifest.json --values-out xpander/values.yaml`. It writes `xpander/values.yaml` and `xpander/values.yaml.release.json` (Helm values, not the manifest). Keep the manifest for your record and pass both files, release values last: > > ```bash theme={"dark"} > helm upgrade --install xpander ./xpander-airgap-.tgz \ > --namespace xpander --create-namespace \ > -f xpander/values.yaml -f xpander/values.yaml.release.json > ``` > > For Argo CD: > > ```yaml theme={"dark"} > apiVersion: argoproj.io/v1alpha1 > kind: Application > metadata: > name: xpander > namespace: argocd > spec: > project: default > destination: > server: https://kubernetes.default.svc > namespace: xpander > sources: > - repoURL: registry.corp.example:5000/xpander/charts # OCI repo credential with "Enable OCI" > chart: xpander-airgap > targetRevision: # keep it in step with the release values > helm: > valueFiles: > - $values/xpander/values.yaml # generated once: install.sh --values-out > - $values/xpander/values.yaml.release.json # generated release image pins, applied last > - repoURL: https://git.corp.example/platform/xpander-config.git > targetRevision: main > ref: values > syncPolicy: > syncOptions: > - CreateNamespace=true > ``` > > * **Values**: regenerate per release, review with the chart version, and add no image overrides after the release values. Direct Helm does not run the installer's checks. Strip secrets (LLM keys, DSNs) into your secret manager before committing. > * **Bootstrap secrets**: create `xpander-license` (and any pull secret) once out-of-band ([The License](https://docs.xpander.ai/self-hosted/airgap-install.md#the-license)), or with SealedSecrets / External Secrets Operator. > * **Hooks**: Argo CD maps the chart's hook pipeline onto sync phases and waves. The chart is not certified under Argo CD: **verify the hook ordering on your Argo version**. `install.sh` is the certified path. ## Next steps > [card] [Operate and troubleshoot](https://docs.xpander.ai/self-hosted/airgap-operate.md) > > Backups, monitoring and fixes by symptom. > [card] [Mirror the images](https://docs.xpander.ai/self-hosted/airgap-install.md#sealed-or-manual-mirroring) > > `image_sync.sh` on a connected workstation or the sealed archive flow. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.