Skip to main content
Mirror the new release into your registry first, then upgrade. Publishing a release does not upgrade clusters. Your Helm, GitOps or upgrade schedule decides when.

Upgrading an Air-Gapped install

Upgrade procedure

Check the latest release. Two steps, in this order:
1

Mirror the new bundle FIRST

Pinned digests mean upgrading before mirroring leaves pods in ImagePullBackOff. Re-run image_sync.sh (curl -fsSLO https://charts.xpander.ai/image_sync.sh) for the target release as in Sealed / manual mirroring, and carry the new chart .tgz if needed.
2

Run the upgrade

It reuses the installed values (helm get values), loads the release pins over them, and stops if the manifest is invalid. It runs --atomic. Migrations stay forward-only. Pipelines should use the declarative re-install instead.
Before a release with breaking changes, read its notes:
Carries the changes of Hybrid chart 0.22.5 (Hybrid: Upgrading to the current chart). Before you upgrade:
  • The LLM Gateway is always on and llm-gw.enabled is ignored. If you had it off, mirror llm-gw, give every provider your agents use a key, and open its network paths (LLM Gateway on a self-hosted install).
  • In-cluster Postgres moves to the pgvector build of PostgreSQL 16.15. Mirror pgvector/pgvector, and snapshot the Postgres volume or set postgres.upgradeGuard.backup.pvcName. Postgres restarts once. A pinned postgres.image must stay on a pgvector build of the same major version.
  • Per-agent MCP URLs are gone. Point clients at the install’s MCP address, global.mcp.publicUrl, or at https://api.<your domain>/mcp/. Both list and run your agents and serve each person the skills the Skills registry gives them (MCP gateway). Omni over MCP needs global.mcp.publicUrl.
  • BRAIN_INGEST_API_KEY is added to the generated secret automatically.
Mirror and upgrade (replace <namespace>, <your-registry> and <xpander-token>, Helm 3.14+):
Then run the verification guide.
Carries the changes of Hybrid chart 0.22.0 (Hybrid: Upgrading to the current chart):
  • Mirror with the current image_sync.sh (adds llm-gw and tool-repo-seed, drops --icon-pack-tag).
  • The LLM Gateway is on by default. Its generated keys are kept on every upgrade. Never rotate the content key. Every provider your agents use needs a key, or calls are refused naming the missing key. Check kubectl -n <namespace> get pods -l app=llm-gw: two llm-gw pods (one on a single-node install.sh cluster), 1/1 Running. See LLM Gateway on a self-hosted install.
  • Key names and endpoints: env-var key names matching the install’s own credentials read as no key, and metadata-address endpoints are refused (details).
  • Helm 3.14+ for --reset-then-reuse-values. Migrations run automatically.
For releases with administrator-authorized Slack agent handles, mirror the matching Supabase migrations and bots images before upgrading the chart. The migrations image must include 20260910231402 and 20260911192635, and the chart’s post-migration check requires get_slack_handle_authorization, save_slack_handle_authorization and complete_slack_handle_authorization. Never upgrade the chart alone before those images are in your mirror.Add usergroups:write to your Slack app’s User Token Scopes, then complete administrator consent in Platform > Agents > Channels, on the Slack row. The token is encrypted in the local Supabase Vault. Slack needs explicitly allowed connectivity, and this upgrade does not turn it on.

Declarative re-install for CI

A pipeline should inherit nothing from the release it upgrades. Run bash install.sh --edition airgap ... on every deploy with all values files, the release manifest and mirror pins, the license flags and --org, as on day one. Against a deployed release the installer adds --atomic itself. (install.sh upgrade once let a stored digest ride along and ran a stale image on staging for four days.)
  • --helm-timeout 20m (default 30m). The --atomic rollback waits the same time again, so budget twice the value plus hooks within your job’s credential lifetime. A one-hour OIDC session cannot host a 30-minute timeout.
  • Heal first. The installer refuses a release in pending-install, pending-upgrade or failed. Check helm history xpander -n xpander, then helm rollback xpander <last deployed revision> -n xpander, or helm uninstall xpander -n xpander when none deployed (xpander-generated-keys survives). Delete Failed pods and failed hook Jobs first.
  • Pin the environment id. After the first install, pass --env <existing default environment id>: the default location’s runtime environment cannot move to a new id. Read it first:
  • Wait for Helm’s verdict. The stage watcher exits on the first failed hook pod, even an evicted one whose Job then succeeds. If install.sh exits non-zero while helm upgrade --install xpander still runs, do not kill Helm (it strands the release in pending-upgrade). Wait and read the release status.
  • global.upgradeStrategy: replace on clusters that provision nodes on demand, so a surge upgrade does not trigger provisioning waves that outlive the timeout.
A running harness turn holds the roll for up to the fleet drain window.
A running harness turn holds the roll: on SIGTERM an executor drains for up to global.fleet.executor.drainSeconds (3600 s). The installer’s timeout covers it. On a hand-driven helm upgrade, set --timeout to at least the drain window, or upgrade when quiet. A turn still running at the deadline stops with “The executor is restarting; send the message again to continue where it left off.” and resumes when the user sends again. An idle executor (zero replicas, xpander.ai/fleet-idle-reason) is skipped and never stalls an upgrade.
What survives what:
  • Generated keys survive every upgrade and rollback (lookup-preserve, and the Secret is Job-created, never Helm-rendered).
  • helm rollback xpander <N> restores any prior revision, except the database schema: migrations are forward-only, so a schema rollback means restore-from-backup.
  • The one-day token lifetime (supabase.db.jwtExpiry) is re-applied by the migrations hook on every upgrade.
Avoid --reuse-values on hand-driven air-gap upgrades. It replays the old computed values, keeping changed defaults (from v2026.10.15 the LLM Gateway would stay off) and resurrecting removed settings, such as a stale edge-runtime image pin that crash-loops and fails every --atomic upgrade. Re-supply your values files, or clear it once with --set supabase.edgeRuntime.image.repository=null --set supabase.edgeRuntime.image.tag=null.

Releases

The release manifest supplies xpander image versions and digests to Helm. Helm still deploys. By default, the installer and mirror script use the latest release. A manifest names every xpander image: the services (including billing, openapi-spec-generator and llm-gw), the agent runtime images, and the artifacts (UI, migrations, edge functions, skills, catalog specs and tool-repo-seed). Logos ship in the UI image.
  • Latest release: omit --release-manifest on a connected machine.
  • Repeat a release: pass the same --release-manifest FILE to both scripts, so a new release between mirroring and installing changes nothing.
  • Invalid release: stops the run. It never falls back to latest or guesses tags.
  • Consistency: tag flags must match the manifest, and its pins override conflicting image values. Registry, infrastructure and secrets stay yours.
Third-party image versions come from the chart, so keep the matching chart package with each release. Keep mirror tags longer than the releases you may re-install. A “keep the last twenty images” rule spans only days, and a re-install then fails on an expired image. Keep hundreds of tags, or exempt pinned ones. If repositories must exist before a push, create one per release image, including agent-harness, openapi-spec-generator, the third-party data-tier and Supabase images, and charts/xpander-airgap. image_sync.sh creates missing ECR repositories itself.
Other flags: --manifest-out <file> writes the copied tags as a values file (pass it last with --values). --release-candidate selects a candidate. Credentials: image_sync.sh --source-user xpanderaihub --source-token <token> (and --dest-user / --dest-token), installer --xpander-registry-token <token>. --icon-pack-tag no longer exists, and image_sync.sh refuses unknown flags.Tracking pre-release builds (staging only): set XPANDER_IMAGE_STREAM=dev for the installer and pass --stream dev to image_sync.sh. Otherwise an air-gapped install refuses development overrides.
Installer 0.17.1 cannot parse the JSON release manifest, so --release-manifest alone fails. Until the fix ships, mirror with --manifest-out mirror-pins.yaml and pass --values mirror-pins.yaml as the last values file, alongside --release-manifest.

Existing Helm and GitOps pipelines

Refresh the installer and mirror scripts together, and keep chart and image values in step. Per flow:
Example pipeline: CI/CD & GitOps.

Automation and CI

  • Every prompt has a flag. --non-interactive reads nothing from the terminal, exits 2 with the usage and a rerun line when a value is missing, and implies --yes and --on-probe-failure abort.
  • Stable progress lines such as [3/8] license: ok. --quiet prints phases and the outcome only. Color only on a terminal.
  • status is a read-only health report (edition, chart, release, hooks, readiness, URLs). --json emits one object.
Air-gap, end to end, no terminal (release already mirrored):
If the mirror may be empty, add --mirror now --xpander-registry-token <token>. Without a --mirror answer, an empty mirror exits 2. Poll until healthy:

Reference values file

A complete production values file. Hosts, buckets and endpoints are placeholders from your infrastructure outputs. Pass passwords with --set, and the mirror manifest last for image pins.
Never repeat the installer’s own keys (orgSeed, generated passwords, image pins): extra values files merge last and override them.

Pipelines and GitOps

Fetch the release manifest once, mirror it with CI-stored credentials, re-run the install with that file, your values and the license flags (declarative re-install), then gate on status. The agent needs cluster access, Helm 3.13+, skopeo and Python 3.
The install step fails on 1 (upgrade failed, already rolled back by --atomic), 2 (usage) and 4 (preflight). status fails on 1 (degraded, with reasons) and 3 (not installed). status --json emits one object if you prefer to parse.
Mirror the chart with the images and keep configuration and generated release values in Git. A newer chart alone does not fetch the release manifest. Generate values with your install arguments plus --release-manifest ./release-manifest.json --values-out xpander/values.yaml. It writes xpander/values.yaml and xpander/values.yaml.release.json (Helm values, not the manifest). Keep the manifest for your record and pass both files, release values last:
For Argo CD:
  • Values: regenerate per release, review with the chart version, and add no image overrides after the release values. Direct Helm does not run the installer’s checks. Strip secrets (LLM keys, DSNs) into your secret manager before committing.
  • Bootstrap secrets: create xpander-license (and any pull secret) once out-of-band (The License), or with SealedSecrets / External Secrets Operator.
  • Hooks: Argo CD maps the chart’s hook pipeline onto sync phases and waves. The chart is not certified under Argo CD: verify the hook ordering on your Argo version. install.sh is the certified path.

Next steps

Operate and troubleshoot

Backups, monitoring and fixes by symptom.

Mirror the images

image_sync.sh on a connected workstation or the sealed archive flow.