Upgrading an Air-Gapped install
Upgrade procedure
Check the latest release. Two steps, in this order:Mirror the new bundle FIRST
ImagePullBackOff. Re-run image_sync.sh (curl -fsSLO https://charts.xpander.ai/image_sync.sh) for the target release as in Sealed / manual mirroring, and carry the new chart .tgz if needed.Run the upgrade
helm get values), loads the release pins over them, and stops if the manifest is invalid. It runs --atomic. Migrations stay forward-only. Pipelines should use the declarative re-install instead.Release v2026.10.22 (chart xpander-airgap 0.17.2)
Release v2026.10.22 (chart xpander-airgap 0.17.2)
- The LLM Gateway is always on and
llm-gw.enabledis ignored. If you had it off, mirrorllm-gw, give every provider your agents use a key, and open its network paths (LLM Gateway on a self-hosted install). - In-cluster Postgres moves to the pgvector build of PostgreSQL 16.15. Mirror
pgvector/pgvector, and snapshot the Postgres volume or setpostgres.upgradeGuard.backup.pvcName. Postgres restarts once. A pinnedpostgres.imagemust stay on a pgvector build of the same major version. - Per-agent MCP URLs are gone. Point clients at the install’s MCP address,
global.mcp.publicUrl, or athttps://api.<your domain>/mcp/. Both list and run your agents and serve each person the skills the Skills registry gives them (MCP gateway). Omni over MCP needsglobal.mcp.publicUrl. BRAIN_INGEST_API_KEYis added to the generated secret automatically.
<namespace>, <your-registry> and <xpander-token>, Helm 3.14+):Release v2026.10.15 (chart xpander-airgap 0.16.2)
Release v2026.10.15 (chart xpander-airgap 0.16.2)
- Mirror with the current
image_sync.sh(addsllm-gwandtool-repo-seed, drops--icon-pack-tag). - The LLM Gateway is on by default. Its generated keys are kept on every upgrade. Never rotate the content key. Every provider your agents use needs a key, or calls are refused naming the missing key. Check
kubectl -n <namespace> get pods -l app=llm-gw: twollm-gwpods (one on a single-nodeinstall.shcluster),1/1Running. See LLM Gateway on a self-hosted install. - Key names and endpoints: env-var key names matching the install’s own credentials read as no key, and metadata-address endpoints are refused (details).
- Helm 3.14+ for
--reset-then-reuse-values. Migrations run automatically.
Slack handle management compatibility
Slack handle management compatibility
20260910231402 and 20260911192635, and the chart’s post-migration check requires get_slack_handle_authorization, save_slack_handle_authorization and complete_slack_handle_authorization. Never upgrade the chart alone before those images are in your mirror.Add usergroups:write to your Slack app’s User Token Scopes, then complete administrator consent in Platform > Agents > Channels, on the Slack row. The token is encrypted in the local Supabase Vault. Slack needs explicitly allowed connectivity, and this upgrade does not turn it on.Declarative re-install for CI
A pipeline should inherit nothing from the release it upgrades. Runbash install.sh --edition airgap ... on every deploy with all values files, the release manifest and mirror pins, the license flags and --org, as on day one. Against a deployed release the installer adds --atomic itself. (install.sh upgrade once let a stored digest ride along and ran a stale image on staging for four days.)
-
--helm-timeout 20m(default30m). The--atomicrollback waits the same time again, so budget twice the value plus hooks within your job’s credential lifetime. A one-hour OIDC session cannot host a 30-minute timeout. -
Heal first. The installer refuses a release in
pending-install,pending-upgradeorfailed. Checkhelm history xpander -n xpander, thenhelm rollback xpander <last deployed revision> -n xpander, orhelm uninstall xpander -n xpanderwhen none deployed (xpander-generated-keyssurvives). Delete Failed pods and failed hook Jobs first. -
Pin the environment id. After the first install, pass
--env <existing default environment id>: the default location’s runtime environment cannot move to a new id. Read it first: -
Wait for Helm’s verdict. The stage watcher exits on the first failed hook pod, even an evicted one whose Job then succeeds. If
install.shexits non-zero whilehelm upgrade --install xpanderstill runs, do not kill Helm (it strands the release inpending-upgrade). Wait and read the release status. -
global.upgradeStrategy: replaceon clusters that provision nodes on demand, so a surge upgrade does not trigger provisioning waves that outlive the timeout.
The executor fleet and --atomic
The executor fleet and --atomic
SIGTERM an executor drains for up to global.fleet.executor.drainSeconds (3600 s). The installer’s timeout covers it. On a hand-driven helm upgrade, set --timeout to at least the drain window, or upgrade when quiet. A turn still running at the deadline stops with “The executor is restarting; send the message again to continue where it left off.” and resumes when the user sends again. An idle executor (zero replicas, xpander.ai/fleet-idle-reason) is skipped and never stalls an upgrade.What survives an upgrade or rollback
What survives an upgrade or rollback
- Generated keys survive every upgrade and rollback (lookup-preserve, and the Secret is Job-created, never Helm-rendered).
helm rollback xpander <N>restores any prior revision, except the database schema: migrations are forward-only, so a schema rollback means restore-from-backup.- The one-day token lifetime (
supabase.db.jwtExpiry) is re-applied by the migrations hook on every upgrade.
Releases
The release manifest supplies xpander image versions and digests to Helm. Helm still deploys. By default, the installer and mirror script use the latest release. A manifest names every xpander image: the services (includingbilling, openapi-spec-generator and llm-gw), the agent runtime images, and the artifacts (UI, migrations, edge functions, skills, catalog specs and tool-repo-seed). Logos ship in the UI image.
- Latest release: omit
--release-manifeston a connected machine. - Repeat a release: pass the same
--release-manifest FILEto both scripts, so a new release between mirroring and installing changes nothing. - Invalid release: stops the run. It never falls back to
latestor guesses tags. - Consistency: tag flags must match the manifest, and its pins override conflicting image values. Registry, infrastructure and secrets stay yours.
agent-harness, openapi-spec-generator, the third-party data-tier and Supabase images, and charts/xpander-airgap. image_sync.sh creates missing ECR repositories itself.
Per-artifact tag flags, credentials and pre-release builds
Per-artifact tag flags, credentials and pre-release builds
--manifest-out <file> writes the copied tags as a values file (pass it last with --values). --release-candidate selects a candidate. Credentials: image_sync.sh --source-user xpanderaihub --source-token <token> (and --dest-user / --dest-token), installer --xpander-registry-token <token>. --icon-pack-tag no longer exists, and image_sync.sh refuses unknown flags.Tracking pre-release builds (staging only): set XPANDER_IMAGE_STREAM=dev for the installer and pass --stream dev to image_sync.sh. Otherwise an air-gapped install refuses development overrides.Existing Helm and GitOps pipelines
Refresh the installer and mirror scripts together, and keep chart and image values in step. Per flow:Release handling per deployment flow
Release handling per deployment flow
Automation and CI
- Every prompt has a flag.
--non-interactivereads nothing from the terminal, exits 2 with the usage and a rerun line when a value is missing, and implies--yesand--on-probe-failure abort. - Stable progress lines such as
[3/8] license: ok.--quietprints phases and the outcome only. Color only on a terminal. statusis a read-only health report (edition, chart, release, hooks, readiness, URLs).--jsonemits one object.
Exit codes
Exit codes
--mirror now --xpander-registry-token <token>. Without a --mirror answer, an empty mirror exits 2.
Poll until healthy:
Reference values file
A complete production values file. Hosts, buckets and endpoints are placeholders from your infrastructure outputs. Pass passwords with--set, and the mirror manifest last for image pins.
Reference values file (YAML)
Reference values file (YAML)
orgSeed, generated passwords, image pins): extra values files merge last and override them.
Pipelines and GitOps
Jenkins (or any CI): watch the releases feed, mirror, upgrade
Jenkins (or any CI): watch the releases feed, mirror, upgrade
status. The agent needs cluster access, Helm 3.13+, skopeo and Python 3.1 (upgrade failed, already rolled back by --atomic), 2 (usage) and 4 (preflight). status fails on 1 (degraded, with reasons) and 3 (not installed). status --json emits one object if you prefer to parse.ArgoCD: the chart as Helm OCI from your registry
ArgoCD: the chart as Helm OCI from your registry
--release-manifest ./release-manifest.json --values-out xpander/values.yaml. It writes xpander/values.yaml and xpander/values.yaml.release.json (Helm values, not the manifest). Keep the manifest for your record and pass both files, release values last:- Values: regenerate per release, review with the chart version, and add no image overrides after the release values. Direct Helm does not run the installer’s checks. Strip secrets (LLM keys, DSNs) into your secret manager before committing.
- Bootstrap secrets: create
xpander-license(and any pull secret) once out-of-band (The License), or with SealedSecrets / External Secrets Operator. - Hooks: Argo CD maps the chart’s hook pipeline onto sync phases and waves. The chart is not certified under Argo CD: verify the hook ordering on your Argo version.
install.shis the certified path.
Next steps
Operate and troubleshoot
Mirror the images
image_sync.sh on a connected workstation or the sealed archive flow.
