Air-Gapped Edition

​

Overview

1. The license file is the identity. It carries your organization id — there are no console IDs and no API key. The installer generates the environment id, and the first install’s seed pipeline creates everything else in-cluster, including your organization. Single-tenant: the license binds one organization.
2. Images come from your own registry mirror. xpander delivers the artifact set through the gated xpanderaihub Docker Hub org with a per-customer read token; the installer mirrors it into a registry your cluster can reach — or you run the mirroring yourself on a sealed network.
3. Image tags are pinned per release. The chart refuses to render against unpinned images, and upgrades require mirroring the new bundle first.

​

What Talks to What

​

Prerequisites

​

Sizing Tiers

​

Scaling with Usage

• Static capacity — size the node group for your peak concurrent workspaces and leave it fixed. Simplest, fully offline, and the recommended posture for sealed networks.
• Cluster Autoscaler — adds nodes automatically when pods can’t schedule. It works in a no-internet VPC, but it talks to your cloud’s scaling API to do its job, so that API must be reachable from inside the network (on AWS: the autoscaling and ec2 VPC endpoints).

​

Install in Three Steps

curl -fsSL https://charts.xpander.ai/install.sh | bash -s -- \
  --edition airgap --license-file ./license.json

bash install.sh --edition airgap \
  --license-file ./license.json \
  --registry <your-registry>/xpander \
  --chart-path xpander-airgap-<X.Y.Z>.tgz \
  --domain xpander.acme.internal \
  --admin-email <you@company.example> --org-name "<Your Company>" --yes

​

What Happens When You Run the Installer

✓ pinned 20 images at their own latest (~/.xpander/logs/mirror-manifest-<timestamp>.yaml)

preflight → keygen → data-checks → workloads → migrations → org-seed → catalog-seed → smoke

kubectl -n xpander get secret xpander-generated-keys \
  -o jsonpath='{.data.XPANDER_ADMIN_PASSWORD}' | base64 -d

​

Image Tags: Resolved from Your Mirror

• Explicit flags always win: --image-tag pins deliberately — the installer honours it and does not go looking for anything newer; same for --migrations-tag / --edge-functions-tag, and a digest-pinned --values file stays in charge when given.
• An unresolvable mirror is treated as an empty one: the wizard first offers to fill it, and only then falls back to asking for the tags from your bundle manifest; a non-interactive run stops with a FIX: line instead.
• The resolution is reproducible: the pinned manifest is a file you keep, and the RERUN: line replays the same install.

​

When the Mirror Is Empty, the Wizard Offers to Fill It

• Mirror them now — it asks for the xpander pull token from your delivery email (username defaults to xpanderaihub; input stays hidden) and, optionally, credentials for your registry (Enter reuses what docker login already holds). It resolves the current release from xpander’s registry, then copies every image the install pulls — platform, UI, data tier, Supabase stack, artifact images — with per-image progress ([12/47] mirroring agent-controller:0.3.2 ... ok). ECR destinations get their repositories created automatically (idempotently, per repo). The copied versions are pinned into the same per-image manifest the install uses, tags re-resolve from your now-populated mirror, and the install continues. Re-running after a partial failure is safe — copies that already landed are skipped fast.
• Print the sealed-network runbook — for machines with no outbound internet; see Sealed / manual mirroring.
• Stop here — nothing has changed.

bash install.sh upgrade --release xpander --namespace xpander \
  --set supabase.enabled=true --set migrations.image.tag=<tag>

​

Sealed / Manual Mirroring

curl -fsSLO https://charts.xpander.ai/image_sync.sh
docker login -u xpanderaihub          # the pull token from your delivery email
bash image_sync.sh --source docker.io/xpanderaihub --dest <your-registry>/xpander

# on a connected workstation:
bash image_sync.sh --source docker.io/xpanderaihub --archive-dir ./xpander-images
# carry ./xpander-images across the gap, then, on the inside:
bash image_sync.sh --from-archive ./xpander-images --dest <your-registry>/xpander

helm registry login registry-1.docker.io --username <user> --password-stdin
helm pull oci://registry-1.docker.io/xpanderaihub/xpander-airgap --version <X.Y.Z>
# copy xpander-airgap-<X.Y.Z>.tgz onto the sealed-network host, then --chart-path it

bash image_sync.sh --source docker.io/xpanderaihub \
  --dest nexus.corp.example:5000/xpander \
  --dest-user <user> --dest-token <token>

--registry <nexus-host>/<proxy-repo>/xpanderaihub

​

Data Tier: In-Cluster or External

bash install.sh --edition airgap ... \
  --pg-mode external --pg-dsn 'postgresql://xpander:password@db.internal:5432/xpander' \
  --redis-mode external --redis-endpoint my-redis.internal \
  --redis-tls --redis-cluster --redis-username <acl-user> --redis-password <pass>

​

Local LLM Inference

​

The License

kubectl -n xpander create secret generic xpander-license \
  --from-file=license.json=<path/to/license.json> \
  --dry-run=client -o yaml | kubectl -n xpander apply -f -

• Expired license: the installer warns at the license phase and the chart’s preflight refuses the install. A running install whose license expires needs a renewed license file — the xpander.ai team issues one and delivers it over email.
• Renewing / replacing: recreate the xpander-license Secret with the new file (the command above), then run helm upgrade (or install.sh upgrade) — license consumers carry a checksum annotation over the Secret, so the upgrade rolls them onto the new file.
• Single-tenant: the license binds one organization; the org seed creates exactly that org.

​

Domains, Ingress & DNS

• --ui-host <host> replaces the derived chat.<domain> only; any other host is overridable per service via global.hosts.<name> in values.
• --ingress-scheme https|http (default https) sets how the browser reaches those hosts — it feeds every derived URL. Use http only for rigs without certificates.
• No ingress controller is assumed. Set the class and annotations per component (<name>.ingress.className / annotations) or once for all of them via global.ingress.className / global.ingress.annotations.

global:
  ingress:
    className: traefik
    tls:
      - hosts: ["*.xpander.acme.internal"]
        secretName: xpander-wildcard-tls

​

Making the Hosts Resolve

*.xpander.acme.internal.  CNAME  <your-ingress-controller's address>

sudo kubectl -n <ingress-namespace> port-forward svc/<controller-service> 443:443
# installed with --ingress-scheme http, forward port 80 instead:
sudo kubectl -n <ingress-namespace> port-forward svc/<controller-service> 80:80

# /etc/hosts
127.0.0.1 chat.xpander.acme.internal api.xpander.acme.internal agent-controller.xpander.acme.internal supabase.xpander.acme.internal webhook.xpander.acme.internal client-auth.xpander.acme.internal actions.xpander.acme.internal

​

Signing In

1. Bootstrap password (default) — open the UI (https://chat.<domain>) at /login-password with your --admin-email and the password from the summary; read it again any time:
   kubectl -n xpander get secret xpander-generated-keys \
     -o jsonpath='{.data.XPANDER_ADMIN_PASSWORD}' | base64 -d
   
2. Email OTP without SMTP — the login page’s email flow works with no mailer: the code is generated and stored, just not sent. Print it with the chart’s scripts/get_login_code.sh <email> <namespace>. With SMTP configured (global.smtp.* in values), the same codes arrive by email.
3. OIDC / OAuth / SAML — configure your provider through the chart’s Supabase auth settings (see the chart README, section “Signing in”).

kubectl -n xpander port-forward svc/ui 8080:80   # then http://localhost:8080

​

Upgrading an Air-Gapped Install

bash install.sh upgrade --chart-path xpander-airgap-<X.Y.Z>.tgz

• Generated keys survive every upgrade — the keygen hook is lookup-preserve, the xpander-generated-keys Secret is Job-created (never Helm-rendered), and rollback never rewinds it.
• helm rollback xpander <N> restores any prior revision — everything the chart owns is release-versioned.
• Except the database schema. Migrations run as a versioned Job per revision (kept as the audit trail) and are forward-only: a schema rollback means restore-from-backup, never an automatic down-migration.

​

Automation & CI

• Every prompt has a flag. With --non-interactive nothing is read from the terminal, even when one exists: missing required values exit 2 with the usage and a completed rerun line rather than blocking. It implies --yes and --on-probe-failure abort.
• Stable progress output. Progress is one line per transition in a grep-friendly shape — [3/8] license: ok — and --quiet reduces output to phase lines and the outcome only. Color is emitted only when stdout is a terminal.
• status is the poll target. A read-only health report: edition, chart, release status, hook jobs, workload readiness, and URLs. --json emits one JSON object.

bash install.sh --non-interactive --quiet \
  --edition airgap \
  --license-file ./license.json --registry registry.example:5000/xpander \
  --org-name "Example Inc" --admin-email ops@example.com \
  --domain xpander.example.internal --storage-class gp3 \
  --on-probe-failure abort

until bash install.sh status --quiet; do sleep 30; done

​

CI/CD & GitOps

pipeline {
  agent { label 'xpander-admin' }        // has kubectl access to the cluster
  triggers { cron('H 6 * * 1') }         // or trigger from releases.atom
  environment {
    XPANDER_PULL = credentials('xpander-pull-token')  // username: xpanderaihub, password: the read token
  }
  stages {
    stage('Mirror the release') {
      steps {
        sh '''
          curl -fsSLO https://charts.xpander.ai/image_sync.sh
          bash image_sync.sh --source docker.io/xpanderaihub \
            --dest registry.corp.example:5000/xpander \
            --source-user "$XPANDER_PULL_USR" --source-token "$XPANDER_PULL_PSW"
        '''
      }
    }
    stage('Upgrade and verify') {
      steps {
        sh '''
          curl -fsSL https://charts.xpander.ai/install.sh -o install.sh
          bash install.sh upgrade --non-interactive --quiet
          bash install.sh status --quiet
        '''
      }
    }
  }
}

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: xpander
  namespace: argocd
spec:
  project: default
  destination:
    server: https://kubernetes.default.svc
    namespace: xpander
  sources:
    - repoURL: registry.corp.example:5000/xpander/charts   # OCI repo credential with "Enable OCI"
      chart: xpander-airgap
      targetRevision: 0.3.2                                # the chart version you mirrored
      helm:
        valueFiles:
          - $values/xpander/values.yaml           # generated once: install.sh --values-out
          - $values/xpander/mirror-manifest.yaml  # the pinned per-image manifest, committed
    - repoURL: https://git.corp.example/platform/xpander-config.git
      targetRevision: main
      ref: values
  syncPolicy:
    syncOptions:
      - CreateNamespace=true

​

Troubleshooting

​

Mirroring and Registries

writing blob: initiating layer upload to /v2/xpander/ai-gateway/blobs/uploads/ in
123456789012.dkr.ecr.eu-west-2.amazonaws.com: name unknown: The repository with
name 'xpander/ai-gateway' does not exist in the registry with id '123456789012'

curl -fsSLO https://charts.xpander.ai/image_sync.sh

aws ecr create-repository --repository-name xpander/ai-gateway --region <region>

bash image_sync.sh --source docker.io/xpanderaihub --dest <registry>/<path> \
  --dest-user <user> --dest-token <token>

✗ deployment-manager:0.0.417 not found in registry.example:5000/xpander

crane digest registry.example:5000/xpander/deployment-manager:0.0.417

​

Cluster and Network

failed to pull image "docker.io/...": dial tcp 44.208.254.194:443: i/o timeout

kubectl -n xpander get pods -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.containers[*].image}{"\n"}{end}' | grep docker.io

✗ no node with a supported architecture: cluster has [arm64], the images need [amd64]

could not find any suitable subnets for creating the ELB

aws ec2 create-tags --resources <subnet-ids> \
  --tags Key=kubernetes.io/role/internal-elb,Value=1

metadata:
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-internal: "true"

Error from server (NotFound): namespaces "xpander" not found

kubectl create namespace xpander

Missing or invalid environment variables

kubectl -n xpander get ingress

​

Install and Upgrade

helm history xpander -n xpander
helm rollback xpander <last-good-revision> -n xpander   # if a good revision exists
helm uninstall xpander -n xpander                        # first install that never converged

kubectl -n xpander create secret generic xpander-license \
  --from-file=license.json=<new-file> \
  --dry-run=client -o yaml | kubectl -n xpander apply -f -

kubectl -n xpander logs job/<failed-job> --tail=100
kubectl -n xpander describe job/<failed-job>
kubectl -n xpander get pods,jobs,events --sort-by=.lastTimestamp

--set supabase.edgeRuntime.image.repository=null --set supabase.edgeRuntime.image.tag=null

​

Next Steps

• Advanced: review-first / manual install — read the installer before running it, plus --manual, --values-out, and --dry-run
• The chart’s own README (inside the .tgz) covers advanced topics: bring-your-own ingress for the UI/API/Supabase gateway hosts, private CA and corporate proxy support, SMTP, internet posture and tool offering, builtin-skills seeding, and Bedrock region pinning