Air-Gapped Edition

​

Overview

1. The license file is the identity. It carries your organization id: there are no console IDs and no API key. The installer generates the environment id, and the first install’s seed pipeline creates everything else in-cluster, including your organization. Single-tenant: the license binds one organization.
2. Images come from your own registry mirror. xpander delivers the artifact set through the gated xpanderaihub Docker Hub org with a per-customer read token; the installer mirrors it into a registry your cluster can reach, or you run the mirroring yourself on a sealed network.
3. Image tags are pinned per release. The chart refuses to render against unpinned images, and upgrades require mirroring the new bundle first.

​

What Talks to What

​

Prerequisites

​

Sizing Tiers

​

Scaling with Usage

• Static capacity: size the node group for your peak concurrent workspaces and leave it fixed. Simplest, fully offline, and the recommended posture for sealed networks.
• Cluster Autoscaler: adds nodes automatically when pods can’t schedule. It works in a no-internet VPC, but it talks to your cloud’s scaling API to do its job, so that API must be reachable from inside the network (on AWS: the autoscaling and ec2 VPC endpoints).

​

Install in Three Steps

curl -fsSL https://charts.xpander.ai/install.sh | bash -s -- \
  --edition airgap --license-file ./license.json

bash install.sh --edition airgap \
  --license-file ./license.json \
  --registry <your-registry>/xpander \
  --chart-path xpander-airgap-<X.Y.Z>.tgz \
  --domain xpander.acme.internal \
  --admin-email <you@company.example> --org-name "<Your Company>" --yes

​

What Happens When You Run the Installer

preflight → keygen → data-checks → workloads → migrations → org-seed → catalog-seed → smoke

kubectl -n xpander get secret xpander-generated-keys \
  -o jsonpath='{.data.XPANDER_ADMIN_PASSWORD}' | base64 -d

​

Releases

• Use the latest release: omit --release-manifest when running the installer or mirror tool on a connected machine.
• Repeat one release: retain its JSON manifest and pass the same --release-manifest FILE to both tools. This also prevents a new release between mirroring and installation from changing the selected images.
• Stop on an invalid release: a missing, incomplete, or invalid manifest stops the operation. It does not fall back to latest, guess individual tags, or silently omit missing artifact images.
• Keep the release consistent: explicit image tag flags must match the manifest. The installer applies the release image pins over conflicting image values. Customer configuration such as registry addresses, infrastructure settings, and secrets remains separate.

​

Existing Helm and GitOps Pipelines

​

When the Mirror Is Empty, the Wizard Offers to Fill It

​

Sealed / Manual Mirroring

curl -fsSLO https://charts.xpander.ai/image_sync.sh
curl -fsSLO https://charts.xpander.ai/install.sh
curl -fsSL https://github.com/xpander-ai/releases/releases/download/latest/release-manifest.json \
  -o release-manifest.json
bash image_sync.sh --source docker.io/xpanderaihub \
  --dest <your-registry>/xpander \
  --release-manifest ./release-manifest.json

# On a connected workstation:
bash image_sync.sh --source docker.io/xpanderaihub \
  --release-manifest ./release-manifest.json --archive-dir ./xpander-images

# After transferring the complete directory, on the inside:
bash image_sync.sh --from-archive ./xpander-images --dest <your-registry>/xpander

helm registry login registry-1.docker.io --username <user> --password-stdin
helm pull oci://registry-1.docker.io/xpanderaihub/xpander-airgap --version <X.Y.Z>
# copy xpander-airgap-<X.Y.Z>.tgz onto the sealed-network host, then --chart-path it

bash image_sync.sh --source docker.io/xpanderaihub \
  --dest nexus.corp.example:5000/xpander \
  --dest-user <user> --dest-token <token>

--registry <nexus-host>/<proxy-repo>/xpanderaihub

​

Data Tier: In-Cluster or External

bash install.sh --edition airgap ... \
  --pg-mode external --pg-dsn 'postgresql://xpander:password@db.internal:5432/xpander' \
  --redis-mode external --redis-endpoint my-redis.internal \
  --redis-tls --redis-cluster --redis-username <acl-user> --redis-password <pass>

​

Local LLM Inference

​

Harness Agents on Air-Gap

​

AI Connector Generation

​

The License

kubectl -n xpander create secret generic xpander-license \
  --from-file=license.json=<path/to/license.json> \
  --dry-run=client -o yaml | kubectl -n xpander apply -f -

• Expired license: the installer warns at the license phase and the chart’s preflight refuses the install. A running install whose license expires needs a renewed license file: the xpander.ai team issues one and delivers it over email.
• Renewing / replacing: recreate the xpander-license Secret with the new file (the command above), then run helm upgrade (or install.sh upgrade). License consumers carry a checksum annotation over the Secret, so the upgrade rolls them onto the new file.
• Single-tenant: the license binds one organization; the org seed creates exactly that org.

​

Domains, Ingress & DNS

• --ui-host <host> replaces the derived chat.<domain> only; any other host is overridable per service via global.hosts.<name> in values.
• --ingress-scheme https|http (default https) sets how the browser reaches those hosts: it feeds every derived URL. Use http only for rigs without certificates.
• No ingress controller is assumed. Set the class and annotations per component (<name>.ingress.className / annotations) or once for all of them via global.ingress.className / global.ingress.annotations.

global:
  ingress:
    className: traefik
    tls:
      - hosts: ["*.xpander.acme.internal"]
        secretName: xpander-wildcard-tls

​

Making the Hosts Resolve

*.xpander.acme.internal.  CNAME  <your-ingress-controller's address>

sudo kubectl -n <ingress-namespace> port-forward svc/<controller-service> 443:443
# installed with --ingress-scheme http, forward port 80 instead:
sudo kubectl -n <ingress-namespace> port-forward svc/<controller-service> 80:80

# /etc/hosts
127.0.0.1 chat.xpander.acme.internal api.xpander.acme.internal agent-controller.xpander.acme.internal supabase.xpander.acme.internal webhook.xpander.acme.internal client-auth.xpander.acme.internal actions.xpander.acme.internal

​

Signing In

1. Bootstrap password (default): open the UI (https://chat.<domain>) at /login-password with your --admin-email and the password from the summary; read it again any time:
   kubectl -n xpander get secret xpander-generated-keys \
     -o jsonpath='{.data.XPANDER_ADMIN_PASSWORD}' | base64 -d
   
2. Email OTP without SMTP: the login page’s email flow works with no mailer. The code is generated and stored, just not sent. Print it with the chart’s scripts/get_login_code.sh <email> <namespace>. With SMTP configured (global.smtp.* in values), the same codes arrive by email, and on installs with inbound agent mailboxes, agents’ email replies deliver through the same relay (INBOUND_EMAIL_REPLY_DOMAIN controls the From domain; the SMTP identity must be allowed to send as the agents’ mailbox subdomain).
3. SSO with your identity provider (OIDC): the SSO button is driven by the OIDC configuration an administrator enters in the app (Settings, OIDC), served by the client-auth component, not by GoTrue’s external providers. Setup, IdP client settings, the Keycloak preferred_username note and troubleshooting: SSO with your identity provider. Google and GitHub social buttons remain GoTrue providers (supabase.auth.extraEnv).

kubectl -n xpander port-forward svc/ui 8080:80   # then http://localhost:8080

​

Upgrading an Air-Gapped Install

bash install.sh upgrade --chart-path xpander-airgap-<X.Y.Z>.tgz \
  --release-manifest ./release-manifest.json

• Generated keys survive every upgrade: the keygen hook is lookup-preserve, the xpander-generated-keys Secret is Job-created (never Helm-rendered), and rollback never rewinds it.
• helm rollback xpander <N> restores any prior revision: everything the chart owns is release-versioned.
• Except the database schema. Migrations run as a versioned Job per revision (kept as the audit trail) and are forward-only: a schema rollback means restore-from-backup, never an automatic down-migration.
• Settings that converge on every upgrade: the one-day access-token lifetime (supabase.db.jwtExpiry) is re-applied on the database side by the migrations hook, so an install born before that default cannot drift back to one hour.

​

Automation & CI

• Every prompt has a flag. With --non-interactive nothing is read from the terminal, even when one exists: missing required values exit 2 with the usage and a completed rerun line rather than blocking. It implies --yes and --on-probe-failure abort.
• Stable progress output. Progress is one line per transition in a grep-friendly shape, [3/8] license: ok, and --quiet reduces output to phase lines and the outcome only. Color is emitted only when stdout is a terminal.
• status is the poll target. A read-only health report: edition, chart, release status, hook jobs, workload readiness, and URLs. --json emits one JSON object.

bash install.sh --non-interactive --quiet \
  --edition airgap --release-manifest ./release-manifest.json \
  --license-file ./license.json --registry registry.example:5000/xpander \
  --org-name "Example Inc" --admin-email ops@example.com \
  --domain xpander.example.internal --storage-class gp3 \
  --on-probe-failure abort

until bash install.sh status --quiet; do sleep 30; done

​

CI/CD & GitOps

pipeline {
  agent { label 'xpander-admin' }        // has kubectl access to the cluster
  triggers { cron('H 6 * * 1') }         // or trigger from releases.atom
  environment {
    XPANDER_PULL = credentials('xpander-pull-token')  // username: xpanderaihub, password: the read token
  }
  stages {
    stage('Mirror the release') {
      steps {
        sh '''
          curl -fsSLO https://charts.xpander.ai/image_sync.sh
          curl -fsSL https://github.com/xpander-ai/releases/releases/download/latest/release-manifest.json -o release-manifest.json
          bash image_sync.sh --source docker.io/xpanderaihub \
            --dest registry.corp.example:5000/xpander \
            --release-manifest ./release-manifest.json \
            --source-user "$XPANDER_PULL_USR" --source-token "$XPANDER_PULL_PSW"
        '''
      }
    }
    stage('Upgrade and verify') {
      steps {
        sh '''
          curl -fsSL https://charts.xpander.ai/install.sh -o install.sh
          bash install.sh upgrade --non-interactive --quiet --release-manifest ./release-manifest.json
          bash install.sh status --quiet
        '''
      }
    }
  }
}

helm upgrade --install xpander ./xpander-airgap-<X.Y.Z>.tgz \
  --namespace xpander --create-namespace \
  -f xpander/values.yaml -f xpander/values.yaml.release.json

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: xpander
  namespace: argocd
spec:
  project: default
  destination:
    server: https://kubernetes.default.svc
    namespace: xpander
  sources:
    - repoURL: registry.corp.example:5000/xpander/charts   # OCI repo credential with "Enable OCI"
      chart: xpander-airgap
      targetRevision: 0.3.2                                # the chart version you mirrored
      helm:
        valueFiles:
          - $values/xpander/values.yaml           # generated once: install.sh --values-out
          - $values/xpander/values.yaml.release.json  # generated release image pins, applied last
    - repoURL: https://git.corp.example/platform/xpander-config.git
      targetRevision: main
      ref: values
  syncPolicy:
    syncOptions:
      - CreateNamespace=true

​

Troubleshooting

​

Mirroring and Registries

writing blob: initiating layer upload to /v2/xpander/ai-gateway/blobs/uploads/ in
123456789012.dkr.ecr.eu-west-2.amazonaws.com: name unknown: The repository with
name 'xpander/ai-gateway' does not exist in the registry with id '123456789012'

curl -fsSLO https://charts.xpander.ai/image_sync.sh

aws ecr create-repository --repository-name xpander/ai-gateway --region <region>

bash image_sync.sh --source docker.io/xpanderaihub --dest <registry>/<path> \
  --dest-user <user> --dest-token <token>

✗ deployment-manager:0.0.417 not found in registry.example:5000/xpander

crane digest registry.example:5000/xpander/deployment-manager:0.0.417

​

Cluster and Network

failed to pull image "docker.io/...": dial tcp 44.208.254.194:443: i/o timeout

kubectl -n xpander get pods -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.containers[*].image}{"\n"}{end}' | grep docker.io

✗ no node with a supported architecture: cluster has [arm64], the images need [amd64]

could not find any suitable subnets for creating the ELB

aws ec2 create-tags --resources <subnet-ids> \
  --tags Key=kubernetes.io/role/internal-elb,Value=1

metadata:
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-internal: "true"

Error from server (NotFound): namespaces "xpander" not found

kubectl create namespace xpander

Missing or invalid environment variables

kubectl -n xpander get ingress

​

Install and Upgrade

helm history xpander -n xpander
helm rollback xpander <last-good-revision> -n xpander   # if a good revision exists
helm uninstall xpander -n xpander                        # first install that never converged

kubectl -n xpander create secret generic xpander-license \
  --from-file=license.json=<new-file> \
  --dry-run=client -o yaml | kubectl -n xpander apply -f -

kubectl -n xpander logs job/<failed-job> --tail=100
kubectl -n xpander describe job/<failed-job>
kubectl -n xpander get pods,jobs,events --sort-by=.lastTimestamp

--set supabase.edgeRuntime.image.repository=null --set supabase.edgeRuntime.image.tag=null

​

Next Steps

• Air-Gapped Security Architecture: zero egress to xpander Cloud, signed Ed25519 license, local sign-in
• Advanced: review-first / manual install: read the installer before running it, plus --manual, --values-out, and --dry-run
• The chart’s own README (inside the .tgz) covers advanced topics: bring-your-own ingress for the UI/API/Supabase gateway hosts, private CA and corporate proxy support, SMTP, internet posture and tool offering, builtin-skills seeding, and Bedrock region pinning