Air-gapped deployment

​

Overview

1. The license is the identity. Two sealed files (install.json and activation.json, the xpander-license/v2 format) plus the organization id you pass with --org: there are no console IDs and no API key. Installs issued with the older single license.json (v1) carry the org id inside the file. The installer generates the environment id, and the first install’s seed pipeline creates everything else in-cluster, including your organization. Single-tenant: the license binds one organization.
2. Images come from your own registry mirror. xpander delivers the artifact set through the gated xpanderaihub Docker Hub org with a per-customer read token; the installer mirrors it into a registry your cluster can reach, or you run the mirroring yourself on a sealed network.
3. Image tags are pinned per release. The chart refuses to render against unpinned images, and upgrades require mirroring the new bundle first.

​

What Talks to What

​

Prerequisites

​

Install in Three Steps

curl -fsSL https://charts.xpander.ai/install.sh | bash -s -- \
  --edition airgap --org <ORG_ID> \
  --install-license-file ./install.json \
  --activation-license-file ./activation.json

bash install.sh --edition airgap \
  --org <ORG_ID> --install-license-file ./install.json --activation-license-file ./activation.json \
  --registry <your-registry>/xpander \
  --chart-path xpander-airgap-<X.Y.Z>.tgz \
  --domain xpander.acme.internal \
  --admin-email <you@company.example> --org-name "<Your Company>" --yes

​

What Happens When You Run the Installer

preflight → keygen → data-checks → workloads → migrations → org-seed → catalog-seed → smoke

kubectl -n xpander get secret xpander-generated-keys \
  -o jsonpath='{.data.XPANDER_ADMIN_PASSWORD}' | base64 -d

​

Releases

• Use the latest release: omit --release-manifest when running the installer or mirror script on a connected machine.
• Repeat one release: retain its JSON manifest and pass the same --release-manifest FILE to both scripts. This also prevents a new release between mirroring and installation from changing the selected images.
• Stop on an invalid release: a missing, incomplete, or invalid manifest stops the operation. It does not fall back to latest, guess individual tags, or silently omit missing artifact images.
• Keep the release consistent: explicit image tag flags must match the manifest. The installer applies the release image pins over conflicting image values. Customer configuration such as registry addresses, infrastructure settings, and secrets remains separate.

​

Existing Helm and GitOps Pipelines

​

When the Mirror Is Empty, the Wizard Offers to Fill It

​

Sealed / Manual Mirroring

curl -fsSLO https://charts.xpander.ai/image_sync.sh
curl -fsSLO https://charts.xpander.ai/install.sh
curl -fsSL https://github.com/xpander-ai/releases/releases/download/latest/release-manifest.json \
  -o release-manifest.json
bash image_sync.sh --source docker.io/xpanderaihub \
  --dest <your-registry>/xpander \
  --release-manifest ./release-manifest.json

# On a connected workstation:
bash image_sync.sh --source docker.io/xpanderaihub \
  --release-manifest ./release-manifest.json --archive-dir ./xpander-images

# After transferring the complete directory, on the inside:
bash image_sync.sh --from-archive ./xpander-images --dest <your-registry>/xpander

helm registry login registry-1.docker.io --username <user> --password-stdin
helm pull oci://registry-1.docker.io/xpanderaihub/xpander-airgap --version <X.Y.Z>
# copy xpander-airgap-<X.Y.Z>.tgz onto the sealed-network host, then --chart-path it

bash image_sync.sh --source docker.io/xpanderaihub \
  --dest nexus.corp.example:5000/xpander \
  --dest-user <user> --dest-token <token>

--registry <nexus-host>/<proxy-repo>/xpanderaihub

​

Data Tier: In-Cluster or External

bash install.sh --edition airgap ... \
  --pg-mode external --pg-dsn 'postgresql://xpander:password@db.internal:5432/xpander' \
  --redis-mode external --redis-endpoint <clustercfg endpoint, no port> \
  --redis-tls --redis-cluster --redis-username <acl-user> --redis-password <pass>

postgres:
  mode: external
  external:
    host: postgres.internal.acme.example
    port: 5432
redis:
  mode: external
global:
  redis:
    endpoint: <clustercfg endpoint, no port>
    clusterMode: true
    auth:
      username: <acl-user>

bash install.sh --edition airgap ... --values values.yaml \
  --set keygen.overrides.POSTGRES_PASSWORD=<postgres password> \
  --set global.redis.auth.password=<redis password>

​

Local LLM Inference

global:
  llm:
    defaults:
      provider: amazon_bedrock
      model: global.anthropic.claude-sonnet-5
    gateway:
      provider: amazon_bedrock
      model: global.anthropic.claude-sonnet-4-6
    bedrock:
      region: <your Bedrock region>

​

A worked vLLM setup on EKS Auto Mode

1. Kubernetes writes every Service in the namespace into a pod’s environment as <NAME>_PORT (service links). A Service named vllm therefore injects VLLM_PORT=tcp://..., and vLLM refuses to start with VLLM_PORT ... appears to be a URI. Set enableServiceLinks: false on the pod, or name the Service something else.
2. A T4 GPU needs --dtype=half. vLLM 0.11 crash-loops on a T4 because its newer engine needs a newer GPU; vllm/vllm-openai:v0.9.2 runs there.
3. Pass the key as the VLLM_API_KEY environment variable, not the --api-key flag. The flag prints the key in the startup log.
4. Start with --enable-auto-tool-choice --tool-call-parser=hermes, the two vLLM flags that let a Qwen model return function calls. xpander’s Add provider dialog probes whether the model accepts function calls; without these flags the discovered model is marked as unable to run agents.
5. Give the model a context window well above the composed prompt. With --max-model-len=8192 the first turn failed with vLLM’s 400 This model's maximum context length is 8192 tokens. However, you requested 8298 tokens. The conversation shows Unknown model error. The controller logs Non-retryable model provider error: Unknown model error next to a [prompt-budget] system_message_tok=... line, which sizes the prompt it composed. --max-model-len=32768 ran. For a custom model the controller’s context optimizer also warns unknown model.id ... defaulting to 128,000 window.

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: gpu
spec:
  template:
    metadata:
      labels:
        workload: gpu
    spec:
      nodeClassRef:
        group: eks.amazonaws.com
        kind: NodeClass
        name: default
      requirements:
        - key: eks.amazonaws.com/instance-family
          operator: In
          values: ["g4dn"]
        - key: eks.amazonaws.com/instance-size
          operator: In
          values: ["xlarge", "2xlarge"]
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["on-demand"]
        - key: kubernetes.io/arch
          operator: In
          values: ["amd64"]
      taints:
        - key: nvidia.com/gpu
          value: "present"
          effect: NoSchedule
      terminationGracePeriod: 10m
  limits:
    cpu: "16"
    nvidia.com/gpu: "2"
  disruption:
    consolidationPolicy: WhenEmpty
    consolidateAfter: 10m
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: vllm-models
  namespace: <namespace>
spec:
  accessModes: ["ReadWriteOnce"]
  storageClassName: <storage class>
  resources:
    requests:
      storage: 40Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm
  namespace: <namespace>
  labels: { app: vllm }
spec:
  replicas: 1
  strategy: { type: Recreate }
  selector: { matchLabels: { app: vllm } }
  template:
    metadata:
      labels: { app: vllm }
    spec:
      enableServiceLinks: false   # a Service named vllm would inject VLLM_PORT=tcp://... and stop vLLM
      nodeSelector:
        workload: gpu
      tolerations:
        - key: nvidia.com/gpu
          operator: Exists
          effect: NoSchedule
      containers:
        - name: vllm
          image: docker.io/vllm/vllm-openai:v0.9.2   # from your mirror; 0.11 does not run on a T4
          args:
            - --model=Qwen/Qwen2.5-1.5B-Instruct
            - --served-model-name=qwen2.5-1.5b-instruct
            - --dtype=half
            - --max-model-len=32768          # 8192 fails the first turn: the composed prompt is larger
            - --gpu-memory-utilization=0.90
            - --host=0.0.0.0
            - --port=8000
            - --enable-auto-tool-choice     # xpander's Add provider dialog probes for this
            - --tool-call-parser=hermes
          env:
            - name: HF_HOME
              value: /models
            - name: VLLM_API_KEY             # the variable, not --api-key: the flag logs the key
              valueFrom:
                secretKeyRef: { name: vllm-api-key, key: key }
          ports:
            - containerPort: 8000
          resources:
            limits:
              nvidia.com/gpu: 1
              memory: 12Gi
            requests:
              cpu: "2"
              memory: 8Gi
          volumeMounts:
            - { name: models, mountPath: /models }
            - { name: shm, mountPath: /dev/shm }
          readinessProbe:
            httpGet: { path: /health, port: 8000 }
            initialDelaySeconds: 60
            periodSeconds: 15
            failureThreshold: 60
      volumes:
        - name: models
          persistentVolumeClaim: { claimName: vllm-models }
        - name: shm
          emptyDir: { medium: Memory, sizeLimit: 2Gi }
---
apiVersion: v1
kind: Service
metadata:
  name: vllm
  namespace: <namespace>
spec:
  selector: { app: vllm }
  ports:
    - port: 8000
      targetPort: 8000

​

Register the endpoint in xpander

1. Open Settings > LLM settings and click Add custom provider. The dialog is titled Add provider.
2. Name is free text. Slug is derived from it, lowercase with underscores, and stays stable. Protocol is OpenAI-compatible. Base URL is the address the pods reach, for example http://vllm.<namespace>.svc.cluster.local:8000/v1.
3. API key: No key (local or unauthenticated server), Use an existing key, or Create a new key with a Key name and the API key (the value in VLLM_API_KEY).
4. Under Models, click Test connection. A reachable endpoint answers Reachable with the round-trip time and lists the models it serves; a model that accepts function calls carries a tools badge. Click Add on the model, and give it a Display name if you want one.
5. Click Add provider. The provider row reads OpenAI-compatible, how many of its models are in the catalog, and Reachable.
6. Open the provider’s model list and choose Set model cost from the model row’s menu. The dialog says what it is for: “Agents can use the model once it has a cost.” Enter the Price per 1M tokens (USD): Input and Output, and optionally Cache read and Cache write. These prices measure credit burn for the model; they are never charged. Who can use it is Everyone in the workspace or Specific groups and people. Click Save cost.

​

The executor fleet on Air-Gap

​

AI Skill Generation

catalog:
  env:
    OAS_GEN_BUCKET_NAME: <bucket>
    AWS_DEFAULT_REGION: <region>
openapi-spec-generator:
  env:
    OAS_GEN_BUCKET_NAME: <bucket>
    AWS_DEFAULT_REGION: <region>

​

Custom Actions and Code Execution

actions:
  env:
    XPANDER_SANDBOX_DNS: <ClusterIP of the CoreDNS Service>   # 172.20.0.10 on EKS; kubectl -n kube-system get svc kube-dns

​

The License

kubectl -n xpander create secret generic xpander-license \
  --from-file=install.json=./install.json \
  --from-file=activation.json=./activation.json \
  --dry-run=client -o yaml | kubectl -n xpander apply -f -

• Your own --values file: the installer generates the global.license block (secretName, installKey, activationKey, publicKey) into its own values. When you also pass --values of your own, pass --set global.license.installKey=install.json --set global.license.activationKey=activation.json --set global.license.publicKey=<license public key> as well, or your file displaces the installer’s license block.
• Renewing: renewal re-issues only activation.json. Replace that one key in the Secret (leave install.json as it is) and redeploy: roll the license consumers, run install.sh upgrade by hand, or let your pipeline re-run its install. License consumers carry a checksum annotation over the Secret, so a redeploy rolls them onto the new file. No reinstall, no re-mirror. Steps: Renewing: re-activate.
• Expired activation: the installer warns at the license phase and the chart’s preflight refuses a fresh install. A running install degrades in stages instead of stopping: GET /v1/license/status on the API (with a signed-in user’s bearer token) returns the current state, valid, expiring, grace or expired, plus the deployment and granted features; alert on anything past expiring.
• Single-tenant: the license binds one organization; the org seed creates exactly that org.
• Legacy single file (xpander-license/v1): --license-file ./license.json creates the Secret with one license.json key, wired through global.license.key. Renewal on that path is a new license.json in the Secret plus the same redeploy.

​

Domains, Ingress & DNS

• app. for the UI
• auth. for Supabase
• api.
• chat. for agent-controller
• hooks. for the webhook service
• links. for short links (the functions Service)
• actions. and client-auth.
• dm.internal. for deployment-manager, on a second, internal ingress class (deployment-manager.ingress.className), because remote runtimes and the edge functions reach it over a backchannel rather than the user-facing path

• --ui-host <host> replaces the derived chat.<domain> only; any other host is overridable per service via <name>.ingress.host in values (the table above lists the URL knobs each rename drags along).
• --ingress-scheme https|http (default https) sets how the browser reaches those hosts: it feeds every derived URL. Use http only for rigs without certificates.
• No ingress controller is assumed. Set the class and annotations per component (<name>.ingress.className / annotations) or once for all of them via global.ingress.className / global.ingress.annotations.

global:
  ingress:
    className: traefik
    tls:
      - hosts: ["*.xpander.acme.internal"]
        secretName: xpander-wildcard-tls

​

Making the Hosts Resolve

*.xpander.acme.internal.  CNAME  <your-ingress-controller's address>

sudo kubectl -n <ingress-namespace> port-forward svc/<controller-service> 443:443
# installed with --ingress-scheme http, forward port 80 instead:
sudo kubectl -n <ingress-namespace> port-forward svc/<controller-service> 80:80

# /etc/hosts
127.0.0.1 chat.xpander.acme.internal api.xpander.acme.internal agent-controller.xpander.acme.internal supabase.xpander.acme.internal webhook.xpander.acme.internal client-auth.xpander.acme.internal actions.xpander.acme.internal

​

Signing In

1. Bootstrap password (default): open the UI (https://chat.<domain>) at /login-password with your --admin-email and the password from the summary; read it again any time:
   kubectl -n xpander get secret xpander-generated-keys \
     -o jsonpath='{.data.XPANDER_ADMIN_PASSWORD}' | base64 -d
   
2. Email OTP without SMTP: the login page’s email flow works with no mailer. The code is generated and stored, just not sent. Print it with the chart’s scripts/get_login_code.sh <email> <namespace>. With SMTP configured (global.smtp.* in values), the same codes arrive by email. On installs with inbound agent mailboxes, agents’ email replies deliver through the same relay. INBOUND_EMAIL_REPLY_DOMAIN controls the From domain, and the SMTP identity must be allowed to send as the agents’ mailbox subdomain.
3. SSO with your identity provider (OIDC): the SSO button is driven by the OIDC configuration an administrator enters in Xpander Chat (Settings, OIDC), served by the client-auth component, not by GoTrue’s external providers. Setup, IdP client settings, the Keycloak preferred_username note and troubleshooting: SSO with your identity provider. Google and GitHub social buttons remain GoTrue providers (supabase.auth.extraEnv).

kubectl -n xpander port-forward svc/ui 8080:80   # then http://localhost:8080

​

Upgrading an Air-Gapped Install

​

Slack handle management compatibility

​

Upgrade procedure

bash install.sh upgrade --chart-path xpander-airgap-<X.Y.Z>.tgz \
  --release-manifest ./release-manifest.json

​

Declarative re-install for CI

• --helm-timeout 20m (the default is 30m). Upgrades run --atomic, and the rollback waits with the same timeout, so budget twice the value plus a few minutes for hooks inside your job’s credential lifetime. A one-hour OIDC session cannot host a 30-minute timeout with its rollback.
• Heal before you run. The installer refuses to install over a release in pending-install, pending-upgrade or failed. Run helm history xpander -n xpander, then helm rollback xpander <last deployed revision> -n xpander, or helm uninstall xpander -n xpander when no deployed revision exists (the xpander-generated-keys Secret survives an uninstall). Delete Failed-phase pods and failed hook Jobs first so the new run starts clean.
• Pin the environment id on every re-install. The first install creates the organization’s default location, the runtime environment its agents run in, and a later install cannot point it at a new id. A pipeline that re-runs the install must therefore pass --env <existing default environment id> after the first run. Read the id from xpander’s database before installing:
  kubectl -n xpander exec supabase-db-0 -c postgres -- psql -U supabase_admin -d postgres -tAc \
    "SELECT id FROM public.organization_environments WHERE is_default LIMIT 1"
  
• Wait for Helm’s own verdict. The installer’s stage watcher exits on the first failed hook pod, and a pod evicted while the cluster rebalances counts even though its Job retries and succeeds. When install.sh exits non-zero while its helm upgrade --install xpander process is still running, do not kill that process: killing Helm mid-upgrade strands the release in pending-upgrade. Wait for Helm to finish and read the release status; a retried hook pod under a Job that completed is not a failure.
• global.upgradeStrategy: replace on clusters that provision nodes on demand. A surge upgrade of every Deployment can cascade into node-provisioning waves that outlive Helm’s timeout and roll back a healthy upgrade; replace recreates pods in place instead.

• Generated keys survive every upgrade: the keygen hook is lookup-preserve, the xpander-generated-keys Secret is Job-created (never Helm-rendered), and rollback never rewinds it.
• helm rollback xpander <N> restores any prior revision: everything the chart owns is release-versioned.
• Except the database schema. Migrations run as a versioned Job per revision (kept as the audit trail) and are forward-only: a schema rollback means restore-from-backup, never an automatic down-migration.
• Settings that converge on every upgrade: the one-day access-token lifetime (supabase.db.jwtExpiry) is re-applied on the database side by the migrations hook, so an install born before that default cannot drift back to one hour.

​

Automation & CI

• Every prompt has a flag. With --non-interactive nothing is read from the terminal, even when one exists: missing required values exit 2 with the usage and a completed rerun line rather than blocking. It implies --yes and --on-probe-failure abort.
• Stable progress output. Progress is one line per transition in a grep-friendly shape, [3/8] license: ok, and --quiet reduces output to phase lines and the outcome only. Color is emitted only when stdout is a terminal.
• status is the poll target. A read-only health report: edition, chart, release status, hook jobs, workload readiness, and URLs. --json emits one JSON object.

bash install.sh --non-interactive --quiet \
  --edition airgap --release-manifest ./release-manifest.json \
  --org <ORG_ID> --install-license-file ./install.json --activation-license-file ./activation.json \
  --registry registry.example:5000/xpander \
  --org-name "Example Inc" --admin-email ops@example.com \
  --domain xpander.example.internal --storage-class gp3 \
  --on-probe-failure abort

until bash install.sh status --quiet; do sleep 30; done

​

Reference values file

global:
  upgradeStrategy: replace                  # in-place recreation; surge upgrades cascade into node waves on autoscaled clusters
  imageRegistry: <registry>/<mirror path>   # every image is pulled from the mirror
  license:
    secretName: xpander-license
  smtp:                                     # one block feeds sign-in mail, invitations and xpander's notifications
    host: email-smtp.<region>.amazonaws.com
    port: 587
    starttls: true
    user: <smtp user>
    passwordSecret: { name: xpander-smtp, key: password }
    adminEmail: admin@<domain>
    senderName: xpander
  assetStore: { publicUrl: https://auth.<domain> }
  shortLinks: { publicUrl: https://links.<domain> }
  # mcp.publicUrl unset: Settings > Channels lists MCP as not configured until you set the install's MCP origin
  llm:
    defaults: { provider: amazon_bedrock, model: global.anthropic.claude-sonnet-5, apiBase: "", credentialsKey: "" }
    bedrock: { region: us-west-2 }          # may differ from the cluster's region
    gateway: { provider: amazon_bedrock, model: global.anthropic.claude-sonnet-4-6 }
  harness:
    defaultProviders: { claudeCode: amazon_bedrock }
    defaultModels: { claudeCode: global.anthropic.claude-sonnet-5 }
  fleet:
    replicas: 1
    storage: { createClass: true, provisioner: ebs.csi.eks.amazonaws.com }
  network:
    internalFetchAllowlist: [".internal.<domain>"]   # intranet hosts agents may fetch for knowledge ingest
  redis: { endpoint: <clustercfg endpoint>, clusterMode: true, auth: { username: <acl user> } }
redis: { mode: external }
postgres: { mode: external, external: { host: <rds host>, port: 5432 } }
supabase:
  ingress: { enabled: true, className: alb, host: auth.<domain>, annotations: { ... } }
  auth: { apiExternalUrl: https://auth.<domain>, siteUrl: https://app.<domain> }
  storage:
    backend: s3
    s3: { bucket: <bucket>, region: <region> }
    serviceAccountName: storage-api
    publicUrl: https://auth.<domain>
  edgeRuntime:
    publicUrl: https://auth.<domain>
    functionsEnv:
      APP_URL: https://app.<domain>
      INBOUND_GW_URL: https://hooks.<domain>
      AGENT_CONTAINER_INGRESS_DOMAIN: <domain>
      DEPLOYMENT_MANAGER_PUBLIC_URL: https://dm.internal.<domain>
ui:
  ingress: { host: app.<domain>, className: alb }
  authProviders: [sso]
  env:
    VITE_WEBHOOK_URL: https://hooks.<domain>     # the app's baked default is the cloud host
    VITE_EMAIL_DOMAIN: agents.<domain>
    VITE_PLATFORM_URL: https://app.<domain>
    VITE_SUPABASE_URL: https://auth.<domain>
    VITE_AGENTS_API_URL: https://api.<domain>
    VITE_CHAT_BACKEND_URL: https://chat.<domain>/chat-backend/{agent_id}
    VITE_ACTIONS_SERVICE_URL: https://actions.<domain>
    VITE_CLIENT_AUTH_URL: https://client-auth.<domain>
    VITE_LICENSE_MODE: "true"
    VITE_OMNI_MCP_URL: ""
  extraEnv:
    - name: VITE_SUPABASE_ANON_KEY
      valueFrom: { secretKeyRef: { name: xpander-generated-keys, key: SUPABASE_ANON_KEY } }
client-auth: { ingress: { host: client-auth.<domain> }, env: { APP_URL: https://app.<domain>, CLIENT_AUTH_EXTERNAL_URL: https://client-auth.<domain> } }   # the second turns on MCP OAuth
api:
  ingress: { host: api.<domain> }
  env: { API_URL: https://api.<domain>, INBOUND_EMAIL_REPLY_DOMAIN: agents.<domain>, SMTP_HOST: ..., SMTP_PORT: "587", SMTP_STARTTLS: "true", SMTP_USERNAME: ... }
  extraEnv: [SMTP_PASSWORD, INBOUND_EMAIL_TOKEN, MAILGUN_SIGNING_KEY]   # each from a Secret
agent-controller: { ingress: { host: chat.<domain> } }
webhook: { ingress: { host: hooks.<domain> } }
functions: { ingress: { host: links.<domain> } }
deployment-manager: { ingress: { host: dm.internal.<domain>, className: alb-internal } }
actions:
  ingress: { host: actions.<domain> }
  env: { XPANDER_SANDBOX_DNS: <CoreDNS ClusterIP> }
catalog: { env: { OAS_GEN_BUCKET_NAME: <versioned bucket>, AWS_DEFAULT_REGION: <region> } }
openapi-spec-generator: { env: { OAS_GEN_BUCKET_NAME: <versioned bucket>, AWS_DEFAULT_REGION: <region> } }

​

CI/CD & GitOps

pipeline {
  agent { label 'xpander-admin' }        // has kubectl access to the cluster
  triggers { cron('H 6 * * 1') }         // or trigger from releases.atom
  environment {
    XPANDER_PULL = credentials('xpander-pull-token')  // username: xpanderaihub, password: the read token
    XPANDER_ORG  = credentials('xpander-org-id')      // the organization id from the license handoff
  }
  stages {
    stage('Mirror the release') {
      steps {
        sh '''
          curl -fsSLO https://charts.xpander.ai/image_sync.sh
          curl -fsSL https://github.com/xpander-ai/releases/releases/download/latest/release-manifest.json -o release-manifest.json
          bash image_sync.sh --source docker.io/xpanderaihub \
            --dest registry.corp.example:5000/xpander \
            --release-manifest ./release-manifest.json \
            --source-user "$XPANDER_PULL_USR" --source-token "$XPANDER_PULL_PSW"
        '''
      }
    }
    stage('Upgrade and verify') {
      steps {
        sh '''
          curl -fsSL https://charts.xpander.ai/install.sh -o install.sh
          bash install.sh --non-interactive --quiet --edition airgap \
            --org "$XPANDER_ORG" --install-license-file install.json --activation-license-file activation.json \
            --registry registry.corp.example:5000/xpander --mirror skip \
            --release-manifest ./release-manifest.json --values values.yaml \
            --helm-timeout 20m
          bash install.sh status --quiet
        '''
      }
    }
  }
}

helm upgrade --install xpander ./xpander-airgap-<X.Y.Z>.tgz \
  --namespace xpander --create-namespace \
  -f xpander/values.yaml -f xpander/values.yaml.release.json

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: xpander
  namespace: argocd
spec:
  project: default
  destination:
    server: https://kubernetes.default.svc
    namespace: xpander
  sources:
    - repoURL: registry.corp.example:5000/xpander/charts   # OCI repo credential with "Enable OCI"
      chart: xpander-airgap
      targetRevision: <the chart version you mirrored>     # keep it in step with the release values
      helm:
        valueFiles:
          - $values/xpander/values.yaml           # generated once: install.sh --values-out
          - $values/xpander/values.yaml.release.json  # generated release image pins, applied last
    - repoURL: https://git.corp.example/platform/xpander-config.git
      targetRevision: main
      ref: values
  syncPolicy:
    syncOptions:
      - CreateNamespace=true

​

Troubleshooting

​

Mirroring and Registries

writing blob: initiating layer upload to /v2/xpander/ai-gateway/blobs/uploads/ in
123456789012.dkr.ecr.eu-west-2.amazonaws.com: name unknown: The repository with
name 'xpander/ai-gateway' does not exist in the registry with id '123456789012'

curl -fsSLO https://charts.xpander.ai/image_sync.sh

aws ecr create-repository --repository-name xpander/ai-gateway --region <region>

bash image_sync.sh --source docker.io/xpanderaihub --dest <registry>/<path> \
  --dest-user <user> --dest-token <token>

✗ deployment-manager:0.0.417 not found in registry.example:5000/xpander

crane digest registry.example:5000/xpander/deployment-manager:0.0.417

​

Cluster and Network

failed to pull image "docker.io/...": dial tcp 44.208.254.194:443: i/o timeout

kubectl -n xpander get pods -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.containers[*].image}{"\n"}{end}' | grep docker.io

✗ no node with a supported architecture: cluster has [arm64], the images need [amd64]

could not find any suitable subnets for creating the ELB

aws ec2 create-tags --resources <subnet-ids> \
  --tags Key=kubernetes.io/role/internal-elb,Value=1

metadata:
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-internal: "true"

Error from server (NotFound): namespaces "xpander" not found

kubectl create namespace xpander

Missing or invalid environment variables

kubectl -n xpander get ingress

​

Install and Upgrade

helm history xpander -n xpander
helm rollback xpander <last-good-revision> -n xpander   # if a good revision exists
helm uninstall xpander -n xpander                        # first install that never converged

kubectl -n xpander create secret generic xpander-license \
  --from-file=install.json=./install.json \
  --from-file=activation.json=./activation-new.json \
  --dry-run=client -o yaml | kubectl -n xpander apply -f -

-- the connections of the faulty connector
update public.connector_organization set deleted_at = now()
  where connector_id = '<connector id>' and deleted_at is null;
-- take the connection id out of the agent's skill list
update public.ai_agents
  set attached_tools = (select coalesce(jsonb_agg(t), '[]'::jsonb)
    from jsonb_array_elements(attached_tools::jsonb) t
    where t->>'id' <> '<connection id>')::json
  where id = '<agent id>';

kubectl -n xpander logs job/<failed-job> --tail=100
kubectl -n xpander describe job/<failed-job>
kubectl -n xpander get pods,jobs
kubectl -n xpander get events --sort-by=.lastTimestamp

--set supabase.edgeRuntime.image.repository=null --set supabase.edgeRuntime.image.tag=null

​

Next Steps

• Air-Gapped Security Architecture: zero egress to xpander Cloud, signed Ed25519 license, local sign-in
• Advanced: review-first / manual install: read the installer before running it, plus --manual, --values-out, and --dry-run
• The chart’s own README (inside the .tgz) covers advanced topics: bring-your-own ingress for the UI/API/Supabase gateway hosts, private CA and corporate proxy support, SMTP, internet posture and skill offering, builtin-skills seeding, and Bedrock region pinning