Air-gapped deployment

​

What runs where

• In your environment: the web UI, control plane, a self-hosted Supabase stack (auth, storage, edge functions) and the data tier (PostgreSQL, Redis Stack, MongoDB), as one Helm release in one namespace. Agent skill calls (including MCP) and agent workspaces run there too.
• Sent to xpander: nothing. No heartbeats, no catalog sync, no telemetry.
• Your choice: your LLM endpoint, cloud services such as Bedrock, S3 or SES, and the registry you mirror from at install time.
• MCP clients (Claude, ChatGPT, Cursor) reach Omni once you set global.mcp.publicUrl. Until then, with ui.env.VITE_OMNI_MCP_URL empty too, Platform > Agents > Channels shows Not offered on this install, and the License page’s Capabilities card shows the value to add.
• Private mode needs global.network.internetEgress: false. With true, built-in skills that use public APIs (for example arXiv and HackerNews) call them from the cluster. With false, those skills do not work.

​

Three things define an air-gapped install

1. The license is the identity. Two sealed files (install.json and activation.json, format xpander-license/v2) plus the organization id you pass with --org. No console IDs and no API key. Older single-file license.json (v1) installs carry the org id inside. The installer generates the environment id and the first install creates your organization in-cluster. Single-tenant: one license, one organization.
2. Images come from your own registry mirror. xpander delivers them through the gated xpanderaihub Docker Hub org with a per-customer read token. The installer mirrors them, or you mirror them yourself on a sealed network.
3. Image tags are pinned per release. The chart refuses unpinned images, and an upgrade needs the new bundle mirrored first.

​

What talks to what

​

Prerequisites

• The executor fleet (where Claude Code, Codex and OpenCode turns run) adds to these numbers and needs a namespace that admits PodSecurity baseline, Landlock on its nodes (or global.fleet.landlock.enabled: false) and XFS with prjquota. No privileged container or node preparation. See Node prerequisites.
• The LLM Gateway is always on, and agents’ model calls go through it. It adds its pods (0.5 vCPU / 1 GiB requested each, up to 1 vCPU / 4 GiB) and the llm-gw image. See LLM Gateway on a self-hosted install.

​

Install in three steps

curl -fsSL https://charts.xpander.ai/install.sh | bash -s -- \
  --edition airgap --org <ORG_ID> \
  --install-license-file ./install.json \
  --activation-license-file ./activation.json

bash install.sh --edition airgap \
  --org <ORG_ID> --install-license-file ./install.json --activation-license-file ./activation.json \
  --registry <your-registry>/xpander \
  --chart-path xpander-airgap-<X.Y.Z>.tgz \
  --domain xpander.acme.internal \
  --admin-email <you@company.example> --org-name "<Your Company>" --yes

​

What happens when you run the installer

preflight → keygen → data-checks → workloads → migrations → org-seed → catalog-seed → smoke

kubectl -n xpander get secret xpander-generated-keys \
  -o jsonpath='{.data.XPANDER_ADMIN_PASSWORD}' | base64 -d

​

Releases

• Latest release: omit --release-manifest on a connected machine.
• Repeat a release: pass the same --release-manifest FILE to both scripts, so a new release between mirroring and installing changes nothing.
• Invalid release: stops the run. It never falls back to latest or guesses tags.
• Consistency: tag flags must match the manifest, and its pins override conflicting image values. Registry, infrastructure and secrets stay yours.

​

Existing Helm and GitOps pipelines

​

When the mirror is empty, the wizard offers to fill it

​

Sealed / manual mirroring

curl -fsSLO https://charts.xpander.ai/image_sync.sh
curl -fsSLO https://charts.xpander.ai/install.sh
curl -fsSL https://github.com/xpander-ai/releases/releases/download/latest/release-manifest.json \
  -o release-manifest.json
bash image_sync.sh --source docker.io/xpanderaihub \
  --dest <your-registry>/xpander \
  --release-manifest ./release-manifest.json

# On a connected workstation:
bash image_sync.sh --source docker.io/xpanderaihub \
  --release-manifest ./release-manifest.json --archive-dir ./xpander-images

# After transferring the complete directory, on the inside:
bash image_sync.sh --from-archive ./xpander-images --dest <your-registry>/xpander

helm registry login registry-1.docker.io --username <user> --password-stdin
helm pull oci://registry-1.docker.io/xpanderaihub/xpander-airgap --version <X.Y.Z>
# copy xpander-airgap-<X.Y.Z>.tgz onto the sealed-network host, then --chart-path it

bash image_sync.sh --source docker.io/xpanderaihub \
  --dest nexus.corp.example:5000/xpander \
  --dest-user <user> --dest-token <token>

--registry <nexus-host>/<proxy-repo>/xpanderaihub

​

Data tier: in-cluster or external

bash install.sh --edition airgap ... \
  --pg-mode external --pg-dsn 'postgresql://xpander:password@db.internal:5432/xpander' \
  --redis-mode external --redis-endpoint <clustercfg endpoint, no port> \
  --redis-tls --redis-cluster --redis-username <acl-user> --redis-password <pass>

postgres:
  mode: external
  external:
    host: postgres.internal.acme.example
    port: 5432
redis:
  mode: external
global:
  redis:
    endpoint: <clustercfg endpoint, no port>
    clusterMode: true
    auth:
      username: <acl-user>

bash install.sh --edition airgap ... --values values.yaml \
  --set keygen.overrides.POSTGRES_PASSWORD=<postgres password> \
  --set global.redis.auth.password=<redis password>

​

Local LLM inference

global:
  llm:
    defaults:
      provider: amazon_bedrock
      model: global.anthropic.claude-sonnet-5
    gateway:
      provider: amazon_bedrock
      model: global.anthropic.claude-sonnet-4-6
    bedrock:
      region: <your Bedrock region>

​

A worked vLLM setup on EKS Auto Mode

1. A Service named vllm injects VLLM_PORT=tcp://... and vLLM refuses to start (VLLM_PORT ... appears to be a URI). Set enableServiceLinks: false or rename the Service.
2. A T4 needs --dtype=half. vLLM 0.11 crash-loops on a T4. vllm/vllm-openai:v0.9.2 works.
3. Pass the key as VLLM_API_KEY, not --api-key (which logs it). It guards only /v1. /health, /metrics and /tokenize answer without it, so keep port 8000 on a ClusterIP Service, never an ingress or load balancer.
4. Start with --enable-auto-tool-choice --tool-call-parser=hermes, or Add provider marks the model as unable to run agents.
5. Leave room above the composed prompt. --max-model-len=8192 failed with 400 This model's maximum context length is 8192 tokens. However, you requested 8298 tokens, shown as Unknown model error (controller log: Non-retryable model provider error: Unknown model error next to [prompt-budget] system_message_tok=...). --max-model-len=32768 ran. Custom models also log unknown model.id ... defaulting to 128,000 window.

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: gpu
spec:
  template:
    metadata:
      labels:
        workload: gpu
    spec:
      nodeClassRef:
        group: eks.amazonaws.com
        kind: NodeClass
        name: default
      requirements:
        - key: eks.amazonaws.com/instance-family
          operator: In
          values: ["g4dn"]
        - key: eks.amazonaws.com/instance-size
          operator: In
          values: ["xlarge", "2xlarge"]
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["on-demand"]
        - key: kubernetes.io/arch
          operator: In
          values: ["amd64"]
      taints:
        - key: nvidia.com/gpu
          value: "present"
          effect: NoSchedule
      terminationGracePeriod: 10m
  limits:
    cpu: "16"
    nvidia.com/gpu: "2"
  disruption:
    consolidationPolicy: WhenEmpty
    consolidateAfter: 10m
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: vllm-models
  namespace: <namespace>
spec:
  accessModes: ["ReadWriteOnce"]
  storageClassName: <storage class>
  resources:
    requests:
      storage: 40Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm
  namespace: <namespace>
  labels: { app: vllm }
spec:
  replicas: 1
  strategy: { type: Recreate }
  selector: { matchLabels: { app: vllm } }
  template:
    metadata:
      labels: { app: vllm }
    spec:
      enableServiceLinks: false   # a Service named vllm would inject VLLM_PORT=tcp://... and stop vLLM
      nodeSelector:
        workload: gpu
      tolerations:
        - key: nvidia.com/gpu
          operator: Exists
          effect: NoSchedule
      containers:
        - name: vllm
          image: docker.io/vllm/vllm-openai:v0.9.2   # from your mirror; 0.11 does not run on a T4
          args:
            - --model=Qwen/Qwen2.5-1.5B-Instruct
            - --served-model-name=qwen2.5-1.5b-instruct
            - --dtype=half
            - --max-model-len=32768          # 8192 fails the first turn: the composed prompt is larger
            - --gpu-memory-utilization=0.90
            - --host=0.0.0.0
            - --port=8000
            - --enable-auto-tool-choice     # xpander's Add provider dialog probes for this
            - --tool-call-parser=hermes
          env:
            - name: HF_HOME
              value: /models
            - name: VLLM_API_KEY             # the variable, not --api-key: the flag logs the key
              valueFrom:
                secretKeyRef: { name: vllm-api-key, key: key }
          ports:
            - containerPort: 8000
          resources:
            limits:
              nvidia.com/gpu: 1
              memory: 12Gi
            requests:
              cpu: "2"
              memory: 8Gi
          volumeMounts:
            - { name: models, mountPath: /models }
            - { name: shm, mountPath: /dev/shm }
          readinessProbe:
            httpGet: { path: /health, port: 8000 }
            initialDelaySeconds: 60
            periodSeconds: 15
            failureThreshold: 60
      volumes:
        - name: models
          persistentVolumeClaim: { claimName: vllm-models }
        - name: shm
          emptyDir: { medium: Memory, sizeLimit: 2Gi }
---
apiVersion: v1
kind: Service
metadata:
  name: vllm
  namespace: <namespace>
spec:
  selector: { app: vllm }
  ports:
    - port: 8000
      targetPort: 8000

​

Register the endpoint in xpander

1. Platform > AI governance > Models, Add provider.
2. Name is free text, and ID (used in API calls) derives from it and is fixed. Protocol: OpenAI-compatible. Base URL: the address the pods reach, for example http://vllm.<namespace>.svc.cluster.local:8000/v1. In-cluster addresses work on Hybrid and Air-Gapped. xpander cloud accepts only a public https address.
3. API key: No key (local or unauthenticated server), Use an existing key, or Create a new key with a Key name and the API key (the VLLM_API_KEY value).
4. Test connection shows Reachable and the served models. A tools badge marks models that accept function calls. Add the model, with an optional Display name.
5. Click Add provider.
6. In the provider’s Keys and models…, tick the models the organization may use. Prices (USD per 1M tokens, via Edit model) track spend and are never charged. Who may use it… sets access. New models wait under New models unless Approve new models automatically is on. Agent readiness shows, from the last Test model, which harnesses work with each model.

​

The executor fleet on Air-Gapped

​

AI skill generation

catalog:
  env:
    OAS_GEN_BUCKET_NAME: <bucket>
    AWS_DEFAULT_REGION: <region>
openapi-spec-generator:
  env:
    OAS_GEN_BUCKET_NAME: <bucket>
    AWS_DEFAULT_REGION: <region>

​

Custom actions and code execution

actions:
  env:
    XPANDER_SANDBOX_DNS: <ClusterIP of the CoreDNS Service>   # 172.20.0.10 on EKS; kubectl -n kube-system get svc kube-dns

​

The license

kubectl -n xpander create secret generic xpander-license \
  --from-file=install.json=./install.json \
  --from-file=activation.json=./activation.json \
  --dry-run=client -o yaml | kubectl -n xpander apply -f -

• Your own --values file displaces the installer’s license block: also pass --set global.license.installKey=install.json --set global.license.activationKey=activation.json --set global.license.publicKey=<license public key>.
• Renewing replaces only activation.json and redeploys. See Renewing: re-activate.
• Expired activation: the installer warns and preflight refuses a fresh install. A running install degrades in stages. GET /v1/license/status returns valid, expiring, grace or expired. Alert on anything past expiring.
• Single-tenant: one license, one organization.
• Legacy single file (xpander-license/v1): --license-file ./license.json creates a Secret with one license.json key, wired through global.license.key. Renew by replacing that file and redeploying.

​

Domains, ingress & DNS

• --ingress-scheme https|http (default https) feeds every derived URL. Use http only without certificates.
• No ingress controller is assumed: set <name>.ingress.className / annotations, or global.ingress.className / global.ingress.annotations.

global:
  ingress:
    className: traefik
    tls:
      - hosts: ["*.xpander.acme.internal"]
        secretName: xpander-wildcard-tls

​

Making the hosts resolve

*.xpander.acme.internal.  CNAME  <your-ingress-controller's address>

sudo kubectl -n <ingress-namespace> port-forward svc/<controller-service> 443:443
# installed with --ingress-scheme http, forward port 80 instead:
sudo kubectl -n <ingress-namespace> port-forward svc/<controller-service> 80:80

# /etc/hosts
127.0.0.1 chat.xpander.acme.internal api.xpander.acme.internal agent-controller.xpander.acme.internal supabase.xpander.acme.internal webhook.xpander.acme.internal client-auth.xpander.acme.internal actions.xpander.acme.internal

​

Signing in

1. Bootstrap password (default): open https://chat.<domain>/login-password with your --admin-email. Read the password again with:
   kubectl -n xpander get secret xpander-generated-keys \
     -o jsonpath='{.data.XPANDER_ADMIN_PASSWORD}' | base64 -d
   
2. Email OTP without SMTP: codes are not sent. Get one with scripts/get_login_code.sh <email> <namespace>. With global.smtp.* set, codes arrive by email. On Amazon SES in the sandbox, invitations reach only verified identities, and the SMTP IAM policy must list each verified recipient identity ARN: request production access first (Email). Agent mailbox replies use the same relay: INBOUND_EMAIL_REPLY_DOMAIN sets the From domain, and the SMTP identity must be allowed to send as the agents’ mailbox subdomain.
3. SSO (OIDC): configured in xpander Chat under Platform > Organization > Access and served by client-auth. See SSO with your identity provider. Google and GitHub buttons stay GoTrue providers (supabase.auth.extraEnv).

kubectl -n xpander port-forward svc/ui 8080:80   # then http://localhost:8080

​

Upgrading an Air-Gapped install

​

Slack handle management compatibility

​

Release v2026.10.22 (chart xpander-airgap 0.17.2)

• The LLM Gateway is always on and llm-gw.enabled is ignored. If you had it off, mirror llm-gw, give every provider your agents use a key, and open its network paths (LLM Gateway on a self-hosted install).
• In-cluster Postgres moves to the pgvector build of PostgreSQL 16.15. Mirror pgvector/pgvector, and snapshot the Postgres volume or set postgres.upgradeGuard.backup.pvcName. Postgres restarts once. A pinned postgres.image must stay on a pgvector build of the same major version.
• Per-agent MCP URLs are gone. Use https://api.<your domain>/mcp/. There is no per-person library endpoint on air-gap, and Omni over MCP needs global.mcp.publicUrl.
• BRAIN_INGEST_API_KEY is added to the generated secret automatically.

curl -fsSL https://charts.xpander.ai/image_sync.sh -o image_sync.sh
curl -fsSL https://github.com/xpander-ai/releases/releases/download/v2026.10.22/release-manifest.json -o release-manifest.json

bash image_sync.sh --source-user xpanderaihub --source-token <xpander-token> \
  --dest <your-registry> --release-manifest release-manifest.json --manifest-out pins.yaml

helm registry login registry-1.docker.io -u xpanderaihub -p <xpander-token>
helm upgrade xpander oci://registry-1.docker.io/xpanderaihub/xpander-airgap --version 0.17.2 \
  -n <namespace> --reset-then-reuse-values -f pins.yaml --wait --timeout 30m

​

Release v2026.10.15 (chart xpander-airgap 0.16.2)

• Mirror with the current image_sync.sh (adds llm-gw and tool-repo-seed, drops --icon-pack-tag).
• The LLM Gateway is on by default. Its generated keys are kept on every upgrade. Never rotate the content key. Every provider your agents use needs a key, or calls are refused naming the missing key. Check kubectl -n <namespace> get pods -l app=llm-gw: two llm-gw pods (one on a single-node install.sh cluster), 1/1 Running. See LLM Gateway on a self-hosted install.
• Key names and endpoints: env-var key names matching the install’s own credentials read as no key, and metadata-address endpoints are refused (details).
• Helm 3.14+ for --reset-then-reuse-values. Migrations run automatically.

​

Upgrade procedure

bash install.sh upgrade --chart-path xpander-airgap-<X.Y.Z>.tgz \
  --release-manifest ./release-manifest.json

​

Declarative re-install for CI

• --helm-timeout 20m (default 30m). The --atomic rollback waits the same time again, so budget twice the value plus hooks within your job’s credential lifetime. A one-hour OIDC session cannot host a 30-minute timeout.
• Heal first. The installer refuses a release in pending-install, pending-upgrade or failed. Check helm history xpander -n xpander, then helm rollback xpander <last deployed revision> -n xpander, or helm uninstall xpander -n xpander when none deployed (xpander-generated-keys survives). Delete Failed pods and failed hook Jobs first.
• Pin the environment id. After the first install, pass --env <existing default environment id>: the default location’s runtime environment cannot move to a new id. Read it first:
  kubectl -n xpander exec supabase-db-0 -c postgres -- psql -U supabase_admin -d postgres -tAc \
    "SELECT id FROM public.organization_environments WHERE is_default LIMIT 1"
  
• Wait for Helm’s verdict. The stage watcher exits on the first failed hook pod, even an evicted one whose Job then succeeds. If install.sh exits non-zero while helm upgrade --install xpander still runs, do not kill Helm (it strands the release in pending-upgrade). Wait and read the release status.
• global.upgradeStrategy: replace on clusters that provision nodes on demand, so a surge upgrade does not trigger provisioning waves that outlive the timeout.

• Generated keys survive every upgrade and rollback (lookup-preserve, and the Secret is Job-created, never Helm-rendered).
• helm rollback xpander <N> restores any prior revision, except the database schema: migrations are forward-only, so a schema rollback means restore-from-backup.
• The one-day token lifetime (supabase.db.jwtExpiry) is re-applied by the migrations hook on every upgrade.

​

Automation & CI

• Every prompt has a flag. --non-interactive reads nothing from the terminal, exits 2 with the usage and a rerun line when a value is missing, and implies --yes and --on-probe-failure abort.
• Stable progress lines such as [3/8] license: ok. --quiet prints phases and the outcome only. Color only on a terminal.
• status is a read-only health report (edition, chart, release, hooks, readiness, URLs). --json emits one object.

bash install.sh --non-interactive --quiet \
  --edition airgap --release-manifest ./release-manifest.json \
  --org <ORG_ID> --install-license-file ./install.json --activation-license-file ./activation.json \
  --registry registry.example:5000/xpander \
  --org-name "Example Inc" --admin-email ops@example.com \
  --domain xpander.example.internal --storage-class gp3 \
  --on-probe-failure abort

until bash install.sh status --quiet; do sleep 30; done

​

Reference values file

global:
  upgradeStrategy: replace                  # in-place recreation; surge upgrades cascade into node waves on autoscaled clusters
  imageRegistry: <registry>/<mirror path>   # every image is pulled from the mirror
  license:
    secretName: xpander-license
  smtp:                                     # one block feeds sign-in mail, invitations and xpander's notifications
    host: email-smtp.<region>.amazonaws.com
    port: 587
    starttls: true
    user: <smtp user>
    passwordSecret: { name: xpander-smtp, key: password }
    adminEmail: admin@<domain>
    senderName: xpander
  assetStore: { publicUrl: https://auth.<domain> }
  shortLinks: { publicUrl: https://links.<domain> }
  # mcp.publicUrl unset: Platform > Agents > Channels shows the MCP row as "Not offered on this install" until you set it
  llm:
    defaults: { provider: amazon_bedrock, model: global.anthropic.claude-sonnet-5, apiBase: "", credentialsKey: "" }
    bedrock: { region: us-west-2 }          # may differ from the cluster's region
    gateway: { provider: amazon_bedrock, model: global.anthropic.claude-sonnet-4-6 }
  harness:
    defaultProviders: { claudeCode: amazon_bedrock }
    defaultModels: { claudeCode: global.anthropic.claude-sonnet-5 }
  fleet:
    replicas: 1
    storage: { createClass: true, provisioner: ebs.csi.eks.amazonaws.com }
  network:
    internalFetchAllowlist: [".internal.<domain>"]   # intranet hosts agents may fetch for knowledge ingest
  redis: { endpoint: <clustercfg endpoint>, clusterMode: true, auth: { username: <acl user> } }
redis: { mode: external }
postgres: { mode: external, external: { host: <rds host>, port: 5432 } }
supabase:
  ingress: { enabled: true, className: alb, host: auth.<domain> }
  auth: { apiExternalUrl: https://auth.<domain>, siteUrl: https://app.<domain> }
  storage:
    persistence: { size: 100Gi }
    publicUrl: https://auth.<domain>
  edgeRuntime:
    publicUrl: https://auth.<domain>
    functionsEnv:
      APP_URL: https://app.<domain>
      INBOUND_GW_URL: https://hooks.<domain>
      AGENT_CONTAINER_INGRESS_DOMAIN: <domain>
      DEPLOYMENT_MANAGER_PUBLIC_URL: https://dm.internal.<domain>
ui:
  ingress: { host: app.<domain>, className: alb }
  authProviders: [sso]
  env:
    VITE_WEBHOOK_URL: https://hooks.<domain>     # the app's baked default is the cloud host
    VITE_EMAIL_DOMAIN: agents.<domain>
    VITE_PLATFORM_URL: https://app.<domain>
    VITE_SUPABASE_URL: https://auth.<domain>
    VITE_AGENTS_API_URL: https://api.<domain>
    VITE_CHAT_BACKEND_URL: https://chat.<domain>/chat-backend/{agent_id}
    VITE_ACTIONS_SERVICE_URL: https://actions.<domain>
    VITE_CLIENT_AUTH_URL: https://client-auth.<domain>
    VITE_LICENSE_MODE: "true"
    VITE_OMNI_MCP_URL: ""
  extraEnv:
    - name: VITE_SUPABASE_ANON_KEY
      valueFrom: { secretKeyRef: { name: xpander-generated-keys, key: SUPABASE_ANON_KEY } }
client-auth: { ingress: { host: client-auth.<domain> }, env: { APP_URL: https://app.<domain>, CLIENT_AUTH_EXTERNAL_URL: https://client-auth.<domain> } }   # the second turns on MCP OAuth
api:
  ingress: { host: api.<domain> }
  env: { API_URL: https://api.<domain>, INBOUND_EMAIL_REPLY_DOMAIN: agents.<domain>, SMTP_HOST: <smtp host>, SMTP_PORT: "587", SMTP_STARTTLS: "true", SMTP_USERNAME: <smtp user> }
  extraEnv:
    - { name: SMTP_PASSWORD, valueFrom: { secretKeyRef: { name: xpander-smtp, key: password } } }
    - { name: INBOUND_EMAIL_TOKEN, valueFrom: { secretKeyRef: { name: xpander-inbound-mail, key: token } } }
    - { name: MAILGUN_SIGNING_KEY, valueFrom: { secretKeyRef: { name: xpander-inbound-mail, key: signing-key } } }
agent-controller: { ingress: { host: chat.<domain> } }   # API clients on this host add /<organization id> to the base URL
webhook: { ingress: { host: hooks.<domain> } }
functions: { ingress: { host: links.<domain> } }
deployment-manager: { ingress: { host: dm.internal.<domain>, className: alb-internal } }
actions:
  ingress: { host: actions.<domain> }
  env: { XPANDER_SANDBOX_DNS: <CoreDNS ClusterIP> }
catalog: { env: { OAS_GEN_BUCKET_NAME: <versioned bucket>, AWS_DEFAULT_REGION: <region> } }
openapi-spec-generator: { env: { OAS_GEN_BUCKET_NAME: <versioned bucket>, AWS_DEFAULT_REGION: <region> } }

​

CI/CD & GitOps

pipeline {
  agent { label 'xpander-admin' }        // has kubectl access to the cluster
  triggers { cron('H 6 * * 1') }         // or trigger from releases.atom
  environment {
    XPANDER_PULL = credentials('xpander-pull-token')  // username: xpanderaihub, password: the read token
    XPANDER_ORG  = credentials('xpander-org-id')      // the organization id from the license handoff
  }
  stages {
    stage('Mirror the release') {
      steps {
        sh '''
          curl -fsSLO https://charts.xpander.ai/image_sync.sh
          curl -fsSL https://github.com/xpander-ai/releases/releases/download/latest/release-manifest.json -o release-manifest.json
          bash image_sync.sh --source docker.io/xpanderaihub \
            --dest registry.corp.example:5000/xpander \
            --release-manifest ./release-manifest.json \
            --source-user "$XPANDER_PULL_USR" --source-token "$XPANDER_PULL_PSW"
        '''
      }
    }
    stage('Upgrade and verify') {
      steps {
        sh '''
          curl -fsSL https://charts.xpander.ai/install.sh -o install.sh
          bash install.sh --non-interactive --quiet --edition airgap \
            --org "$XPANDER_ORG" --install-license-file install.json --activation-license-file activation.json \
            --registry registry.corp.example:5000/xpander --mirror skip \
            --release-manifest ./release-manifest.json --values values.yaml \
            --helm-timeout 20m
          bash install.sh status --quiet
        '''
      }
    }
  }
}

helm upgrade --install xpander ./xpander-airgap-<X.Y.Z>.tgz \
  --namespace xpander --create-namespace \
  -f xpander/values.yaml -f xpander/values.yaml.release.json

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: xpander
  namespace: argocd
spec:
  project: default
  destination:
    server: https://kubernetes.default.svc
    namespace: xpander
  sources:
    - repoURL: registry.corp.example:5000/xpander/charts   # OCI repo credential with "Enable OCI"
      chart: xpander-airgap
      targetRevision: <the chart version you mirrored>     # keep it in step with the release values
      helm:
        valueFiles:
          - $values/xpander/values.yaml           # generated once: install.sh --values-out
          - $values/xpander/values.yaml.release.json  # generated release image pins, applied last
    - repoURL: https://git.corp.example/platform/xpander-config.git
      targetRevision: main
      ref: values
  syncPolicy:
    syncOptions:
      - CreateNamespace=true

​

Skill credentials on Air-Gapped

• RapidAPI skills (for example Google News, Company Insights and Social Media): set global.network.toolKeys.RAPID_API_KEY, or calls return 401 Invalid API key.
• Skills with their own sign-in (for example BigQuery and Figma): create the connection with valid credentials.
• AWS skills (Redshift, S3, Athena, EKS): the IAM role and its permissions on AWS Operator Setup.
• Public skills need internet egress, and do not work with global.network.internetEgress: false.

​

Backups

​

Monitoring

​

Troubleshooting

​

Mirroring and registries

writing blob: initiating layer upload to /v2/xpander/ai-gateway/blobs/uploads/ in
123456789012.dkr.ecr.eu-west-2.amazonaws.com: name unknown: The repository with
name 'xpander/ai-gateway' does not exist in the registry with id '123456789012'

curl -fsSLO https://charts.xpander.ai/image_sync.sh

aws ecr create-repository --repository-name xpander/ai-gateway --region <region>

bash image_sync.sh --source docker.io/xpanderaihub --dest <registry>/<path> \
  --dest-user <user> --dest-token <token>

✗ deployment-manager:0.0.417 not found in registry.example:5000/xpander

crane digest registry.example:5000/xpander/deployment-manager:0.0.417

​

Cluster and network

failed to pull image "docker.io/...": dial tcp 44.208.254.194:443: i/o timeout

kubectl -n xpander get pods -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.containers[*].image}{"\n"}{end}' | grep docker.io

✗ no node with a supported architecture: cluster has [arm64], the images need [amd64]

could not find any suitable subnets for creating the ELB

aws ec2 create-tags --resources <subnet-ids> \
  --tags Key=kubernetes.io/role/internal-elb,Value=1

metadata:
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-internal: "true"

Error from server (NotFound): namespaces "xpander" not found

kubectl create namespace xpander

Missing or invalid environment variables

kubectl -n xpander get ingress

​

Install and upgrade

helm history xpander -n xpander
helm rollback xpander <last-good-revision> -n xpander   # if a good revision exists
helm uninstall xpander -n xpander                        # first install that never converged

kubectl -n xpander create secret generic xpander-license \
  --from-file=install.json=./install.json \
  --from-file=activation.json=./activation-new.json \
  --dry-run=client -o yaml | kubectl -n xpander apply -f -

-- the connections of the faulty connector
update public.connector_organization set deleted_at = now()
  where connector_id = '<connector id>' and deleted_at is null;
-- take the connection id out of the agent's skill list
update public.ai_agents
  set attached_tools = (select coalesce(jsonb_agg(t), '[]'::jsonb)
    from jsonb_array_elements(attached_tools::jsonb) t
    where t->>'id' <> '<connection id>')::json
  where id = '<agent id>';

kubectl -n xpander logs job/<failed-job> --tail=100
kubectl -n xpander describe job/<failed-job>
kubectl -n xpander get pods,jobs
kubectl -n xpander get events --sort-by=.lastTimestamp

--set supabase.edgeRuntime.image.repository=null --set supabase.edgeRuntime.image.tag=null

​

Next steps