Hybrid deployment

curl -fsSL https://charts.xpander.ai/install.sh | bash -s -- \
  --edition hybrid --org <ORG> --env <ENV> --dm-api-key <KEY>

​

Architecture

​

What gets deployed

​

How it works

​

What talks to what

​

Data residency

​

Requirements

​

Install

​

1. Get your one-liner from xpander Chat

1. In xpander Chat, open Platform > Agents > Runtime and choose the Locations tab (Connected runtimes and regions)
2. Click “Add Location”
3. Choose the Hybrid edition. The dialog shows the minimum footprint for the deployment you picked.
4. Optionally set a URL (your server’s FQDN, e.g. xpander.my-company.com) and the hostname style (dash-based subdomains if your DNS can’t do wildcards)
5. Copy the generated one-liner

curl -fsSL https://charts.xpander.ai/install.sh | bash -s -- \
  --edition hybrid \
  --org <ORGANIZATION_UUID> \
  --env <ENVIRONMENT_UUID> \
  --dm-api-key <DEPLOYMENT_MANAGER_API_KEY> \
  --domain xpander.my-company.com

​

2. Run the installer

​

Install phases

[1/6] preflight       helm ≥ 3.13, kubectl, cluster reachable, Kubernetes ≥ 1.28
[2/6] configuration   context confirmation + masked summary
[3/6] chart-source    helm repo add xpander https://charts.xpander.ai (or --chart-path)
[4/6] values          secrets written to a mode-600 temp file, removed on exit
[5/6] install         helm upgrade --install xpander xpander/xpander
[6/6] verify          waits for the agent-controller rollout (5-minute timeout)

​

Amazon Bedrock: keyless IAM or API key

​

3. Verify the connection

kubectl get pods -n xpander

kubectl -n xpander port-forward service/xpander-agent-controller 9016:9016 &
kubectl -n xpander port-forward service/xpander-ai-gateway 9018:9018 &
kubectl -n xpander port-forward service/xpander-mcp 8081:8081 &

curl http://localhost:9016/health
curl http://localhost:9018/health
curl http://localhost:8081/health

​

PrivateLink

​

How PrivateLink works

​

Set it up

aws ec2 create-vpc-endpoint \
  --vpc-id <VPC_ID> \
  --vpc-endpoint-type Interface \
  --service-name com.amazonaws.vpce.us-west-2.vpce-svc-0101884b32f655197 \
  --service-region us-west-2 \
  --subnet-ids <PRIVATE_SUBNET_A_ID> <PRIVATE_SUBNET_B_ID> \
  --no-private-dns-enabled \
  --tag-specifications 'ResourceType=vpc-endpoint,Tags=[{Key=Name,Value=xpander-privatelink}]' \
  --region <REGION> --profile <PROFILE>

aws ec2 authorize-security-group-ingress \
  --group-id <ENDPOINT_SG_ID> \
  --protocol tcp \
  --port 443 \
  --cidr <VPC_CIDR> \
  --region <REGION> --profile <PROFILE>

aws ec2 describe-vpc-endpoints \
  --vpc-endpoint-ids <VPCE_ID> \
  --query 'VpcEndpoints[0].State' \
  --region <REGION> --profile <PROFILE>

aws route53 create-hosted-zone \
  --name deployment-manager-privatelink.xpander.ai \
  --caller-reference "xpander-privatelink-$(date +%s)" \
  --vpc VPCRegion=<REGION>,VPCId=<VPC_ID> \
  --hosted-zone-config PrivateZone=true \
  --profile <PROFILE>

cat <<EOF > /tmp/privatelink-dns.json
{
  "Changes": [{
    "Action": "UPSERT",
    "ResourceRecordSet": {
      "Name": "deployment-manager-privatelink.xpander.ai.",
      "Type": "A",
      "AliasTarget": {
        "DNSName": "<VPCE_DNS_NAME>",
        "HostedZoneId": "<VPCE_HOSTED_ZONE_ID>",
        "EvaluateTargetHealth": true
      }
    }
  }]
}
EOF

aws route53 change-resource-record-sets \
  --hosted-zone-id <PRIVATE_ZONE_ID> \
  --change-batch file:///tmp/privatelink-dns.json \
  --profile <PROFILE>

kubectl run curl-test --restart=Never --image=curlimages/curl:latest -n default \
  --command -- sh -c "curl -sk -o /dev/null -w '%{http_code}' \
  --connect-timeout 10 https://deployment-manager-privatelink.xpander.ai/health; echo"

sleep 15 && kubectl logs curl-test
# Expected output: 200

kubectl delete pod curl-test

​

External access & DNS

​

Option A: VPN / corporate network access

chat.<DOMAIN>              → xpander-chat.xpander.svc.cluster.local
agent-controller.<DOMAIN>  → xpander-agent-controller.xpander.svc.cluster.local
ai-gateway.<DOMAIN>        → xpander-ai-gateway.xpander.svc.cluster.local

​

Option B: Public access via ingress + NLB

​

SSL certificate

aws acm request-certificate \
  --domain-name "<DOMAIN>" \
  --subject-alternative-names "*.<DOMAIN>" "*.chat.<DOMAIN>" \
  --validation-method DNS \
  --region <REGION> --profile <PROFILE>

aws acm describe-certificate --certificate-arn <CERT_ARN> \
  --region <REGION> --profile <PROFILE> \
  --query 'Certificate.DomainValidationOptions[0].ResourceRecord'

aws acm wait certificate-validated --certificate-arn <CERT_ARN> \
  --region <REGION> --profile <PROFILE>

​

Ingress controller

helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update

helm upgrade --install ingress-nginx ingress-nginx/ingress-nginx \
  --namespace ingress-nginx --create-namespace \
  --set controller.service.type=LoadBalancer \
  --set controller.service.annotations."service\.beta\.kubernetes\.io/aws-load-balancer-type"=nlb \
  --set controller.service.annotations."service\.beta\.kubernetes\.io/aws-load-balancer-scheme"=internet-facing \
  --set controller.service.annotations."service\.beta\.kubernetes\.io/aws-load-balancer-ssl-cert"="<ACM_CERT_ARN>" \
  --set controller.service.annotations."service\.beta\.kubernetes\.io/aws-load-balancer-ssl-ports"="443" \
  --set controller.service.annotations."service\.beta\.kubernetes\.io/aws-load-balancer-backend-protocol"="tcp" \
  --set controller.service.targetPorts.https=http

kubectl get svc -n ingress-nginx ingress-nginx-controller \
  -o jsonpath='{.status.loadBalancer.ingress[0].hostname}'

​

DNS records

cat <<EOF > /tmp/dns-record.json
{
  "Changes": [
    {
      "Action": "UPSERT",
      "ResourceRecordSet": {
        "Name": "*.<DOMAIN>.",
        "Type": "CNAME",
        "TTL": 300,
        "ResourceRecords": [{ "Value": "<NLB_HOSTNAME>" }]
      }
    },
    {
      "Action": "UPSERT",
      "ResourceRecordSet": {
        "Name": "*.chat.<DOMAIN>.",
        "Type": "CNAME",
        "TTL": 300,
        "ResourceRecords": [{ "Value": "<NLB_HOSTNAME>" }]
      }
    }
  ]
}
EOF

aws route53 change-resource-record-sets \
  --hosted-zone-id <HOSTED_ZONE_ID> \
  --change-batch file:///tmp/dns-record.json \
  --profile <PROFILE>

curl -sk https://api.<DOMAIN>/health

​

LLM Gateway

spec:
  ignoreDifferences:
    - kind: Secret
      name: xpander-llm-gw
      jsonPointers: ["/data"]
    - kind: Secret
      name: xpander-llm-gw-content
      jsonPointers: ["/data"]

kubectl -n <namespace> get pods -l app=llm-gw

llm-gw:
  networkPolicy:
    providerCidrs: []        # model endpoints on your own network, any port
    cloudEndpointCidrs: []   # private cloud endpoints such as Bedrock or Azure, port 443
    redisCidrs: []           # an external Redis
  env:                       # the gateway does not read global.env
    HTTPS_PROXY: http://<proxy-host>:<port>
    NO_PROXY: <hosts that bypass the proxy>

​

Managing LLM API keys

helm upgrade xpander xpander/xpander \
  --namespace xpander --reset-then-reuse-values \
  --set secrets.static.anthropicApiKey=<KEY> \
  --timeout 5m

helm upgrade xpander xpander/xpander \
  --namespace xpander --reset-then-reuse-values \
  --set secrets.static.anthropicApiKey=<KEY> \
  --set secrets.static.agentsOpenaiApiKey=<KEY> \
  --set secrets.static.fireworksApiKey=<KEY> \
  --timeout 5m

​

Your own keys and credits

# Set the key directly in the Kubernetes secret
kubectl patch secret xpander-static -n xpander --type=merge \
  -p "{\"data\":{\"ANTHROPIC_API_KEY\":\"$(echo -n '<YOUR_KEY>' | base64)\"}}"

# Restart the worker to pick up the new key
kubectl rollout restart deployment xpander-agent-worker -n xpander

​

Key names read from an environment variable

• names starting with AWS_ (except AWS_BEARER_TOKEN_BEDROCK), XPANDER_, FLEET_, LLM_GW, LLM_GATEWAY, LITELLM_, GITHUB_APP, SUPABASE, REDIS, OAUTH, JWT, LICENSE, MONGO, POSTGRES, PG_, ON_PREM_PG, SMTP, REGISTRY, SANDBOX_, HARNESS_ or KUBE
• names ending with _SERVICE_API_KEY, _PASSWORD, _PRIVATE_KEY, _SECRET, _SECRET_KEY, _SESSION_TOKEN or _TOKEN_FILE
• the platform’s own service keys (DEPLOYMENT_MANAGER_API_KEY, AGENT_CONTROLLER_API_KEY and the like)

​

Using Kubernetes Secrets (recommended for production)

kubectl create secret generic ai-service-keys \
  --namespace xpander \
  --from-literal=openai-api-key=sk-your-openai-key \
  --from-literal=anthropic-api-key=sk-ant-your-anthropic-key

# In your values file
agent-worker:
  envFromSecretKeys:
    AGENTS_OPENAI_API_KEY:
      secretName: "ai-service-keys"
      key: "openai-api-key"
    ANTHROPIC_API_KEY:
      secretName: "ai-service-keys"
      key: "anthropic-api-key"

​

Upgrading

curl -fsSL https://charts.xpander.ai/install.sh | bash -s -- upgrade
# or, pinned:
curl -fsSL https://charts.xpander.ai/install.sh | bash -s -- upgrade --chart-version <X.Y.Z>

• finds the release (default xpander in namespace xpander, override with --release / --namespace),
• infers its edition from the installed chart: you never pass --edition to an upgrade,
• reuses the values the release was installed with (helm get values), plus anything extra you pass (--set, --values),
• shows current → target version with a link to the release notes, and asks before changing anything,
• always runs with --atomic: a failed upgrade (including a failed hook) rolls back automatically and the previous revision keeps serving.

helm repo update
helm upgrade xpander xpander/xpander --namespace xpander --reset-then-reuse-values

​

Upgrading to the current chart

helm repo add xpander https://charts.xpander.ai && helm repo update xpander
helm upgrade xpander xpander/xpander --version 0.22.5 -n <namespace> --reset-then-reuse-values --wait --timeout 30m

• In-cluster Postgres moves to the pgvector build (0.22.1). The postgres StatefulSet runs pgvector/pgvector:0.8.7-pg15-trixie instead of postgres:15: PostgreSQL 15.19, the same major version, on the same volume. If you use global.imageRegistry, mirror docker.io/pgvector/pgvector:0.8.7-pg15-trixie first. Take a volume snapshot of postgres-data-xpander-postgres-0, or set postgres.upgradeGuard.backup.pvcName to an existing PVC with room for a full dump. Postgres restarts once. If you already run 0.22.1 or later, Postgres is on pgvector and this step is done.
• A pinned Postgres image stays on pgvector (0.22.1). If your values set images.postgres, keep it on a pgvector build of PostgreSQL 15. A pre-upgrade check refuses the upgrade when the major version would change.
• A new key in the generated Secret (0.22.3). BRAIN_INGEST_API_KEY is added automatically and kept from then on. No action needed.
• Environment from your own Secrets and ConfigMaps renders (0.22.4). global.envFromSecret, global.envFromConfigMap and the per-component envFromSecret and envFromConfigMap lists now render as the container’s envFrom. Earlier charts failed the render when they were set. Installs that never set them render the same as before.
• The LLM Gateway is always on (0.22.5). The chart ignores llm-gw.enabled. If your install had llm-gw.enabled: false, mirror the llm-gw image if you mirror images, make sure every model provider your agents use has a key, and open the gateway’s network paths (LLM Gateway). The upgrade starts the gateway pods and restarts agent-controller and api once. Then check kubectl -n <namespace> get pods -l app=llm-gw. If the gateway was already on, nothing changes.
• Per-agent MCP URLs move to the MCP host (release change, no chart change). MCP clients that used a per-agent MCP URL connect to https://mcp.<DOMAIN> instead. Per-agent MCP URLs are no longer served. The API’s own MCP endpoint, https://api.<DOMAIN>/mcp/ (Model Context Protocol), does not change, so clients set up on it keep working. The chart has served mcp.<DOMAIN> from its ingress since 0.22.0. There is no value to set.

kubectl -n <namespace> logs job/xpander-postgres-pgvector

• The LLM Gateway runs. Before you upgrade, mirror the llm-gw image if you mirror images, make sure every provider your agents use has a key, and open any network path the gateway needs (LLM Gateway). agent-controller and api restart once to pick up the gateway’s address. After the upgrade, check kubectl -n <namespace> get pods -l app=llm-gw (healthy result).
• Argo CD. Add ignoreDifferences on /data for xpander-llm-gw and xpander-llm-gw-content, or set llm-gw.masterKey.existingSecret and llm-gw.contentKey.existingSecret.
• Harness agents run only on the fleet executors. There is no pod per agent. When no executor is up, a turn waits in the queue. The chart always keeps at least one executor: fleet.replicas: 0 (or blank) renders one, and with fleet.keda.enabled, fleet.keda.maxReplicas: 0 holds one. agentContainers.harness (enabled, resources, storage, shm) is ignored and can be removed from your values. images.agentHarness stays. It is the image every executor runs.
• The release Role reads pods but no longer writes them. agent-controller keeps get, list and watch on pods and still manages the Deployments, Services, volumes, Secrets and Jobs of agent workspaces. If your own automation used the release ServiceAccount to create, update or delete pods, add the rule back:
  rbac:
    additionalRules:
      - apiGroups: [""]
        resources: ["pods"]
        verbs: ["create", "update", "patch", "delete"]
  
• Fleet agents reach private networks like any other pod. fleet.networkPolicy.exceptCidrs now defaults to ["169.254.0.0/16"] and exceptCidrsV6 to ["fe80::/10"], so agents and the MCP endpoints they call reach 10/8, 172.16/12, 192.168/16 and in-cluster services. Cloud metadata stays blocked. --reset-then-reuse-values picks this up. With --reuse-values, pass --set-json 'fleet.networkPolicy.exceptCidrs=["169.254.0.0/16"]' --set-json 'fleet.networkPolicy.exceptCidrsV6=["fe80::/10"]'. An install that set exceptCidrs itself keeps its own list.
• Executor nodes need Landlock, or fleet.landlock.enabled: false. See Node prerequisites.
• Model key names and endpoints. Environment-variable key names that match the install’s own credentials read as no key, and model endpoints on metadata addresses are refused (details). Check the key names on your agents, custom providers and the organization’s default model before you upgrade.
• Custom-container agents move to the standard agent runtime during the upgrade.

helm upgrade xpander xpander/xpander --namespace xpander --reuse-values \
  --set-json 'fleet.networkPolicy.exceptCidrs=["169.254.0.0/16"]' \
  --set-json 'fleet.networkPolicy.exceptCidrsV6=["fe80::/10"]'

• PostgreSQL no longer restarts on every chart release. The config checksum hashes the ConfigMap data only, so a release whose Postgres configuration is byte-identical leaves the StatefulSet alone. A real postgres.* change still rolls it.
• The fleet is always rendered, and --atomic waits for it. A running turn holds the roll for up to the drain window (fleet.executor.drainSeconds, 3600 s). The idle guard holds the executor at zero replicas on a cluster without its StorageClass, so the upgrade completes anyway. If you drive helm upgrade --wait by hand, give --timeout at least the drain window or upgrade in a quiet period. Details on Sizing and requirements.

​

Installer flags reference

bash install.sh [flags]            # install (the wizard, then helm upgrade --install)
bash install.sh upgrade [flags]    # upgrade an existing release in place
bash install.sh status [flags]     # read-only health report (--json for one object)

​

Automation & CI

• Every prompt has a flag. With --non-interactive nothing is read from the terminal, even when one exists: missing required values exit 2 with the usage and a completed rerun line rather than blocking. It implies --yes.
• Stable progress output. Progress is one line per transition in a grep-friendly shape, [3/6] chart-source: ok, and --quiet reduces output to phase lines and the outcome only. Color is emitted only when stdout is a terminal.
• status is the poll target. A read-only health report: edition, chart, release status, hook jobs, workload readiness, and URLs. --json emits one JSON object.

bash install.sh --non-interactive --quiet \
  --edition hybrid --org <uuid> --env <uuid> --dm-api-key <key> \
  --domain xpander.example.com --openai-key <key>

until bash install.sh status --quiet; do sleep 30; done

​

Advanced: Review-first / manual install

​

Read the installer before running it

curl -fsSL https://charts.xpander.ai/install.sh | less
curl -fsSL https://charts.xpander.ai/install-v<version>.sh | less   # this exact version

• The versioned URL is immutable: install-v<version>.sh is published once per release and never rewritten in place, so pin it in runbooks and CI (bash install.sh --version prints the version you’re holding).
• The script is shellcheck-clean, enforced by CI on every push.
• Secrets never touch the helm command line: they are written to a temporary values file (mode 600, removed on exit), so they don’t land in shell history or process listings.
• Nothing runs against a cluster before the kube-context confirmation: the installer names the context and cluster it is about to install into and waits for your yes.

​

Do it yourself

bash install.sh --edition hybrid --org <ORG> --env <ENV> --dm-api-key <KEY> --manual

bash install.sh --edition hybrid --org <ORG> --env <ENV> --dm-api-key <KEY> \
  --domain <DOMAIN> --values-out xpander-values.yaml
helm upgrade --install xpander xpander/xpander \
  --namespace xpander --create-namespace -f xpander-values.yaml

helm repo add xpander https://charts.xpander.ai
helm repo update

helm upgrade --install xpander xpander/xpander \
  --namespace xpander --create-namespace \
  --set ingress.enabled=true \
  --set domain=<DOMAIN> \
  --set global.organizationId=<ORGANIZATION_ID> \
  --set global.environmentId=<ENVIRONMENT_ID> \
  --set secrets.static.deploymentManagerApiKey=<DEPLOYMENT_MANAGER_API_KEY> \
  --set urls.deploymentManager=<DEPLOYMENT_MANAGER_URL> \
  --timeout 10m

domain: "xpander.production.com"

global:
  organizationId: "your-org-id"
  environmentId: "your-env-id"
  env:
    LOG_LEVEL: "info"
    ENVIRONMENT: "production"

urls:
  deploymentManager: "https://deployment-manager.xpander.ai"

secrets:
  static:
    deploymentManagerApiKey: "your-key"
    agentsOpenaiApiKey: "sk-your-openai-key"
    anthropicApiKey: "sk-ant-your-anthropic-key"

ingress:
  enabled: true
  tls:
    enabled: true
    source: "cert-manager"
  annotations:
    cert-manager.io/cluster-issuer: "letsencrypt-prod"
    nginx.ingress.kubernetes.io/ssl-redirect: "true"

resources:
  agentController:
    requests:
      cpu: "1000m"
      memory: "2Gi"
    limits:
      cpu: "3000m"
      memory: "4Gi"

redis:
  storage:
    size: "32Gi"
    storageClass: "fast-ssd"

helm upgrade --install xpander xpander/xpander \
  --namespace xpander --create-namespace \
  --values xpander-values.yaml

​

Troubleshooting

​

Installer

helm history xpander -n xpander
helm rollback xpander <last-good-revision> -n xpander   # if a good revision exists
helm uninstall xpander -n xpander                        # first install that never converged

​

Cluster and pods

kubectl logs -n xpander deployment/xpander-agent-controller

kubectl -n xpander get pvc
kubectl -n xpander describe pvc

kubectl get storageclass  # Should show a default (gp3)
kubectl get pods -n kube-system -l app.kubernetes.io/name=aws-ebs-csi-driver  # Should show Running

kubectl patch deployment xpander-agent-worker -n xpander \
  --type='json' \
  -p='[{"op":"replace","path":"/spec/template/spec/containers/0/resources/requests/cpu","value":"500m"}]'

kubectl -n xpander logs deployment/xpander-agent-controller
kubectl -n xpander logs deployment/xpander-agent-worker
kubectl -n xpander logs deployment/xpander-mcp

​

PrivateLink

aws ec2 create-vpc-endpoint \
  --service-name com.amazonaws.vpce.us-west-2.vpce-svc-0101884b32f655197 \
  --service-region us-west-2 \
  ...

aws ec2 authorize-security-group-ingress \
  --group-id <ENDPOINT_SG_ID> \
  --protocol tcp --port 443 --cidr <VPC_CIDR> \
  --region <REGION> --profile <PROFILE>

​

Ingress and networking

kubectl -n xpander describe ingress
kubectl -n xpander get ingress

kubectl get svc -n ingress-nginx ingress-nginx-controller \
  -o jsonpath='{.status.loadBalancer.ingress[0].hostname}'

{
  "Effect": "Allow",
  "Principal": { "Service": "eks.amazonaws.com" },
  "Action": ["sts:AssumeRole", "sts:TagSession"]
}

aws acm request-certificate \
  --domain-name "<DOMAIN>" \
  --subject-alternative-names "*.<DOMAIN>" "*.chat.<DOMAIN>" \
  --validation-method DNS \
  --region <REGION> --profile <PROFILE>

​

API keys

kubectl patch secret xpander-static -n xpander --type=merge \
  -p "{\"data\":{\"ANTHROPIC_API_KEY\":\"$(echo -n '<YOUR_KEY>' | base64)\"}}"

kubectl rollout restart deployment xpander-agent-worker -n xpander

kubectl -n <namespace> get pods -l app=llm-gw
kubectl -n <namespace> describe pods -l app=llm-gw

kubectl -n xpander exec deployment/xpander-agent-worker -- env | grep API_KEY

​

Debug commands

# Get all resources
kubectl -n xpander get all

# Check events
kubectl -n xpander get events --sort-by=.metadata.creationTimestamp

# Describe problematic pods
kubectl -n xpander describe pod <pod-name>

# Check service endpoints
kubectl -n xpander get endpoints

​

Uninstalling

# Remove the Helm release
helm uninstall xpander --namespace xpander

# Clean up the namespace (optional)
kubectl delete namespace xpander

​

Next steps