Hybrid Edition - xpander.ai

The Hybrid edition: your data and agents run in your cluster, control plane in xpander cloud. All execution, runtime data, credentials, and model keys stay inside your VPC; xpander cloud manages agent definitions, identity, and the connector catalog. This page is the complete guide — from architecture review to a running, connected location.

curl -fsSL https://charts.xpander.ai/install.sh | bash -s -- \
  --edition hybrid --org <ORG> --env <ENV> --dm-api-key <KEY>

Self-hosted locations are unlocked per account — contact our team if the option isn’t available on yours.

Comparing editions first? See the Self-Hosted overview. The full-platform edition with no internet access is Air-Gapped.

Architecture

Hybrid follows the control plane / data plane separation pattern — the same model used by AWS EKS, Datadog, and other infrastructure platforms:

Control plane (xpander Cloud) — manages environment metadata, agent registration, connector definitions, and event logging. AI engineers use the Agent Workbench to define agents, configure connectors, and validate tasks. The control plane never touches runtime data.
Data plane (your VPC) — runs all task execution, agent memory, connector credentials, model API keys, and user-facing services. The user’s browser connects directly to the data plane. Chat threads, activity views, and task results are served from the cluster — nothing passes through the cloud.

The data plane connects outbound only to the control plane over port 443 for metadata sync and heartbeats. The control plane never initiates connections into your cluster. MCP and external-system calls run from your cluster, and agent workspaces — the agents’ computers — run there too.

What Gets Deployed

Component	Description
Agent Controller	Task scheduling, queue management, and orchestration
Agent Worker	Task execution runtime
API Proxy	API endpoint and request routing
API	REST API service
MCP	Accepts MCP client connections, generates tasks
Database	Agent memory, message history, and monitoring (PostgreSQL)
Redis	Cache and fast lookups
Chat UI	Chat interface with per-thread subdomains
Code Runner	Code execution runtime
AWS APIs Operator	AWS resource management

How It Works

Clients — Slack, the web UI, MCP clients, the SDK and API — connect directly to the cluster in your VPC. When using app.xpander.ai, the cloud UI handles authentication and tells the client which self-hosted cluster to connect to; all data then flows client-to-cluster. Inside the cluster, the Agent Controller schedules tasks and Agent Workers execute them — calling LLMs, persisting messages in PostgreSQL, and reaching external tools exclusively through the AI Gateway. The only connection to xpander Cloud is the deployment manager sync: the Agent Controller authenticates with organization_id + environment_id + api_key, reports heartbeats, and pulls agent, connector, and MCP configuration.

What Talks to What

The egress surface — the answers a security review needs. Inbound from xpander cloud: none; connections are outbound-only.

From your cluster	To	Port	Purpose
agent-controller	`deployment-manager.xpander.ai` (or the PrivateLink / staging endpoint)	443	Metadata sync, heartbeats, connector updates
agent-controller, agent-worker	Your LLM providers (OpenAI, Anthropic, Google, Bedrock, …)	443	Model calls
Agent tools, MCP, connectors	The external systems you connect	As configured	Run from your cluster, never from xpander cloud

Outbound connectivity options to the control plane:

Option	Target	Details
Global Accelerator	`15.197.85.80`, `166.117.85.46`	Public internet, encrypted over TLS
PrivateLink	`com.amazonaws.vpce.us-west-2.vpce-svc-0101884b32f655197`	AWS private backbone — setup below

Data Residency

Feature	Stays in your VPC?
Invoking agents (API to agent)	Yes — client connects directly to cluster
Chat threads and activity view	Yes — fetched from cluster, not cloud
Debugging agents (Monitor tab)	Yes — all data served from cluster
Knowledge Base file uploads and queries	Yes — stored and queried in cluster
User identity and login	No
Agent metadata (agent list)	No

Data at rest stays in PostgreSQL and Redis inside your VPC; credentials and API keys are Kubernetes secrets; the control plane sync carries metadata and heartbeats only — user payloads never leave the VPC.

Requirements & Sizing

Minimum footprint (the installer shows the same numbers, published at charts.xpander.ai/requirements.json):

	Hybrid minimum
Nodes	3 × 8 vCPU / 32 GiB
Storage	~50 GiB PVC
Kubernetes	≥ 1.28
Architecture	amd64
Network	Egress to `deployment-manager.xpander.ai` (or PrivateLink) and to your LLM providers
Extras	Ingress controller + wildcard DNS optional

xpander container images are amd64 only. Do not use ARM/Graviton instances (t4g, m7g, c7g, etc.).

Need a cluster? Follow the EKS Cluster Setup guide.

Example Estimates

Agents are autonomous — tasks come from schedules, webhooks, the SDK and API, and other agents, not only chat (in xpander cloud production, only ~5% of executions are interactive chat). Size for the number of tasks running concurrently at peak, whatever triggers them.

Concurrent tasks	Agents	Cluster capacity	Example nodes	Persistent storage
3	10	10 vCPU / 19 GiB	3 × `m5.xlarge` (or 2 × `t3.xlarge` for evaluation)	~110 GiB
10	25	19 vCPU / 34 GiB	3 × `m5.2xlarge`	~180 GiB
20	50	29 vCPU / 49 GiB	4 × `m5.2xlarge`	~310 GiB
50	100	66 vCPU / 105 GiB	9 × `m5.2xlarge`, enable autoscaling	~570 GiB

Capacity figures include ~50% headroom over pod resource requests for burst load, Kubernetes system pods, and node upgrades. Use the calculator below to size for your exact numbers.

Sizing Calculator

How the estimate works

The calculator applies the same model we use to size xpander cloud:

Concurrent tasks is the primary driver. A task is any agent execution, from any source: chat, API/SDK calls, schedules, webhooks, MCP clients, Slack, or another agent. If you don’t know your peak concurrency yet, assume one concurrent task per agent - that covers most production organizations we measured. In practice most teams peak at just a few concurrent tasks (typically under 5), and even the busiest organization peaks around 30.
Agent Workers = concurrent tasks ÷ 4 (each worker replica processes 4 tasks by default, configurable with MAX_CONCURRENT_EXECUTIONS). Each replica requests 2 vCPU / 2.5 GiB.
Agent Controllers: 1 replica up to ~8 concurrent tasks, 2 up to ~20, 3 beyond. Each requests 1 vCPU / 2 GiB.
Active workspaces = concurrent tasks × workspace share, capped at the agent count. Budget ~0.5 vCPU / 1 GiB per active workspace (measured typical load; each can burst to its 2 vCPU / 3 GiB limit).
Fixed base (~2 vCPU / 6 GiB): AI Gateway, MCP, Chat, Code Runner, AWS Operator, API, Redis, PostgreSQL, container registry, metrics-server at one replica each.
Cluster capacity = requests × 1.5 for burst, DaemonSets, and safe node drains. Storage = 48 GiB of chart volumes + 5 GiB per agent (each agent that uses a workspace keeps a persistent volume) + image cache per node.

Per-service requirements

Per-replica values. Defaults are what the Helm chart applies out of the box; recommended raises the values we found low against measured production usage.

Service	CPU request	Memory request	CPU limit	Memory limit	Notes
Agent Controller	1000m	2 Gi	3000m	4 Gi	Orchestrator; observed steady-state memory is ~1.5-2 GiB, and the limit carries headroom for peaks
Agent Worker	2000m	2.25 Gi → 2.5 Gi	4000m	3 Gi	Scale replicas for task concurrency, not size
API	100m → 250m	256 Mi → 1 Gi	- → 2000m	- → 1.5 Gi	Chart sets no limits; observed usage ~0.9-1.4 GiB
MCP	200m	750 Mi	400m	1 Gi
Chat	300m	1 Gi	600m	1.5 Gi
AI Gateway	200m	400 Mi	400m	600 Mi
Code Runner	400m	800 Mi	800m	1229 Mi	Runs privileged (Docker-in-Docker); each concurrent code execution adds up to 1 vCPU / 2 GiB
AWS Operator	200m	400 Mi	400m	600 Mi	Optional component

Override any of these in your Helm values (keys are camelCase):

resources:
  agentController:
    requests:
      cpu: "1000m"
      memory: "2Gi"
    limits:
      cpu: "3000m"
      memory: "4Gi"

Horizontal Pod Autoscaling is available for Agent Controller, AI Gateway, Agent Worker, and MCP via the chart’s autoscaling values. The chart installs metrics-server by default to support it.

Agent workspaces

Each agent that uses workspace tools gets a dedicated pod, created on demand and reclaimed when idle. Size the cluster for the number of concurrently active workspaces, not total agents.

Resource	Value
CPU	250m request, 2000m limit
Memory	256 Mi request, 3 Gi limit
Storage	5 GiB persistent volume per agent
Idle shutdown	5 minutes without activity (configurable via `SANDBOX_IDLE_TIMEOUT` on the Agent Controller)
Maximum lifetime	3 hours, then the pod is recycled

Idle workspaces consume almost nothing (~50 Mi observed); an active workspace typically uses ~0.5-1 vCPU and up to 1 GiB, bursting to its limits for heavy work. The workspace volume persists after the pod is reclaimed, so a returning agent keeps its files - plan 5 GiB of storage per workspace-using agent, not per pod.

The workspace image is ~1 GiB. The first workspace start on each node pulls it; subsequent starts on that node are fast. Budget ~2 GiB of node disk for image cache.

Data services, storage, and the production checklist

The chart deploys single-replica Redis and PostgreSQL StatefulSets and a container registry alongside the platform:

Component	Persistent volume (default)	Guidance
Redis	8 GiB	Queues, caches, and runtime state. 2-4 GiB of memory is sufficient for a single organization; give the pod a 512 Mi-1 GiB memory request
PostgreSQL	20 GiB	Platform state; grows slowly
Container registry	20 GiB	Stores agent container images built in your cluster

Baseline storage: 48 GiB of persistent volumes, plus 5 GiB per workspace-using agent (see above). All volumes use your cluster’s default StorageClass unless overridden.Production checklist:

Node types: x86_64 only. General-purpose instances with a 1:4 vCPU:GiB ratio (m5, m6i) fit the workload profile; t3 burstable instances are fine for evaluation, not for production.
Raise the low defaults: API memory (table above) before real load.
Enable autoscaling for Agent Worker once concurrency grows beyond a couple of replicas.
Keep headroom: pod requests should stay under ~65% of cluster capacity so node upgrades and bursts don’t evict workloads.
Storage class: SSD-backed volumes for Redis and PostgreSQL.

Install

1. Get Your One-Liner from the Console

Go to app.xpander.ai/admin_settings#locations
Click “Add Location”
Choose the Hybrid edition. The dialog shows the minimum footprint for the edition you picked.
Optionally set a URL (your server’s FQDN, e.g. xpander.my-company.com) and the hostname style (dash-based subdomains if your DNS can’t do wildcards)
Copy the generated one-liner

The one-liner carries your identity — --org, --env, and --dm-api-key (plus --domain when you set a URL). That is how the cluster authenticates to the xpander control plane:

curl -fsSL https://charts.xpander.ai/install.sh | bash -s -- \
  --edition hybrid \
  --org <ORGANIZATION_UUID> \
  --env <ENVIRONMENT_UUID> \
  --dm-api-key <DEPLOYMENT_MANAGER_API_KEY> \
  --domain xpander.my-company.com

The one-liner deliberately carries no LLM provider credentials — the wizard prompts for those interactively (input hidden), or you pass them as flags. Secrets are written to a temporary values file (mode 600, removed on exit) and never appear on the helm command line.

2. Run the Installer

Run it from a terminal with kubectl access to the target cluster. The command downloads the installer from charts.xpander.ai and runs an interactive wizard — preflight checks, configuration, Helm install, then the connection back to xpander. Run bare (curl -fsSL https://charts.xpander.ai/install.sh | bash), it is the zero-knowledge entry point: the first question is which edition you are installing. The wizard walks through, in order:

Edition & requirements

If --edition wasn’t given, it asks. It then prints the minimum footprint for that edition (the same numbers as charts.xpander.ai/requirements.json) and asks you to confirm your cluster meets it — stopping cleanly if not.

Identity

Organization ID, Environment ID, and the Deployment Manager API key (input hidden) — all from the console’s install card, all skipped if passed as flags.

Ingress domain (optional)

Press Enter to skip — services then keep cluster-internal Services only, which is fine for a first install (port-forward reaches them; add ingress later). With a domain, it asks how to build hostnames: subdomain style (agent-controller.xpander.<DOMAIN>, needs wildcard DNS) or dash style (agent-controller-xpander.<DOMAIN>, individual A records).

Control plane reachability

Public internet (https://deployment-manager.xpander.ai), AWS PrivateLink (https://deployment-manager-privatelink.xpander.ai), or Staging. The installer auto-detects staging/local environments (a .stg. or local domain, or a kind/k3d/minikube/docker-desktop kube context) and leads with that option; any of --staging, --privatelink, or --dm-url overrides detection.

LLM providers (all optional)

OpenAI, Anthropic, Google/Gemini, Amazon Bedrock, then optionally Fireworks, Nebius, Helicone, and OpenRouter. Press Enter to skip any — the platform installs and starts without keys, and model calls light up as soon as you add a provider key in the UI or with a helm upgrade.For Bedrock it asks how to authenticate: an API key / bearer token (AWS_BEARER_TOKEN_BEDROCK), or keyless IAM role / EKS pod identity. See the caveat below before choosing IAM.

Namespace, context confirmation, summary

Namespace (default xpander, created if missing), an explicit confirmation of the kubectl context it is about to install into, and a masked configuration summary before anything changes.

Install Phases

The Hybrid install runs six phases, each announced as a stable [N/6] name: ... progress line (the same shape scripts can grep — see Automation & CI):

[1/6] preflight       helm ≥ 3.13, kubectl, cluster reachable, Kubernetes ≥ 1.28
[2/6] configuration   context confirmation + masked summary
[3/6] chart-source    helm repo add xpander https://charts.xpander.ai (or --chart-path)
[4/6] values          secrets written to a mode-600 temp file, removed on exit
[5/6] install         helm upgrade --install xpander xpander/xpander
[6/6] verify          waits for the agent-controller rollout (5-minute timeout)

Every failure prints a FIX: line naming the action that repairs it and a RERUN: line with your personalized non-interactive one-liner (secrets typed at prompts are placeholdered, never echoed). Fix the condition and paste the rerun line — the installer is idempotent.

Amazon Bedrock: key vs IAM

--bedrock-auth key (default when --bedrock-key is given) stores the bearer token; --bedrock-auth iam is keyless and reads the pod’s own AWS credential chain.

Keyless IAM Bedrock requires the updated xpander SDK release. On older SDK versions, chats work but task executions fail — the SDK sets an empty bearer token unconditionally. On EKS it also needs IRSA or pod identity wired to a role with Bedrock access. If in doubt, use the bearer-token mode.

3. Verify the Connection

When the install completes, a summary box prints the release, control-plane URL, ingress state, and configured providers. Your cluster registers with the control plane and the xpander console flips to “Connection established” in a minute or two.

kubectl get pods -n xpander

Expect one Running, 1/1-ready pod per application service (agent-controller, agent-worker, ai-gateway, api, aws-operator, chat, code-runner, mcp) plus the postgres and redis StatefulSets — and, at defaults, docker-registry and metrics-server. Test locally without ingress:

kubectl -n xpander port-forward service/xpander-agent-controller 9016:9016 &
kubectl -n xpander port-forward service/xpander-ai-gateway 9018:9018 &
kubectl -n xpander port-forward service/xpander-mcp 8081:8081 &

curl http://localhost:9016/health
curl http://localhost:9018/health
curl http://localhost:8081/health

In the console (admin_settings#locations), the location shows as “Self-deploy”; the Components tab shows heartbeat status per service, and the Configuration panel toggles Cloud Controls (Chat, Streaming, Scheduler).

PrivateLink

Your cluster must reach the xpander deployment manager. Two options:

Option	URL	Security
Public (TLS)	`https://deployment-manager.xpander.ai`	Encrypted over public internet — no setup needed, the installer’s default
Private (PrivateLink)	`https://deployment-manager-privatelink.xpander.ai`	Encrypted over AWS private backbone — this section

If public works for you, skip this section entirely: your cluster nodes just need outbound HTTPS (via NAT Gateway if in private subnets). If your security policy requires traffic to stay within the AWS network, set up PrivateLink, then tell the installer with --privatelink (or pick AWS PrivateLink at the wizard’s reachability question). --privatelink, --staging, and --dm-url are mutually exclusive — pass exactly one, or none for the public endpoint.

How PrivateLink Works

AWS PrivateLink is a two-sided connection. xpander exposes the deployment manager as a VPC Endpoint Service (provider). You create a VPC Endpoint (consumer) in your VPC, which provisions a private ENI with a 10.x.x.x IP address. Your pods connect to this private IP — traffic goes over the AWS backbone, never the internet.

Property	Value
Service Name	`com.amazonaws.vpce.us-west-2.vpce-svc-0101884b32f655197`
Service Region	`us-west-2`
Supported Regions	`us-west-2`, `us-west-1`, `eu-west-2`, `eu-central-1`, `ap-south-1`
Acceptance Required	No (auto-accepted for allowed accounts)
Private DNS Name	`deployment-manager-privatelink.xpander.ai`

Contact xpander to have your AWS account ID added as an allowed principal before creating the VPC endpoint.

Set It Up

Create the VPC endpoint

The xpander endpoint service is in us-west-2. If your cluster is in a different region, you must include the --service-region us-west-2 flag — this is the most common mistake (see Troubleshooting).

aws ec2 create-vpc-endpoint \
  --vpc-id <VPC_ID> \
  --vpc-endpoint-type Interface \
  --service-name com.amazonaws.vpce.us-west-2.vpce-svc-0101884b32f655197 \
  --service-region us-west-2 \
  --subnet-ids <PRIVATE_SUBNET_A_ID> <PRIVATE_SUBNET_B_ID> \
  --no-private-dns-enabled \
  --tag-specifications 'ResourceType=vpc-endpoint,Tags=[{Key=Name,Value=xpander-privatelink}]' \
  --region <REGION> --profile <PROFILE>

Save from the output: VpcEndpointId, Groups[0].GroupId (the security group), DnsEntries[0].DnsName, and DnsEntries[0].HostedZoneId.

If your cluster is in us-west-2 (same region as the service), you can omit --service-region.

Allow HTTPS on the endpoint security group

The VPC endpoint is created with the VPC’s default security group, which only allows inbound traffic from itself. Allow HTTPS from your VPC CIDR:

aws ec2 authorize-security-group-ingress \
  --group-id <ENDPOINT_SG_ID> \
  --protocol tcp \
  --port 443 \
  --cidr <VPC_CIDR> \
  --region <REGION> --profile <PROFILE>

Wait for the endpoint to become available

The endpoint transitions from pending to available in 1-3 minutes (auto-accepted):

aws ec2 describe-vpc-endpoints \
  --vpc-endpoint-ids <VPCE_ID> \
  --query 'VpcEndpoints[0].State' \
  --region <REGION> --profile <PROFILE>

Create private DNS

Create a private hosted zone so deployment-manager-privatelink.xpander.ai resolves to the endpoint ENI inside your VPC:

aws route53 create-hosted-zone \
  --name deployment-manager-privatelink.xpander.ai \
  --caller-reference "xpander-privatelink-$(date +%s)" \
  --vpc VPCRegion=<REGION>,VPCId=<VPC_ID> \
  --hosted-zone-config PrivateZone=true \
  --profile <PROFILE>

Save the HostedZone.Id from the output, then create the alias record:

cat <<EOF > /tmp/privatelink-dns.json
{
  "Changes": [{
    "Action": "UPSERT",
    "ResourceRecordSet": {
      "Name": "deployment-manager-privatelink.xpander.ai.",
      "Type": "A",
      "AliasTarget": {
        "DNSName": "<VPCE_DNS_NAME>",
        "HostedZoneId": "<VPCE_HOSTED_ZONE_ID>",
        "EvaluateTargetHealth": true
      }
    }
  }]
}
EOF

aws route53 change-resource-record-sets \
  --hosted-zone-id <PRIVATE_ZONE_ID> \
  --change-batch file:///tmp/privatelink-dns.json \
  --profile <PROFILE>

Verify connectivity

kubectl run curl-test --restart=Never --image=curlimages/curl:latest -n default \
  --command -- sh -c "curl -sk -o /dev/null -w '%{http_code}' \
  --connect-timeout 10 https://deployment-manager-privatelink.xpander.ai/health; echo"

sleep 15 && kubectl logs curl-test
# Expected output: 200

kubectl delete pod curl-test

A timeout or 000? See Troubleshooting below.

External Access & DNS

The xpander services communicate with each other internally via Kubernetes Services — no ingress is needed for that. You only need external access so that users can reach the chat UI, API, and other endpoints from their browser or client.

Already have an ingress controller? Give the installer a --domain (or set ingress.enabled=true in Helm values) and skip to DNS records.

Option A: VPN / Corporate Network Access

If your organization has a VPN that routes into the VPC, users can access xpander services directly — no ingress controller, NLB, ACM certificate, or public DNS needed. Point internal DNS records to the Kubernetes services:

chat.<DOMAIN>              → xpander-chat.xpander.svc.cluster.local
agent-controller.<DOMAIN>  → xpander-agent-controller.xpander.svc.cluster.local
ai-gateway.<DOMAIN>        → xpander-ai-gateway.xpander.svc.cluster.local

When installing, skip the domain prompt (ingress stays disabled, the default).

Option B: Public Access via Ingress + NLB

For public or internet-facing access, set up an ACM certificate, nginx ingress controller, and NLB.

SSL Certificate

Request an ACM certificate for your domain.

The certificate must include a wildcard for *.chat.<DOMAIN> because the xpander chat UI generates per-thread subdomains (e.g., moccasin-prawn.chat.<DOMAIN>). Standard wildcards only match one level, so *.<DOMAIN> does not cover these. If you omit *.chat.<DOMAIN>, the chat UI will show SSL certificate errors for thread URLs.

aws acm request-certificate \
  --domain-name "<DOMAIN>" \
  --subject-alternative-names "*.<DOMAIN>" "*.chat.<DOMAIN>" \
  --validation-method DNS \
  --region <REGION> --profile <PROFILE>

Add the DNS validation CNAME records to your Route 53 hosted zone. There will be two unique validation records — one for <DOMAIN> / *.<DOMAIN> (shared) and one for *.chat.<DOMAIN>:

aws acm describe-certificate --certificate-arn <CERT_ARN> \
  --region <REGION> --profile <PROFILE> \
  --query 'Certificate.DomainValidationOptions[0].ResourceRecord'

Wait for validation:

aws acm wait certificate-validated --certificate-arn <CERT_ARN> \
  --region <REGION> --profile <PROFILE>

Ingress Controller

Install the nginx ingress controller with AWS NLB and ACM SSL termination:

helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update

helm upgrade --install ingress-nginx ingress-nginx/ingress-nginx \
  --namespace ingress-nginx --create-namespace \
  --set controller.service.type=LoadBalancer \
  --set controller.service.annotations."service\.beta\.kubernetes\.io/aws-load-balancer-type"=nlb \
  --set controller.service.annotations."service\.beta\.kubernetes\.io/aws-load-balancer-scheme"=internet-facing \
  --set controller.service.annotations."service\.beta\.kubernetes\.io/aws-load-balancer-ssl-cert"="<ACM_CERT_ARN>" \
  --set controller.service.annotations."service\.beta\.kubernetes\.io/aws-load-balancer-ssl-ports"="443" \
  --set controller.service.annotations."service\.beta\.kubernetes\.io/aws-load-balancer-backend-protocol"="tcp" \
  --set controller.service.targetPorts.https=http

Get the NLB hostname (you’ll need it for the DNS records):

kubectl get svc -n ingress-nginx ingress-nginx-controller \
  -o jsonpath='{.status.loadBalancer.ingress[0].hostname}'

DNS Records

Create wildcard CNAME records pointing *.<DOMAIN> and *.chat.<DOMAIN> to the NLB hostname. Both are required — the chat UI generates per-thread subdomains under chat.<DOMAIN>.

cat <<EOF > /tmp/dns-record.json
{
  "Changes": [
    {
      "Action": "UPSERT",
      "ResourceRecordSet": {
        "Name": "*.<DOMAIN>.",
        "Type": "CNAME",
        "TTL": 300,
        "ResourceRecords": [{ "Value": "<NLB_HOSTNAME>" }]
      }
    },
    {
      "Action": "UPSERT",
      "ResourceRecordSet": {
        "Name": "*.chat.<DOMAIN>.",
        "Type": "CNAME",
        "TTL": 300,
        "ResourceRecords": [{ "Value": "<NLB_HOSTNAME>" }]
      }
    }
  ]
}
EOF

aws route53 change-resource-record-sets \
  --hosted-zone-id <HOSTED_ZONE_ID> \
  --change-batch file:///tmp/dns-record.json \
  --profile <PROFILE>

This creates the following endpoints:

Service	URL
API	`https://api.<DOMAIN>`
Chat	`https://chat.<DOMAIN>`
AI Gateway	`https://ai-gateway.<DOMAIN>`
Agent Controller	`https://agent-controller.<DOMAIN>`
Agent Worker	`https://agent-worker.<DOMAIN>`
MCP	`https://mcp.<DOMAIN>`
Code Runner	`https://code-runner.<DOMAIN>`
AWS Operator	`https://aws-operator.<DOMAIN>`

Test it:

curl -sk https://api.<DOMAIN>/health

Managing LLM API Keys

Use helm upgrade with --reuse-values to add or change API keys without affecting existing configuration:

helm upgrade xpander xpander/xpander \
  --namespace xpander --reuse-values \
  --set secrets.static.anthropicApiKey=<KEY> \
  --timeout 5m

helm upgrade xpander xpander/xpander \
  --namespace xpander --reuse-values \
  --set secrets.static.anthropicApiKey=<KEY> \
  --set secrets.static.agentsOpenaiApiKey=<KEY> \
  --set secrets.static.fireworksApiKey=<KEY> \
  --timeout 5m

After the upgrade, pods that use the updated secrets will automatically restart. Verify with kubectl get pods -n xpander.

Secret not updating after helm upgrade?

The xpander-static secret has a Helm resource keep policy — helm upgrade may not update it on subsequent installs. If your API key isn’t being picked up after a helm upgrade, set it directly in the secret:

# Set the key directly in the Kubernetes secret
kubectl patch secret xpander-static -n xpander --type=merge \
  -p "{\"data\":{\"ANTHROPIC_API_KEY\":\"$(echo -n '<YOUR_KEY>' | base64)\"}}"

# Restart the worker to pick up the new key
kubectl rollout restart deployment xpander-agent-worker -n xpander

Secret field name mapping:

Helm Value	Secret Field
`secrets.static.anthropicApiKey`	`ANTHROPIC_API_KEY`
`secrets.static.agentsOpenaiApiKey`	`AGENTS_OPENAI_API_KEY`
`secrets.static.fireworksApiKey`	`FIREWORKS_API_KEY`
`secrets.static.awsBedrockApiKey`	`AWS_BEARER_TOKEN_BEDROCK`
`secrets.static.googleApiKey`	`GOOGLE_API_KEY`
`secrets.static.nebiusApiKey`	`NEBIUS_API_KEY`
`secrets.static.heliconeApiKey`	`HELICONE_API_KEY`
`secrets.static.openrouterApiKey`	`OPENROUTER_API_KEY`
`secrets.static.deploymentManagerApiKey`	`DEPLOYMENT_MANAGER_API_KEY`

Using Kubernetes Secrets (Recommended for Production)

kubectl create secret generic ai-service-keys \
  --namespace xpander \
  --from-literal=openai-api-key=sk-your-openai-key \
  --from-literal=anthropic-api-key=sk-ant-your-anthropic-key

# In your values file
agent-worker:
  envFromSecretKeys:
    AGENTS_OPENAI_API_KEY:
      secretName: "ai-service-keys"
      key: "openai-api-key"
    ANTHROPIC_API_KEY:
      secretName: "ai-service-keys"
      key: "anthropic-api-key"

Using the SDK with Self-Hosted

Configure the xpander SDK to point to your Agent Controller endpoint.

When using self-hosted deployment, use the Agent Controller API key generated during installation, not your xpander.ai cloud API key.

from xpander_sdk import Configuration

config = Configuration(
    api_key="your-agent-controller-api-key",  # From the installation
    organization_id="your-org-id",
    base_url="https://agent-controller.my-company.com"
)

Using with Agno Framework

from xpander_sdk import Backend, Configuration
from agno.agent import Agent

config = Configuration(
    api_key="your-agent-controller-api-key",  # From the installation
    organization_id="your-org-id",
    base_url="https://agent-controller.my-company.com"
)

backend = Backend(configuration=config)
agno_agent = Agent(**backend.get_args(agent_id="agent-123", task=task))

result = await agno_agent.arun(input="What can you help me with?")

Make sure your base_url points to the Agent Controller endpoint (e.g., https://agent-controller.{your-domain}), not the root domain.For more SDK examples, see the Self-Hosted SDK Configuration.

Upgrading

Upgrades work like the Datadog agent flow: watch the xpander releases feed (Releases page / Atom feed), then run the upgrade subcommand:

curl -fsSL https://charts.xpander.ai/install.sh | bash -s -- upgrade
# or, pinned:
curl -fsSL https://charts.xpander.ai/install.sh | bash -s -- upgrade --chart-version <X.Y.Z>

install.sh upgrade:

finds the release (default xpander in namespace xpander; override with --release / --namespace),
infers its edition from the installed chart — you never pass --edition to an upgrade,
reuses the values the release was installed with (helm get values), plus anything extra you pass (--set, --values),
shows current → target version with a link to the release notes, and asks before changing anything,
always runs with --atomic: a failed upgrade (including a failed hook) rolls back automatically and the previous revision keeps serving.

Manual-Helm equivalent:

helm repo update
helm upgrade xpander xpander/xpander --namespace xpander --reuse-values

Installer Flags Reference

bash install.sh [flags]            # install (the wizard, then helm upgrade --install)
bash install.sh upgrade [flags]    # upgrade an existing release in place
bash install.sh status [flags]     # read-only health report (--json for one object)

Identity (Hybrid identifies with the three console values; required non-interactively):

Flag	Description
`--edition hybrid\|airgap`	Edition. `hybrid` (default): worker plane here, control plane in xpander cloud. `airgap`: the whole platform here, no internet
`--org UUID`	Organization ID
`--env UUID`	Environment ID
`--dm-api-key KEY`	Deployment manager API key

LLM providers (all optional — the wizard asks for these):

Flag	Description
`--openai-key KEY`	OpenAI
`--anthropic-key KEY`	Anthropic
`--google-key KEY`	Google / Gemini
`--bedrock-key KEY`	Bedrock bearer token (`AWS_BEARER_TOKEN_BEDROCK`)
`--bedrock-auth key\|iam`	`key` (default when `--bedrock-key` is given) or keyless IAM role / EKS pod identity — see caveat
`--fireworks-key KEY`	Fireworks AI
`--nebius-key KEY`	Nebius
`--helicone-key KEY`	Helicone observability
`--openrouter-key KEY`	OpenRouter

Hybrid connectivity & ingress:

Flag	Description
`--domain DOMAIN`	Enable ingress for this domain
`--dash-domains`	Dash-separated hostnames (`agent-controller-xpander.<DOMAIN>`)
`--privatelink`	Reach the deployment manager over AWS PrivateLink
`--staging`	Point at the xpander staging control plane (auto-detected for local/staging domains and local kube contexts)
`--dm-url URL`	Custom deployment manager URL. `--privatelink` / `--staging` / `--dm-url` are mutually exclusive

Chart and release:

Flag	Description
`--chart-path PATH`	Install from a local chart directory or `.tgz`
`--chart-version VERSION`	Pin the chart version
`--set KEY=VALUE`	Extra helm `--set` (repeatable)
`--values FILE`	Extra helm `-f` values file (repeatable)
`--namespace NAME`	Default: `xpander`
`--release NAME`	Default: `xpander`
`--kube-context NAME`	kubectl/helm context

Other:

Flag	Description
`--manual`	Print the whole install as copy-pasteable commands, personalized from the flags you passed, and do nothing. Both editions; secrets stay placeholders. See Advanced
`--values-out PATH`	Write the generated values file to PATH and exit — no cluster needed. Useful for GitOps: review the file, then `helm upgrade --install` it yourself
`--dry-run`	Render everything through helm, install nothing
`-y`, `--yes`	Accept every confirmation
`--non-interactive`	Never read the terminal, even if one exists. Implies `--yes`; missing required flags exit 2 instead of prompting
`-q`, `--quiet`	Phase lines and the outcome only
`--json`	`status` only: emit one JSON object
`--help` / `--version`	Usage text / installer version

Air-gap-only flags (license, mirroring, image tags, data tier, base domain, code execution): Air-Gapped edition.

Automation & CI

The installer is built to be driven by scripts, CI pipelines, and coding agents:

Every prompt has a flag. With --non-interactive nothing is read from the terminal, even when one exists: missing required values exit 2 with the usage and a completed rerun line rather than blocking. It implies --yes.
Stable progress output. Progress is one line per transition in a grep-friendly shape — [3/6] chart-source: ok — and --quiet reduces output to phase lines and the outcome only. Color is emitted only when stdout is a terminal.
status is the poll target. A read-only health report: edition, chart, release status, hook jobs, workload readiness, and URLs. --json emits one JSON object.

Exit codes:

Code	Meaning
`0`	Success
`1`	Install or upgrade failure (`status`: degraded)
`2`	Usage error, or a required flag missing with no terminal
`3`	`status`: no such release installed
`4`	Preflight failure (missing helm/kubectl, unreachable cluster, version too old)
`5`	External data-tier probe failed and `--on-probe-failure` was `abort` (air-gap external data tier)

Hybrid, end to end, no terminal:

bash install.sh --non-interactive --quiet \
  --edition hybrid --org <uuid> --env <uuid> --dm-api-key <key> \
  --domain xpander.example.com --openai-key <key>

Poll until healthy:

until bash install.sh status --quiet; do sleep 30; done

When a scripted run fails, the exit code plus the last FIX:/RERUN: lines tell you what to repair — see Troubleshooting.

Advanced: Review-First / Manual Install

For teams that review before they run — everything the installer does can be inspected first, reproduced by hand, or both.

Read the installer before running it

curl -fsSL https://charts.xpander.ai/install.sh | less
curl -fsSL https://charts.xpander.ai/install-v<version>.sh | less   # this exact version

The versioned URL is immutable — install-v<version>.sh is published once per release and never rewritten in place, so pin it in runbooks and CI (bash install.sh --version prints the version you’re holding).
The script is shellcheck-clean, enforced by CI on every push.
Secrets never touch the helm command line: they are written to a temporary values file (mode 600, removed on exit), so they don’t land in shell history or process listings.
Nothing runs against a cluster before the kube-context confirmation — the installer names the context and cluster it is about to install into and waits for your yes.

Do it yourself

Path	What you get
`--manual`	The whole install as copy-pasteable commands — repo setup, secrets, a complete values file, the helm command, verification — personalized from the flags you passed, executing nothing. Secrets stay placeholders
`--values-out PATH`	Just the generated values file (mode 600), plus the printed `helm upgrade --install` command to run yourself. No cluster needed
`--dry-run`	The full flow rendered through `helm --dry-run`, installing nothing

bash install.sh --edition hybrid --org <ORG> --env <ENV> --dm-api-key <KEY> --manual

Manual Helm install (no installer at all)

The installer is a wrapper around helm upgrade --install; everything it does can be done by hand. The cleanest middle path is --values-out:

bash install.sh --edition hybrid --org <ORG> --env <ENV> --dm-api-key <KEY> \
  --domain <DOMAIN> --values-out xpander-values.yaml
helm upgrade --install xpander xpander/xpander \
  --namespace xpander --create-namespace -f xpander-values.yaml

Or fully by hand. Add the repository:

helm repo add xpander https://charts.xpander.ai
helm repo update

Set the deployment manager URL based on your connectivity choice — public https://deployment-manager.xpander.ai, or https://deployment-manager-privatelink.xpander.ai with PrivateLink — and install:

helm upgrade --install xpander xpander/xpander \
  --namespace xpander --create-namespace \
  --set ingress.enabled=true \
  --set domain=<DOMAIN> \
  --set global.organizationId=<ORGANIZATION_ID> \
  --set global.environmentId=<ENVIRONMENT_ID> \
  --set secrets.static.deploymentManagerApiKey=<DEPLOYMENT_MANAGER_API_KEY> \
  --set urls.deploymentManager=<DEPLOYMENT_MANAGER_URL> \
  --timeout 10m

LLM provider keys are secrets.static.* values — the full list is in the key mapping table. Common optional parameters:

Parameter	Default	Description
`domain`	`""` (ingress disabled)	Base domain for ingress hostnames
`ingress.enabled`	`false`	Expose services via ingress
`ingress.tls.enabled`	`false`	Enable TLS on ingress
`ingress.tls.source`	`self-signed`	`self-signed`, `cert-manager`, or `external`
`redis.storage.size`	`8Gi`	Redis PVC size
`redis.storage.storageClass`	cluster default	Storage class for Redis PVC
`dockerRegistry.enabled`	`true`	In-cluster registry for agent container images — disable if your cluster provides one
`metricsServer.enabled`	`true`	metrics-server for autoscaling — disable if your cluster runs one

For production, use a values file instead of --set flags:

xpander-values.yaml

domain: "xpander.production.com"

global:
  organizationId: "your-org-id"
  environmentId: "your-env-id"
  env:
    LOG_LEVEL: "info"
    ENVIRONMENT: "production"

urls:
  deploymentManager: "https://deployment-manager.xpander.ai"

secrets:
  static:
    deploymentManagerApiKey: "your-key"
    agentsOpenaiApiKey: "sk-your-openai-key"
    anthropicApiKey: "sk-ant-your-anthropic-key"

ingress:
  enabled: true
  tls:
    enabled: true
    source: "cert-manager"
  annotations:
    cert-manager.io/cluster-issuer: "letsencrypt-prod"
    nginx.ingress.kubernetes.io/ssl-redirect: "true"

resources:
  agentController:
    requests:
      cpu: "1000m"
      memory: "2Gi"
    limits:
      cpu: "3000m"
      memory: "4Gi"

redis:
  storage:
    size: "32Gi"
    storageClass: "fast-ssd"

helm upgrade --install xpander xpander/xpander \
  --namespace xpander --create-namespace \
  --values xpander-values.yaml

See Requirements & Sizing for recommended resource values per service.

Troubleshooting

Every installer failure prints two lines: FIX: — the action that repairs the condition — and RERUN: — your personalized non-interactive one-liner (secrets typed at prompts are shown as placeholders). The installer is idempotent: fix the condition, paste the rerun line, and it picks up where the failure left off. A secret-scrubbed session log is kept under ~/.xpander/logs/ (latest.log points at the most recent run). For scripted runs, the exit code names the failure class (Automation & CI), and bash install.sh status gives a read-only health verdict.

Installer

Release stuck in pending-install / pending-upgrade / failed

install.sh upgrade requires the release to be in deployed status — operating on a stuck release (pending-install, pending-upgrade, pending-rollback, failed, uninstalling) would strand it. Recover first:

helm history xpander -n xpander
helm rollback xpander <last-good-revision> -n xpander   # if a good revision exists
helm uninstall xpander -n xpander                        # first install that never converged

Then paste your RERUN: line. A pending-install status usually means a first install was interrupted (Ctrl-C, lost connection) or timed out — helm uninstall is the right recovery for it.

Release already runs the other edition's chart

The two editions share nothing and cannot be upgraded into each other. If the release name in the namespace already runs the other chart, the installer stops with the collision. Install under a different release or namespace (--release / --namespace), or remove the existing release first (helm uninstall xpander -n xpander).

Piped run exits with 'these required flags are missing'

Piping the installer into bash leaves no stdin to prompt on, so a piped run is non-interactive (the same contract as --non-interactive): Hybrid requires --org, --env, and --dm-api-key. Missing flags exit with code 2 and a completed rerun line. Copy the personalized one-liner from the xpander console — it carries the identity flags — or download the script and run bash install.sh for the interactive wizard.

Cluster and Pods

Pods stuck in Init

The init containers wait for agent-controller to be ready. Check its logs:

kubectl logs -n xpander deployment/xpander-agent-controller

Common cause: Cannot reach the deployment manager. If using PrivateLink, verify the DNS and security groups are configured correctly — see PrivateLink.

Pods stuck in Pending

Cause: Storage class or PVC issues.

kubectl -n xpander get pvc
kubectl -n xpander describe pvc

Check that a default StorageClass exists and the EBS CSI driver is running:

kubectl get storageclass  # Should show a default (gp3)
kubectl get pods -n kube-system -l app.kubernetes.io/name=aws-ebs-csi-driver  # Should show Running

If using EKS, ensure the node role has AmazonEBSCSIDriverPolicy attached.

exec format error

You are running on ARM/Graviton nodes. xpander images are amd64 only. Switch to x86 instance types (t3, m5, c5, etc.).

Insufficient CPU

The agent-worker pod requests 2 CPU by default. Options:

Add more nodes or use larger instances
For non-production environments only:

kubectl patch deployment xpander-agent-worker -n xpander \
  --type='json' \
  -p='[{"op":"replace","path":"/spec/template/spec/containers/0/resources/requests/cpu","value":"500m"}]'

Health checks failing

Check application logs:

kubectl -n xpander logs deployment/xpander-agent-controller
kubectl -n xpander logs deployment/xpander-ai-gateway
kubectl -n xpander logs deployment/xpander-agent-worker
kubectl -n xpander logs deployment/xpander-mcp

PrivateLink

PrivateLink InvalidServiceName (cross-region)

When creating a VPC endpoint to the xpander service from any region other than us-west-2, you must include --service-region us-west-2:

aws ec2 create-vpc-endpoint \
  --service-name com.amazonaws.vpce.us-west-2.vpce-svc-0101884b32f655197 \
  --service-region us-west-2 \
  ...

Without --service-region, AWS looks for the service in your local region and fails with InvalidServiceName.

PrivateLink connection timeout (HTTP 000)

Check the security group on the VPC endpoint allows inbound TCP 443 from your VPC CIDR:

aws ec2 authorize-security-group-ingress \
  --group-id <ENDPOINT_SG_ID> \
  --protocol tcp --port 443 --cidr <VPC_CIDR> \
  --region <REGION> --profile <PROFILE>

Also verify the private DNS hosted zone and alias record were created correctly — see PrivateLink → Create private DNS.

Ingress and Networking

Ingress not accessible

Verify ingress configuration:

kubectl -n xpander describe ingress
kubectl -n xpander get ingress

Check that the NLB was provisioned and DNS CNAME records point to it:

kubectl get svc -n ingress-nginx ingress-nginx-controller \
  -o jsonpath='{.status.loadBalancer.ingress[0].hostname}'

Load Balancer not provisioning

Symptom: the ingress controller’s Service stays <pending>. Check its events (kubectl -n ingress-nginx describe svc ingress-nginx-controller):

could not find any suitable subnets for creating the ELB — the VPC’s subnets are missing the ELB role tags. Tag public subnets kubernetes.io/role/elb=1 for an internet-facing NLB; for an internal one, tag the private subnets kubernetes.io/role/internal-elb=1 and annotate the Service service.beta.kubernetes.io/aws-load-balancer-internal: "true".
On EKS Auto Mode, ensure the cluster role trust policy includes sts:TagSession:

{
  "Effect": "Allow",
  "Principal": { "Service": "eks.amazonaws.com" },
  "Action": ["sts:AssumeRole", "sts:TagSession"]
}

SSL certificate errors on chat URLs

The ACM certificate must include *.chat.<DOMAIN> as a subject alternative name. The chat UI generates per-thread subdomains (e.g., moccasin-prawn.chat.<DOMAIN>) that are not covered by *.<DOMAIN>.Request a new certificate with:

aws acm request-certificate \
  --domain-name "<DOMAIN>" \
  --subject-alternative-names "*.<DOMAIN>" "*.chat.<DOMAIN>" \
  --validation-method DNS \
  --region <REGION> --profile <PROFILE>

API Keys

API key not being picked up after helm upgrade

The xpander-static secret has a Helm resource keep policy. Set the key directly:

kubectl patch secret xpander-static -n xpander --type=merge \
  -p "{\"data\":{\"ANTHROPIC_API_KEY\":\"$(echo -n '<YOUR_KEY>' | base64)\"}}"

kubectl rollout restart deployment xpander-agent-worker -n xpander

See the secret field name mapping for all key names.

LLM calls fail with a missing API key

Verify the keys are reaching the agent-worker pod:

kubectl -n xpander exec deployment/xpander-agent-worker -- env | grep API_KEY

If missing, re-run the upgrade with the key set (--set secrets.static.<key>=...), or check that your secret is correctly mounted through envFromSecretKeys — see Managing LLM API Keys.

Debug Commands

# Get all resources
kubectl -n xpander get all

# Check events
kubectl -n xpander get events --sort-by=.metadata.creationTimestamp

# Describe problematic pods
kubectl -n xpander describe pod <pod-name>

# Check service endpoints
kubectl -n xpander get endpoints

Uninstalling

# Remove the Helm release
helm uninstall xpander --namespace xpander

# Clean up the namespace (optional)
kubectl delete namespace xpander

​Architecture

​What Gets Deployed

​How It Works

​What Talks to What

​Data Residency

​Requirements & Sizing

​Example Estimates

​Sizing Calculator

​Install

​1. Get Your One-Liner from the Console

​2. Run the Installer

​Install Phases

​Amazon Bedrock: key vs IAM

​3. Verify the Connection

​PrivateLink

​How PrivateLink Works

​Set It Up

​External Access & DNS

​Option A: VPN / Corporate Network Access

​Option B: Public Access via Ingress + NLB

​SSL Certificate

​Ingress Controller

​DNS Records

​Managing LLM API Keys

​Using Kubernetes Secrets (Recommended for Production)

​Using the SDK with Self-Hosted

​Using with Agno Framework

​Upgrading

​Installer Flags Reference

​Automation & CI

​Advanced: Review-First / Manual Install

​Read the installer before running it

​Do it yourself

​Troubleshooting

​Installer

​Cluster and Pods

​PrivateLink

​Ingress and Networking

​API Keys

​Debug Commands

​Uninstalling

Architecture

What Gets Deployed

How It Works

What Talks to What

Data Residency

Requirements & Sizing

Example Estimates

Sizing Calculator

Install

1. Get Your One-Liner from the Console

2. Run the Installer

Install Phases

Amazon Bedrock: key vs IAM

3. Verify the Connection

PrivateLink

How PrivateLink Works

Set It Up

External Access & DNS

Option A: VPN / Corporate Network Access

Option B: Public Access via Ingress + NLB

SSL Certificate

Ingress Controller

DNS Records

Managing LLM API Keys

Using Kubernetes Secrets (Recommended for Production)

Using the SDK with Self-Hosted

Using with Agno Framework

Upgrading

Installer Flags Reference

Automation & CI

Advanced: Review-First / Manual Install

Read the installer before running it

Do it yourself

Troubleshooting

Installer

Cluster and Pods

PrivateLink

Ingress and Networking

API Keys

Debug Commands

Uninstalling