> ## Documentation Index
> Fetch the complete documentation index at: https://docs.xpander.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Hybrid deployment

> The Hybrid deployment end-to-end, the runtime in your cluster with the control plane on xpander cloud: architecture, requirements, the installer wizard, PrivateLink, LLM keys, upgrades, and troubleshooting

**Hybrid** is the self-hosted deployment where your data and agents run in your cluster and the control plane stays on xpander cloud. Execution, runtime data, credentials, and model keys stay inside your VPC; xpander cloud manages agent definitions, identity, and the skills catalog. The Agent Controller is the only cluster component that reaches Cloud (heartbeats, agent and skill pulls). This page is the complete install guide: from architecture review to a running, connected location.

```bash theme={"dark"}
curl -fsSL https://charts.xpander.ai/install.sh | bash -s -- \
  --edition hybrid --org <ORG> --env <ENV> --dm-api-key <KEY>
```

<Note>
  Self-hosted locations are enabled per account: [contact our team](https://cal.com/team/xpander-ai/activate) if the option isn't available on yours.
</Note>

Comparing deployments first? See [Deployment options](/self-hosted/index). The deployment that runs every component with no connection to xpander cloud at run time is [Air-gapped deployment](/self-hosted/airgap). Network boundary, PrivateLink, and exactly what leaves your VPC are on [Hybrid Architecture](/resources/security-compliance/security-architecture-hybrid): this page does not repeat those diagrams.

***

## Architecture

The AWS network boundary, PrivateLink, authentication and runtime flows, and the Hybrid → Cloud sync surface (cluster identity, unpublished heartbeat schema, agent catalog, skill schema) are on [Hybrid Security Architecture](/resources/security-compliance/security-architecture-hybrid). This section is the operational view: what the chart deploys, how clients connect, and the residency table for an install review.

Hybrid follows the **control plane / data plane** separation pattern, the same model used by AWS EKS, Datadog, and other infrastructure platforms:

* **Control plane (xpander Cloud)**: manages environment metadata, agent registration, skill definitions, and event logging. Admins and team members use Xpander Chat to assemble agents from approved skills and validate tasks. The control plane never touches runtime data.
* **Data plane (your VPC)**: runs all task execution, agent memory, skill credentials, model API keys, and user-facing services. The user's browser connects directly to the data plane. Chat threads, activity views, and task results are served from the cluster: they do not pass through the cloud. Agent definitions and the skills catalog are authored in Cloud and pulled by the Agent Controller.

The data plane connects **outbound only** to the control plane over port 443 for metadata sync and heartbeats. The control plane never initiates connections into your cluster. MCP and external-system calls run from your cluster, and agent workspaces, the agents' computers, run there too.

### What Gets Deployed

| Component | Description |
| - | - |
| **Agent Controller** | Task scheduling, queue management, and dispatch |
| **Execution tier** (`agent-worker`) | Task execution runtime |
| **API Proxy** | API endpoint and request routing |
| **API** | REST API service |
| **MCP** | Accepts MCP client connections, generates tasks |
| **Database** | Agent memory, message history, and monitoring (PostgreSQL) |
| **Redis** | Cache and fast lookups |
| **Chat UI** | Chat interface with per-thread subdomains |
| **Code Runner** | Code execution runtime |
| **AWS APIs Operator** | AWS resource management |

### How It Works

Clients (Slack, the web UI, MCP clients, the SDK and API) connect directly to the cluster in your VPC. When using `chat.xpander.ai`, the cloud UI handles authentication and tells the client which self-hosted cluster to connect to; chat, tasks, and results then flow client-to-cluster. Inside the cluster, the **Agent Controller** schedules tasks and the execution tier (the `agent-worker` service) executes them: calling LLMs, persisting messages in PostgreSQL, and reaching the external systems behind skills through the Agent Controller's skill relay.

The only connection to xpander Cloud is the **deployment manager sync**: the Agent Controller authenticates with `organization_id` + `environment_id` + `api_key`, reports heartbeats, and pulls agent and skill (including MCP) configuration. The heartbeat JSON schema is not published. See [Data that leaves the on-prem installation](/resources/security-compliance/security-architecture-hybrid#data-that-leaves-the-on-prem-installation).

### What Talks to What

The egress surface: the answers a security review needs. Inbound from xpander cloud: **none**; connections are outbound-only.

| From your cluster | To | Port | Purpose |
| - | - | - | - |
| `agent-controller` | `deployment-manager.xpander.ai` (or the [PrivateLink](#privatelink) / staging endpoint) | 443 | Metadata sync, heartbeats, skill updates |
| `agent-controller`, `agent-worker` | Your LLM providers (OpenAI, Anthropic, Google, Bedrock, ...) | 443 | Model calls |
| Skills (including MCP) | The external systems you connect | As configured | Run from your cluster, never from xpander cloud |

Outbound connectivity options to the control plane:

| Option | Target | Details |
| - | - | - |
| **Global Accelerator** | `15.197.85.80`, `166.117.85.46` | Public internet, encrypted over TLS |
| **PrivateLink** | `com.amazonaws.vpce.us-west-2.vpce-svc-0101884b32f655197` | AWS private backbone: [setup below](#privatelink) |

### Data Residency

| Feature | Stays in your VPC? |
| - | :-: |
| Invoking agents (API to agent) | Yes: client connects directly to cluster |
| Chat threads and activity view | Yes: fetched from cluster, not cloud |
| Debugging agents (Task Manager) | Yes: all data served from cluster |
| Knowledge Base file uploads and queries | Yes: stored and queried in cluster |
| User identity and login | No |
| Agent metadata (agent list) | No |

Data at rest stays in PostgreSQL and Redis inside your VPC; credentials and API keys are Kubernetes secrets; the control plane sync carries metadata and heartbeats only: user payloads never leave the VPC. Heartbeat field names are not published. Full field-level inventory: [Data that leaves the on-prem installation](/resources/security-compliance/security-architecture-hybrid#data-that-leaves-the-on-prem-installation).

***

## Requirements

Sizing lives on one page: the minimum footprint, the sizing model and calculator, per-service requirements, the executor fleet and the data services are all on [Sizing and requirements](/self-hosted/sizing). The installer checks the same minimum numbers, published at [charts.xpander.ai/requirements.json](https://charts.xpander.ai/requirements.json).

<Warning>
  xpander container images are **amd64 only**. Do not use ARM/Graviton instances (`t4g`, `m7g`, `c7g`, etc.).
</Warning>

Need a cluster? Follow the [EKS Cluster Setup](/self-hosted/eks-setup) guide.

**Bedrock needs your own credentials.** xpander cloud sends the cluster its own Anthropic, OpenAI and Google keys, never Bedrock credentials. To run Claude Code on Bedrock here, add a Bedrock key or role under **Settings > LLM**, or give the cluster an AWS identity for Bedrock; see [AI vendors](/run/integrations/ai-vendors#anthropic-on-amazon-bedrock).

***

## Install

### 1. Get your one-liner from Xpander Chat

1. In Xpander Chat, open Settings and choose **Locations** (Connected runtimes and regions)
2. Click **"Add Location"**
3. Choose the **Hybrid** edition. The dialog shows the minimum footprint for the deployment you picked.
4. Optionally set a **URL** (your server's FQDN, e.g. `xpander.my-company.com`) and the hostname style (dash-based subdomains if your DNS can't do wildcards)
5. Copy the generated one-liner

The one-liner carries your identity, `--org`, `--env`, and `--dm-api-key` (plus `--domain` when you set a URL). That is how the cluster authenticates to the xpander control plane:

```bash theme={"dark"}
curl -fsSL https://charts.xpander.ai/install.sh | bash -s -- \
  --edition hybrid \
  --org <ORGANIZATION_UUID> \
  --env <ENVIRONMENT_UUID> \
  --dm-api-key <DEPLOYMENT_MANAGER_API_KEY> \
  --domain xpander.my-company.com
```

<Note>
  The one-liner deliberately carries **no LLM provider credentials**: the wizard prompts for those interactively (input hidden), or you pass them as flags. Secrets are written to a temporary values file (mode 600, removed on exit) and never appear on the helm command line.
</Note>

### 2. Run the Installer

Run it from a terminal with `kubectl` access to the target cluster. The command downloads the installer from `charts.xpander.ai` and runs an interactive wizard: preflight checks, configuration, Helm install, then the connection back to xpander. Run bare (`curl -fsSL https://charts.xpander.ai/install.sh | bash`), it is the zero-knowledge entry point: the first question is which edition you are installing.

The wizard walks through, in order:

<Steps>
  <Step title="Edition & requirements">
    If `--edition` wasn't given, it asks. It then prints the minimum footprint for that edition (the same numbers as [charts.xpander.ai/requirements.json](https://charts.xpander.ai/requirements.json)) and asks you to confirm your cluster meets it, stopping cleanly if not.
  </Step>

  <Step title="Identity">
    Organization ID, Environment ID, and the Deployment Manager API key (input hidden): all from the install card in Xpander Chat, all skipped if passed as flags.
  </Step>

  <Step title="Ingress domain (optional)">
    Press Enter to skip: services then keep cluster-internal Services only, which is fine for a first install (port-forward reaches them; add ingress later). With a domain, it asks how to build hostnames: **subdomain** style (`agent-controller.xpander.<DOMAIN>`, needs wildcard DNS) or **dash** style (`agent-controller-xpander.<DOMAIN>`, individual A records).
  </Step>

  <Step title="Control plane reachability">
    Public internet (`https://deployment-manager.xpander.ai`), AWS PrivateLink (`https://deployment-manager-privatelink.xpander.ai`), or Staging. The installer auto-detects staging/local environments (a `.stg.` or local domain, or a kind/k3d/minikube/docker-desktop kube context) and leads with that option; any of `--staging`, `--privatelink`, or `--dm-url` overrides detection.
  </Step>

  <Step title="LLM providers (all optional)">
    OpenAI, Anthropic, Google/Gemini, Amazon Bedrock, then optionally Fireworks, Nebius, Helicone, and OpenRouter. Press Enter to skip any: the xpander installs and starts without keys, and model calls light up as soon as you add a provider key in the UI or with a helm upgrade.

    For **Bedrock** it asks how to authenticate: an API key / bearer token (`AWS_BEARER_TOKEN_BEDROCK`), or keyless IAM role / EKS pod identity. See [Amazon Bedrock: keyless IAM or API key](#amazon-bedrock-keyless-iam-or-api-key).
  </Step>

  <Step title="Namespace, context confirmation, summary">
    Namespace (default `xpander`, created if missing), an explicit confirmation of the kubectl context it is about to install into, and a masked configuration summary before anything changes.
  </Step>
</Steps>

### Install Phases

The Hybrid install runs six phases, each announced as a stable `[N/6] name: ...` progress line (the same shape scripts can grep, see [Automation & CI](#automation--ci)):

```
[1/6] preflight       helm ≥ 3.13, kubectl, cluster reachable, Kubernetes ≥ 1.28
[2/6] configuration   context confirmation + masked summary
[3/6] chart-source    helm repo add xpander https://charts.xpander.ai (or --chart-path)
[4/6] values          secrets written to a mode-600 temp file, removed on exit
[5/6] install         helm upgrade --install xpander xpander/xpander
[6/6] verify          waits for the agent-controller rollout (5-minute timeout)
```

Every failure prints a `FIX:` line naming the action that repairs it and a `RERUN:` line with your personalized non-interactive one-liner (secrets typed at prompts are placeholdered, never echoed). Fix the condition and paste the rerun line: the installer is idempotent.

### Amazon Bedrock: keyless IAM or API key

`--bedrock-auth iam` is keyless: the pods sign Bedrock calls with their own AWS identity (EKS Pod Identity or IRSA) for chat, agent task executions and the summaries around them, supported since xpander-sdk 2.0.503 (images built after 27 August 2026). The role bound to the pods needs the Bedrock permissions itself (`bedrock:InvokeModel`, `bedrock:InvokeModelWithResponseStream`, `bedrock:Converse`, `bedrock:ConverseStream` on `*`) and must be bound to both ServiceAccounts the LLM services run as: `default` and `agent-controller`. The chat and task paths do not assume a second role. `--bedrock-auth key` (default when `--bedrock-key` is given) stores a bearer token instead. Either way the account must be opted in to Bedrock's data retention mode for the models you use. The "IAM role" method in Settings, LLM settings, Providers is not the path for chat and tasks; it applies to the Claude Code leg described next. If chat fails with a Bedrock entry configured that way, remove the entry and restart the `agent-controller`, `agent-worker`, `agents`, `actions` and `functions` deployments.

Claude Code on Bedrock takes a different path. The CLI talks to the Agent Controller's LLM proxy, and the controller resolves AWS auth itself, in this order: `AWS_BEARER_TOKEN_BEDROCK`, then `BEDROCK_ASSUME_ROLE_ARN` (STS, self-refreshing), then static keys, then the pod's default credential chain (Pod Identity / IRSA). Set whichever you use as controller environment through the chart's env values, with `BEDROCK_REGION` or `AWS_REGION` next to it. The cloud relays no Bedrock credential. An organization that wants Claude Code on Bedrock adds its own Bedrock key in Settings > LLM; the proxy prefers that key over the controller's credential.

<Warning>
  **Fable- and Mythos-class Anthropic models on Bedrock need the AWS account's data-retention opt-in, per region.** With the default mode Bedrock answers `400 data retention mode 'default' is not available for this model`. There is no console switch: `PUT https://bedrock.<region>.amazonaws.com/data-retention` with `{"mode": "provider_data_share"}` (or `aws_review` where offered), SigV4-signed with `bedrock:PutAccountDataRetention`, in every region a `global.` inference profile can route to, on whichever account signs the call (the controller's, or the organization's own Bedrock key account). Details on [AI vendors](/run/integrations/ai-vendors#anthropic-on-amazon-bedrock).
</Warning>

### 3. Verify the Connection

When the install completes, a summary box prints the release, control-plane URL, ingress state, and configured providers. Your cluster registers with the control plane and the xpander console flips to **"Connection established"** in a minute or two.

```bash theme={"dark"}
kubectl get pods -n xpander
```

Expect one `Running`, `1/1`-ready pod per application service: `agent-controller`, `agent-worker`, `ai-gateway`, `api`, `aws-operator`, `chat`, `code-runner`, and `mcp`. `ai-gateway` is a registration-only compatibility component; skill calls, including AWS ones, are served by `agent-controller` since chart 0.16.1. You should also see the `postgres` and `redis` StatefulSets, and, at defaults, `docker-registry` and `metrics-server`.

Test locally without ingress:

```bash theme={"dark"}
kubectl -n xpander port-forward service/xpander-agent-controller 9016:9016 &
kubectl -n xpander port-forward service/xpander-ai-gateway 9018:9018 &
kubectl -n xpander port-forward service/xpander-mcp 8081:8081 &

curl http://localhost:9016/health
curl http://localhost:9018/health
curl http://localhost:8081/health
```

In Xpander Chat, the sidebar **ENVIRONMENTS** switcher lists the installation next to "xpander.ai cloud - Managed". Settings > **Locations** shows it under Connected runtimes and regions. Its **Components** tab shows heartbeat status per service, and its **Configuration** panel holds the Cloud Controls switches: Chat, Streaming and Scheduler.

***

## PrivateLink

Your cluster must reach the xpander **deployment manager**. Two options:

| Option | URL | Security |
| - | - | - |
| **Public (TLS)** | `https://deployment-manager.xpander.ai` | Encrypted over public internet. No setup needed; the installer's default |
| **Private (PrivateLink)** | `https://deployment-manager-privatelink.xpander.ai` | Encrypted over AWS private backbone. This section |

If public works for you, skip this section entirely: your cluster nodes just need outbound HTTPS (via NAT Gateway if in private subnets). If your security policy requires traffic to stay within the AWS network, set up PrivateLink, then tell the installer with `--privatelink` (or pick **AWS PrivateLink** at the wizard's reachability question). `--privatelink`, `--staging`, and `--dm-url` are mutually exclusive: pass exactly one, or none for the public endpoint.

### How PrivateLink Works

AWS PrivateLink is a two-sided connection. xpander exposes the deployment manager as a **VPC Endpoint Service** (provider). You create a **VPC Endpoint** (consumer) in your VPC, which provisions a private ENI with a `10.x.x.x` IP address. Your pods connect to this private IP: traffic goes over the AWS backbone, never the internet.

| Property | Value |
| - | - |
| Service Name | `com.amazonaws.vpce.us-west-2.vpce-svc-0101884b32f655197` |
| Service Region | `us-west-2` |
| Supported Regions | `us-west-2`, `us-west-1`, `eu-west-2`, `eu-central-1`, `ap-south-1` |
| Acceptance Required | No (auto-accepted for allowed accounts) |
| Private DNS Name | `deployment-manager-privatelink.xpander.ai` |

<Warning>
  Contact xpander to have your AWS account ID added as an allowed principal before creating the VPC endpoint.
</Warning>

### Set It Up

<Steps>
  <Step title="Create the VPC endpoint">
    <Warning>
      The xpander endpoint service is in `us-west-2`. If your cluster is in a different region, you **must** include the `--service-region us-west-2` flag: this is the most common mistake (see [Troubleshooting](#troubleshooting)).
    </Warning>

    ```bash theme={"dark"}
    aws ec2 create-vpc-endpoint \
      --vpc-id <VPC_ID> \
      --vpc-endpoint-type Interface \
      --service-name com.amazonaws.vpce.us-west-2.vpce-svc-0101884b32f655197 \
      --service-region us-west-2 \
      --subnet-ids <PRIVATE_SUBNET_A_ID> <PRIVATE_SUBNET_B_ID> \
      --no-private-dns-enabled \
      --tag-specifications 'ResourceType=vpc-endpoint,Tags=[{Key=Name,Value=xpander-privatelink}]' \
      --region <REGION> --profile <PROFILE>
    ```

    Save from the output: `VpcEndpointId`, `Groups[0].GroupId` (the security group), `DnsEntries[0].DnsName`, and `DnsEntries[0].HostedZoneId`.

    <Tip>
      If your cluster is in `us-west-2` (same region as the service), you can omit `--service-region`.
    </Tip>
  </Step>

  <Step title="Allow HTTPS on the endpoint security group">
    The VPC endpoint is created with the VPC's default security group, which only allows inbound traffic from itself. Allow HTTPS from your VPC CIDR:

    ```bash theme={"dark"}
    aws ec2 authorize-security-group-ingress \
      --group-id <ENDPOINT_SG_ID> \
      --protocol tcp \
      --port 443 \
      --cidr <VPC_CIDR> \
      --region <REGION> --profile <PROFILE>
    ```
  </Step>

  <Step title="Wait for the endpoint to become available">
    The endpoint transitions from `pending` to `available` in 1-3 minutes (auto-accepted):

    ```bash theme={"dark"}
    aws ec2 describe-vpc-endpoints \
      --vpc-endpoint-ids <VPCE_ID> \
      --query 'VpcEndpoints[0].State' \
      --region <REGION> --profile <PROFILE>
    ```
  </Step>

  <Step title="Create private DNS">
    Create a private hosted zone so `deployment-manager-privatelink.xpander.ai` resolves to the endpoint ENI inside your VPC:

    ```bash theme={"dark"}
    aws route53 create-hosted-zone \
      --name deployment-manager-privatelink.xpander.ai \
      --caller-reference "xpander-privatelink-$(date +%s)" \
      --vpc VPCRegion=<REGION>,VPCId=<VPC_ID> \
      --hosted-zone-config PrivateZone=true \
      --profile <PROFILE>
    ```

    Save the `HostedZone.Id` from the output, then create the alias record:

    ```bash theme={"dark"}
    cat <<EOF > /tmp/privatelink-dns.json
    {
      "Changes": [{
        "Action": "UPSERT",
        "ResourceRecordSet": {
          "Name": "deployment-manager-privatelink.xpander.ai.",
          "Type": "A",
          "AliasTarget": {
            "DNSName": "<VPCE_DNS_NAME>",
            "HostedZoneId": "<VPCE_HOSTED_ZONE_ID>",
            "EvaluateTargetHealth": true
          }
        }
      }]
    }
    EOF

    aws route53 change-resource-record-sets \
      --hosted-zone-id <PRIVATE_ZONE_ID> \
      --change-batch file:///tmp/privatelink-dns.json \
      --profile <PROFILE>
    ```
  </Step>

  <Step title="Verify connectivity">
    ```bash theme={"dark"}
    kubectl run curl-test --restart=Never --image=curlimages/curl:latest -n default \
      --command -- sh -c "curl -sk -o /dev/null -w '%{http_code}' \
      --connect-timeout 10 https://deployment-manager-privatelink.xpander.ai/health; echo"

    sleep 15 && kubectl logs curl-test
    # Expected output: 200

    kubectl delete pod curl-test
    ```

    A timeout or `000`? See [Troubleshooting](#troubleshooting) below.
  </Step>
</Steps>

***

## External Access & DNS

The xpander services communicate with each other internally via Kubernetes Services: no ingress is needed for that. You only need external access so that **users** can reach the chat UI, API, and other endpoints from their browser or client.

<Info>
  **Already have an ingress controller?** Give the installer a `--domain` (or set `ingress.enabled=true` in Helm values) and skip to [DNS records](#dns-records).
</Info>

### Option A: VPN / Corporate Network Access

If your organization has a VPN that routes into the VPC, users can access xpander services directly: no ingress controller, NLB, ACM certificate, or public DNS needed.

Point internal DNS records to the Kubernetes services:

```
chat.<DOMAIN>              → xpander-chat.xpander.svc.cluster.local
agent-controller.<DOMAIN>  → xpander-agent-controller.xpander.svc.cluster.local
ai-gateway.<DOMAIN>        → xpander-ai-gateway.xpander.svc.cluster.local
```

When installing, skip the domain prompt (ingress stays disabled, the default).

### Option B: Public Access via Ingress + NLB

For public or internet-facing access, set up an ACM certificate, an nginx ingress controller, and an NLB, using the three steps below: [SSL Certificate](#ssl-certificate), [Ingress Controller](#ingress-controller), and [DNS Records](#dns-records).

#### SSL Certificate

Request an ACM certificate for your domain.

<Warning>
  The certificate **must** include a wildcard for `*.chat.<DOMAIN>` because the xpander chat UI generates per-thread subdomains (e.g., `moccasin-prawn.chat.<DOMAIN>`). Standard wildcards only match one level, so `*.<DOMAIN>` does not cover these. If you omit `*.chat.<DOMAIN>`, the chat UI will show SSL certificate errors for thread URLs.
</Warning>

```bash theme={"dark"}
aws acm request-certificate \
  --domain-name "<DOMAIN>" \
  --subject-alternative-names "*.<DOMAIN>" "*.chat.<DOMAIN>" \
  --validation-method DNS \
  --region <REGION> --profile <PROFILE>
```

Add the DNS validation CNAME records to your Route 53 hosted zone. There will be two unique validation records, one for `<DOMAIN>` / `*.<DOMAIN>` (shared) and one for `*.chat.<DOMAIN>`:

```bash theme={"dark"}
aws acm describe-certificate --certificate-arn <CERT_ARN> \
  --region <REGION> --profile <PROFILE> \
  --query 'Certificate.DomainValidationOptions[0].ResourceRecord'
```

Wait for validation:

```bash theme={"dark"}
aws acm wait certificate-validated --certificate-arn <CERT_ARN> \
  --region <REGION> --profile <PROFILE>
```

#### Ingress Controller

Install the nginx ingress controller with AWS NLB and ACM SSL termination:

```bash theme={"dark"}
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update

helm upgrade --install ingress-nginx ingress-nginx/ingress-nginx \
  --namespace ingress-nginx --create-namespace \
  --set controller.service.type=LoadBalancer \
  --set controller.service.annotations."service\.beta\.kubernetes\.io/aws-load-balancer-type"=nlb \
  --set controller.service.annotations."service\.beta\.kubernetes\.io/aws-load-balancer-scheme"=internet-facing \
  --set controller.service.annotations."service\.beta\.kubernetes\.io/aws-load-balancer-ssl-cert"="<ACM_CERT_ARN>" \
  --set controller.service.annotations."service\.beta\.kubernetes\.io/aws-load-balancer-ssl-ports"="443" \
  --set controller.service.annotations."service\.beta\.kubernetes\.io/aws-load-balancer-backend-protocol"="tcp" \
  --set controller.service.targetPorts.https=http
```

Get the NLB hostname (you'll need it for the DNS records):

```bash theme={"dark"}
kubectl get svc -n ingress-nginx ingress-nginx-controller \
  -o jsonpath='{.status.loadBalancer.ingress[0].hostname}'
```

### DNS Records

Create wildcard CNAME records pointing `*.<DOMAIN>` and `*.chat.<DOMAIN>` to the NLB hostname. Both are required: the chat UI generates per-thread subdomains under `chat.<DOMAIN>`.

```bash theme={"dark"}
cat <<EOF > /tmp/dns-record.json
{
  "Changes": [
    {
      "Action": "UPSERT",
      "ResourceRecordSet": {
        "Name": "*.<DOMAIN>.",
        "Type": "CNAME",
        "TTL": 300,
        "ResourceRecords": [{ "Value": "<NLB_HOSTNAME>" }]
      }
    },
    {
      "Action": "UPSERT",
      "ResourceRecordSet": {
        "Name": "*.chat.<DOMAIN>.",
        "Type": "CNAME",
        "TTL": 300,
        "ResourceRecords": [{ "Value": "<NLB_HOSTNAME>" }]
      }
    }
  ]
}
EOF

aws route53 change-resource-record-sets \
  --hosted-zone-id <HOSTED_ZONE_ID> \
  --change-batch file:///tmp/dns-record.json \
  --profile <PROFILE>
```

This creates the following endpoints:

| Service | URL |
| - | - |
| API | `https://api.<DOMAIN>` |
| Chat | `https://chat.<DOMAIN>` |
| AI Gateway | `https://ai-gateway.<DOMAIN>` |
| Agent Controller | `https://agent-controller.<DOMAIN>` |
| Execution tier (`agent-worker`) | `https://agent-worker.<DOMAIN>` |
| MCP | `https://mcp.<DOMAIN>` |
| Code Runner | `https://code-runner.<DOMAIN>` |
| AWS Operator | `https://aws-operator.<DOMAIN>` |

Test it:

```bash theme={"dark"}
curl -sk https://api.<DOMAIN>/health
```

***

## Managing LLM API Keys

Use `helm upgrade` with `--reuse-values` to add or change API keys without affecting existing configuration:

<CodeGroup>
  ```bash Single Key theme={"dark"}
  helm upgrade xpander xpander/xpander \
    --namespace xpander --reuse-values \
    --set secrets.static.anthropicApiKey=<KEY> \
    --timeout 5m
  ```

  ```bash Multiple Keys theme={"dark"}
  helm upgrade xpander xpander/xpander \
    --namespace xpander --reuse-values \
    --set secrets.static.anthropicApiKey=<KEY> \
    --set secrets.static.agentsOpenaiApiKey=<KEY> \
    --set secrets.static.fireworksApiKey=<KEY> \
    --timeout 5m
  ```
</CodeGroup>

After the upgrade, pods that use the updated secrets will automatically restart. Verify with `kubectl get pods -n xpander`.

### Your own keys and credits

Keys can also live in the product rather than the chart: an organization-level key per provider in Settings > LLM, or an agent-level credential on one agent. Whenever a model call runs on a key you brought (agent-level first, then the org-level key), it is **never charged credits**; xpander still measures the usage, so budgets, usage reports and the task's token panel stay accurate. Calls on keys relayed from xpander cloud are billed as usual. Agents on Claude Code, Codex or OpenCode resolve keys the same way through the controller's LLM proxy, so the same rule applies to their turns.

<Accordion title="Secret not updating after helm upgrade?">
  The `xpander-static` secret has a Helm resource keep policy, so `helm upgrade` may not update it on subsequent installs. If your API key isn't being picked up after a helm upgrade, set it directly in the secret:

  ```bash theme={"dark"}
  # Set the key directly in the Kubernetes secret
  kubectl patch secret xpander-static -n xpander --type=merge \
    -p "{\"data\":{\"ANTHROPIC_API_KEY\":\"$(echo -n '<YOUR_KEY>' | base64)\"}}"

  # Restart the worker to pick up the new key
  kubectl rollout restart deployment xpander-agent-worker -n xpander
  ```

  **Secret field name mapping:**

  | Helm Value | Secret Field |
  | - | - |
  | `secrets.static.anthropicApiKey` | `ANTHROPIC_API_KEY` |
  | `secrets.static.agentsOpenaiApiKey` | `AGENTS_OPENAI_API_KEY` |
  | `secrets.static.fireworksApiKey` | `FIREWORKS_API_KEY` |
  | `secrets.static.awsBedrockApiKey` | `AWS_BEARER_TOKEN_BEDROCK` |
  | `secrets.static.googleApiKey` | `GOOGLE_API_KEY` |
  | `secrets.static.nebiusApiKey` | `NEBIUS_API_KEY` |
  | `secrets.static.heliconeApiKey` | `HELICONE_API_KEY` |
  | `secrets.static.openrouterApiKey` | `OPENROUTER_API_KEY` |
  | `secrets.static.deploymentManagerApiKey` | `DEPLOYMENT_MANAGER_API_KEY` |
</Accordion>

### Using Kubernetes Secrets (Recommended for Production)

```bash theme={"dark"}
kubectl create secret generic ai-service-keys \
  --namespace xpander \
  --from-literal=openai-api-key=sk-your-openai-key \
  --from-literal=anthropic-api-key=sk-ant-your-anthropic-key
```

```yaml theme={"dark"}
# In your values file
agent-worker:
  envFromSecretKeys:
    AGENTS_OPENAI_API_KEY:
      secretName: "ai-service-keys"
      key: "openai-api-key"
    ANTHROPIC_API_KEY:
      secretName: "ai-service-keys"
      key: "anthropic-api-key"
```

***

## Using the SDK with Self-Hosted

Set the `base_url` field on the xpander SDK's `Configuration` object to point at your Agent Controller endpoint, as shown below.

<Note>
  When using self-hosted deployment, use the **Agent Controller API key** generated during installation, not your xpander.ai cloud API key.
</Note>

```python theme={"dark"}
from xpander_sdk import Configuration

config = Configuration(
    api_key="your-agent-controller-api-key",  # From the installation
    organization_id="your-org-id",
    base_url="https://agent-controller.my-company.com"
)
```

### Using the SDK against your install

<Note>
  Make sure your `base_url` points to the Agent Controller endpoint (e.g., `https://agent-controller.{your-domain}`), not the root domain.

  For more SDK examples, see the [Self-Hosted SDK Configuration](/developers/sdk-reference/overview#self-hosted).
</Note>

***

## Upgrading

Upgrades work like the Datadog agent flow: watch the [xpander releases feed](https://github.com/xpander-ai/releases) (Releases page / Atom feed), then run the `upgrade` subcommand:

```bash theme={"dark"}
curl -fsSL https://charts.xpander.ai/install.sh | bash -s -- upgrade
# or, pinned:
curl -fsSL https://charts.xpander.ai/install.sh | bash -s -- upgrade --chart-version <X.Y.Z>
```

`install.sh upgrade`:

* finds the release (default `xpander` in namespace `xpander`; override with `--release` / `--namespace`),
* **infers its edition** from the installed chart: you never pass `--edition` to an upgrade,
* **reuses the values the release was installed with** (`helm get values`), plus anything extra you pass (`--set`, `--values`),
* shows current → target version with a link to the release notes, and asks before changing anything,
* always runs with `--atomic`: a failed upgrade (including a failed hook) rolls back automatically and the previous revision keeps serving.

Manual-Helm equivalent:

```bash theme={"dark"}
helm repo update
helm upgrade xpander xpander/xpander --namespace xpander --reuse-values
```

What to expect from the current chart line when you upgrade:

* **No new required values.** The executor drain knobs (`fleet.executor.drainSeconds` 3600, `fleet.executor.terminationGraceSeconds` 7500) ship with defaults; your existing values file upgrades unchanged.
* **PostgreSQL no longer restarts on every chart release.** The config checksum now hashes the ConfigMap data only, so a release whose Postgres configuration is byte-identical leaves the StatefulSet alone; a real `postgres.*` change still rolls it.
* **The fleet is always rendered, and `--atomic` waits for it.** A running turn holds the roll for up to the drain window; the idle guard renders the executor at zero replicas on a cluster without its StorageClass so the upgrade completes anyway. If you drive `helm upgrade --wait` by hand, give `--timeout` at least the drain window or upgrade in a quiet period. Details in the [fleet accordion](#requirements--sizing) above and on [Sizing and requirements](/self-hosted/sizing#safe-rollout-drain-and-upgrades).
* **The drain needs a current `agent-harness` image.** An older executor image ignores the drain setting and exits at once, as before.

***

## Installer Flags Reference

```bash theme={"dark"}
bash install.sh [flags]            # install (the wizard, then helm upgrade --install)
bash install.sh upgrade [flags]    # upgrade an existing release in place
bash install.sh status [flags]     # read-only health report (--json for one object)
```

**Identity** (Hybrid identifies with the three values shown in Xpander Chat; required non-interactively):

| Flag | Description |
| - | - |
| `--edition hybrid\|airgap` | Edition. `hybrid` (default): execution tier here, control plane in xpander cloud. `airgap`: every component here, no connection to xpander cloud at run time |
| `--org UUID` | Organization ID |
| `--env UUID` | Environment ID |
| `--dm-api-key KEY` | Deployment manager API key |

**LLM providers** (all optional, the wizard asks for these):

| Flag | Description |
| - | - |
| `--openai-key KEY` | OpenAI |
| `--anthropic-key KEY` | Anthropic |
| `--google-key KEY` | Google / Gemini |
| `--bedrock-key KEY` | Bedrock bearer token (`AWS_BEARER_TOKEN_BEDROCK`) |
| `--bedrock-auth key\|iam` | `key` (default when `--bedrock-key` is given) or keyless IAM role / EKS pod identity, see [Amazon Bedrock: keyless IAM or API key](#amazon-bedrock-keyless-iam-or-api-key) |
| `--fireworks-key KEY` | Fireworks AI |
| `--nebius-key KEY` | Nebius |
| `--helicone-key KEY` | Helicone observability |
| `--openrouter-key KEY` | OpenRouter |

**Hybrid connectivity & ingress:**

| Flag | Description |
| - | - |
| `--domain DOMAIN` | Enable ingress for this domain |
| `--dash-domains` | Dash-separated hostnames (`agent-controller-xpander.<DOMAIN>`) |
| `--privatelink` | Reach the deployment manager over [AWS PrivateLink](#privatelink) |
| `--staging` | Point at the xpander staging control plane (auto-detected for local/staging domains and local kube contexts) |
| `--dm-url URL` | Custom deployment manager URL. `--privatelink` / `--staging` / `--dm-url` are mutually exclusive |

**Chart and release:**

| Flag | Description |
| - | - |
| `--chart-path PATH` | Install from a local chart directory or `.tgz` |
| `--chart-version VERSION` | Pin the chart version |
| `--set KEY=VALUE` | Extra helm `--set` (repeatable) |
| `--values FILE` | Extra helm `-f` values file (repeatable) |
| `--namespace NAME` | Default: `xpander` |
| `--release NAME` | Default: `xpander` |
| `--kube-context NAME` | kubectl/helm context |

**Other:**

| Flag | Description |
| - | - |
| `--manual` | Print the whole install as copy-pasteable commands, personalized from the flags you passed, and do nothing. Both self-hosted deployments; secrets stay placeholders. See [Advanced](#advanced-review-first--manual-install) |
| `--values-out PATH` | Write the generated values file to PATH and exit: no cluster needed. Useful for GitOps: review the file, then `helm upgrade --install` it yourself |
| `--dry-run` | Render everything through helm, install nothing |
| `-y`, `--yes` | Accept every confirmation |
| `--non-interactive` | Never read the terminal, even if one exists. Implies `--yes`; missing required flags exit 2 instead of prompting |
| `-q`, `--quiet` | Phase lines and the outcome only |
| `--json` | `status` only: emit one JSON object |
| `--help` / `--version` | Usage text / installer version |

Air-gap-only flags (license, mirroring, image tags, data tier, base domain, code execution): [Air-gapped deployment](/self-hosted/airgap).

***

## Automation & CI

The installer is built to be driven by scripts, CI pipelines, and coding agents:

* **Every prompt has a flag.** With `--non-interactive` nothing is read from the terminal, even when one exists: missing required values exit 2 with the usage and a completed rerun line rather than blocking. It implies `--yes`.
* **Stable progress output.** Progress is one line per transition in a grep-friendly shape, `[3/6] chart-source: ok`, and `--quiet` reduces output to phase lines and the outcome only. Color is emitted only when stdout is a terminal.
* **`status` is the poll target.** A read-only health report: edition, chart, release status, hook jobs, workload readiness, and URLs. `--json` emits one JSON object.

**Exit codes:**

| Code | Meaning |
| - | - |
| `0` | Success |
| `1` | Install or upgrade failure (`status`: degraded) |
| `2` | Usage error, or a required flag missing with no terminal |
| `3` | `status`: no such release installed |
| `4` | Preflight failure (missing helm/kubectl, unreachable cluster, version too old) |
| `5` | External data-tier probe failed and `--on-probe-failure` was `abort` (air-gap external data tier) |

**Hybrid, end to end, no terminal:**

```bash theme={"dark"}
bash install.sh --non-interactive --quiet \
  --edition hybrid --org <uuid> --env <uuid> --dm-api-key <key> \
  --domain xpander.example.com --openai-key <key>
```

**Poll until healthy:**

```bash theme={"dark"}
until bash install.sh status --quiet; do sleep 30; done
```

When a scripted run fails, the exit code plus the last `FIX:`/`RERUN:` lines tell you what to repair. See [Troubleshooting](#troubleshooting).

***

## Advanced: Review-First / Manual Install

### Read the installer before running it

```bash theme={"dark"}
curl -fsSL https://charts.xpander.ai/install.sh | less
curl -fsSL https://charts.xpander.ai/install-v<version>.sh | less   # this exact version
```

* The **versioned URL is immutable**: `install-v<version>.sh` is published once per release and never rewritten in place, so pin it in runbooks and CI (`bash install.sh --version` prints the version you're holding).
* The script is **shellcheck-clean**, enforced by CI on every push.
* **Secrets never touch the helm command line**: they are written to a temporary values file (mode 600, removed on exit), so they don't land in shell history or process listings.
* **Nothing runs against a cluster before the kube-context confirmation**: the installer names the context and cluster it is about to install into and waits for your yes.

### Do it yourself

| Path | What you get |
| - | - |
| `--manual` | The whole install as copy-pasteable commands (repo setup, secrets, a complete values file, the helm command, verification) personalized from the flags you passed, **executing nothing**. Secrets stay placeholders |
| `--values-out PATH` | Just the generated values file (mode 600), plus the printed `helm upgrade --install` command to run yourself. No cluster needed |
| `--dry-run` | The full flow rendered through `helm --dry-run`, installing nothing |

```bash theme={"dark"}
bash install.sh --edition hybrid --org <ORG> --env <ENV> --dm-api-key <KEY> --manual
```

<Accordion title="Manual Helm install (no installer at all)">
  The installer is a wrapper around `helm upgrade --install`; everything it does can be done by hand. The cleanest middle path is `--values-out`:

  ```bash theme={"dark"}
  bash install.sh --edition hybrid --org <ORG> --env <ENV> --dm-api-key <KEY> \
    --domain <DOMAIN> --values-out xpander-values.yaml
  helm upgrade --install xpander xpander/xpander \
    --namespace xpander --create-namespace -f xpander-values.yaml
  ```

  Or fully by hand. Add the repository:

  ```bash theme={"dark"}
  helm repo add xpander https://charts.xpander.ai
  helm repo update
  ```

  Set the deployment manager URL based on your connectivity choice (public `https://deployment-manager.xpander.ai`, or `https://deployment-manager-privatelink.xpander.ai` with [PrivateLink](#privatelink)) and install:

  ```bash theme={"dark"}
  helm upgrade --install xpander xpander/xpander \
    --namespace xpander --create-namespace \
    --set ingress.enabled=true \
    --set domain=<DOMAIN> \
    --set global.organizationId=<ORGANIZATION_ID> \
    --set global.environmentId=<ENVIRONMENT_ID> \
    --set secrets.static.deploymentManagerApiKey=<DEPLOYMENT_MANAGER_API_KEY> \
    --set urls.deploymentManager=<DEPLOYMENT_MANAGER_URL> \
    --timeout 10m
  ```

  LLM provider keys are `secrets.static.*` values. The full list is in the [key mapping table](#managing-llm-api-keys). Common optional parameters:

  | Parameter | Default | Description |
  | - | - | - |
  | `domain` | `""` (ingress disabled) | Base domain for ingress hostnames |
  | `ingress.enabled` | `false` | Expose services via ingress |
  | `ingress.tls.enabled` | `false` | Enable TLS on ingress |
  | `ingress.tls.source` | `self-signed` | `self-signed`, `cert-manager`, or `external` |
  | `redis.storage.size` | `8Gi` | Redis PVC size |
  | `redis.storage.storageClass` | cluster default | Storage class for Redis PVC |
  | `dockerRegistry.enabled` | `true` | In-cluster registry for agent container images. Disable it if your cluster provides one |
  | `metricsServer.enabled` | `true` | metrics-server for autoscaling. Disable it if your cluster runs one |

  For production, use a values file instead of `--set` flags:

  ```yaml xpander-values.yaml theme={"dark"}
  domain: "xpander.production.com"

  global:
    organizationId: "your-org-id"
    environmentId: "your-env-id"
    env:
      LOG_LEVEL: "info"
      ENVIRONMENT: "production"

  urls:
    deploymentManager: "https://deployment-manager.xpander.ai"

  secrets:
    static:
      deploymentManagerApiKey: "your-key"
      agentsOpenaiApiKey: "sk-your-openai-key"
      anthropicApiKey: "sk-ant-your-anthropic-key"

  ingress:
    enabled: true
    tls:
      enabled: true
      source: "cert-manager"
    annotations:
      cert-manager.io/cluster-issuer: "letsencrypt-prod"
      nginx.ingress.kubernetes.io/ssl-redirect: "true"

  resources:
    agentController:
      requests:
        cpu: "1000m"
        memory: "2Gi"
      limits:
        cpu: "3000m"
        memory: "4Gi"

  redis:
    storage:
      size: "32Gi"
      storageClass: "fast-ssd"
  ```

  ```bash theme={"dark"}
  helm upgrade --install xpander xpander/xpander \
    --namespace xpander --create-namespace \
    --values xpander-values.yaml
  ```

  See [Requirements & Sizing](#requirements--sizing) for recommended resource values per service.
</Accordion>

***

## Troubleshooting

Every installer failure prints two lines: **`FIX:`**, the action that repairs the condition, and **`RERUN:`**, your personalized non-interactive one-liner (secrets typed at prompts are shown as placeholders). The installer is idempotent: fix the condition, paste the rerun line, and it picks up where the failure left off. A secret-scrubbed session log is kept under `~/.xpander/logs/` (`latest.log` points at the most recent run). For scripted runs, the exit code names the failure class ([Automation & CI](#automation--ci)), and `bash install.sh status` gives a read-only health verdict.

### Installer

<AccordionGroup>
  <Accordion title="Release stuck in pending-install / pending-upgrade / failed">
    `install.sh upgrade` requires the release to be in `deployed` status: operating on a stuck release (`pending-install`, `pending-upgrade`, `pending-rollback`, `failed`, `uninstalling`) would strand it. Recover first:

    ```bash theme={"dark"}
    helm history xpander -n xpander
    helm rollback xpander <last-good-revision> -n xpander   # if a good revision exists
    helm uninstall xpander -n xpander                        # first install that never converged
    ```

    Then paste your `RERUN:` line. A `pending-install` status usually means a first install was interrupted (Ctrl-C, lost connection) or timed out: `helm uninstall` is the right recovery for it.
  </Accordion>

  <Accordion title="Release already runs the other deployment's chart">
    The two self-hosted deployments share nothing and cannot be upgraded into each other. If the release name in the namespace already runs the other chart, the installer stops with the collision. Install under a different release or namespace (`--release` / `--namespace`), or remove the existing release first (`helm uninstall xpander -n xpander`).
  </Accordion>

  <Accordion title="Piped run exits with 'these required flags are missing'">
    Piping the installer into `bash` leaves no stdin to prompt on, so a piped run is non-interactive (the same contract as `--non-interactive`): Hybrid requires `--org`, `--env`, and `--dm-api-key`. Missing flags exit with code 2 and a completed rerun line. Copy the personalized one-liner from the xpander console, it carries the identity flags, or download the script and run `bash install.sh` for the interactive wizard.
  </Accordion>
</AccordionGroup>

### Cluster and Pods

<AccordionGroup>
  <Accordion title="Pods stuck in Init">
    The init containers wait for `agent-controller` to be ready. Check its logs:

    ```bash theme={"dark"}
    kubectl logs -n xpander deployment/xpander-agent-controller
    ```

    **Common cause:** Cannot reach the deployment manager. If using PrivateLink, verify the DNS and security groups are configured correctly: see [PrivateLink](#privatelink).
  </Accordion>

  <Accordion title="Pods stuck in Pending">
    **Cause:** Storage class or PVC issues.

    ```bash theme={"dark"}
    kubectl -n xpander get pvc
    kubectl -n xpander describe pvc
    ```

    Check that a default StorageClass exists and the EBS CSI driver is running:

    ```bash theme={"dark"}
    kubectl get storageclass  # Should show a default (gp3)
    kubectl get pods -n kube-system -l app.kubernetes.io/name=aws-ebs-csi-driver  # Should show Running
    ```

    If using EKS, ensure the node role has `AmazonEBSCSIDriverPolicy` attached.
  </Accordion>

  <Accordion title="exec format error">
    You are running on ARM/Graviton nodes. xpander images are **amd64 only**. Switch to x86 instance types (`t3`, `m5`, `c5`, etc.).
  </Accordion>

  <Accordion title="Insufficient CPU">
    The `agent-worker` pod requests 2 CPU by default. Two options:

    * Add more nodes, or use larger instances.
    * For non-production environments only, lower the CPU request directly on the `xpander-agent-worker` Deployment in the `xpander` namespace:

    ```bash theme={"dark"}
    kubectl patch deployment xpander-agent-worker -n xpander \
      --type='json' \
      -p='[{"op":"replace","path":"/spec/template/spec/containers/0/resources/requests/cpu","value":"500m"}]'
    ```
  </Accordion>

  <Accordion title="Health checks failing">
    Check application logs (skill calls, including AWS ones, log in `agent-controller`):

    ```bash theme={"dark"}
    kubectl -n xpander logs deployment/xpander-agent-controller
    kubectl -n xpander logs deployment/xpander-agent-worker
    kubectl -n xpander logs deployment/xpander-mcp
    ```
  </Accordion>
</AccordionGroup>

### PrivateLink

<AccordionGroup>
  <Accordion title="PrivateLink InvalidServiceName (cross-region)">
    When creating a VPC endpoint to the xpander service from any region other than `us-west-2`, you **must** include `--service-region us-west-2`:

    ```bash theme={"dark"}
    aws ec2 create-vpc-endpoint \
      --service-name com.amazonaws.vpce.us-west-2.vpce-svc-0101884b32f655197 \
      --service-region us-west-2 \
      ...
    ```

    Without `--service-region`, AWS looks for the service in your local region and fails with `InvalidServiceName`.
  </Accordion>

  <Accordion title="PrivateLink connection timeout (HTTP 000)">
    Check the security group on the VPC endpoint allows inbound TCP 443 from your VPC CIDR:

    ```bash theme={"dark"}
    aws ec2 authorize-security-group-ingress \
      --group-id <ENDPOINT_SG_ID> \
      --protocol tcp --port 443 --cidr <VPC_CIDR> \
      --region <REGION> --profile <PROFILE>
    ```

    Also verify the private DNS hosted zone and alias record were created correctly: see [PrivateLink → Create private DNS](#privatelink).
  </Accordion>
</AccordionGroup>

### Ingress and Networking

<AccordionGroup>
  <Accordion title="Ingress not accessible">
    Verify ingress configuration:

    ```bash theme={"dark"}
    kubectl -n xpander describe ingress
    kubectl -n xpander get ingress
    ```

    Check that the NLB was provisioned and DNS CNAME records point to it:

    ```bash theme={"dark"}
    kubectl get svc -n ingress-nginx ingress-nginx-controller \
      -o jsonpath='{.status.loadBalancer.ingress[0].hostname}'
    ```
  </Accordion>

  <Accordion title="Load Balancer not provisioning">
    **Symptom:** the ingress controller's Service stays `<pending>`. Check its events (`kubectl -n ingress-nginx describe svc ingress-nginx-controller`):

    * `could not find any suitable subnets for creating the ELB`: the VPC's subnets are missing the ELB role tags. Tag public subnets `kubernetes.io/role/elb=1` for an internet-facing NLB; for an internal one, tag the private subnets `kubernetes.io/role/internal-elb=1` and annotate the Service `service.beta.kubernetes.io/aws-load-balancer-internal: "true"`.
    * On EKS Auto Mode, ensure the cluster role trust policy includes `sts:TagSession`:

    ```json theme={"dark"}
    {
      "Effect": "Allow",
      "Principal": { "Service": "eks.amazonaws.com" },
      "Action": ["sts:AssumeRole", "sts:TagSession"]
    }
    ```
  </Accordion>

  <Accordion title="SSL certificate errors on chat URLs">
    The ACM certificate must include `*.chat.<DOMAIN>` as a subject alternative name. The chat UI generates per-thread subdomains (e.g., `moccasin-prawn.chat.<DOMAIN>`) that are not covered by `*.<DOMAIN>`.

    Request a new certificate with:

    ```bash theme={"dark"}
    aws acm request-certificate \
      --domain-name "<DOMAIN>" \
      --subject-alternative-names "*.<DOMAIN>" "*.chat.<DOMAIN>" \
      --validation-method DNS \
      --region <REGION> --profile <PROFILE>
    ```
  </Accordion>
</AccordionGroup>

### API Keys

<AccordionGroup>
  <Accordion title="API key not being picked up after helm upgrade">
    The `xpander-static` secret has a Helm resource keep policy, so a `helm upgrade` does not update it on its own. Patch the key directly on the `xpander-static` Kubernetes secret in the `xpander` namespace:

    ```bash theme={"dark"}
    kubectl patch secret xpander-static -n xpander --type=merge \
      -p "{\"data\":{\"ANTHROPIC_API_KEY\":\"$(echo -n '<YOUR_KEY>' | base64)\"}}"

    kubectl rollout restart deployment xpander-agent-worker -n xpander
    ```

    See the [secret field name mapping](#managing-llm-api-keys) for all key names.
  </Accordion>

  <Accordion title="LLM calls fail with a missing API key">
    Verify the keys are reaching the `agent-worker` pod:

    ```bash theme={"dark"}
    kubectl -n xpander exec deployment/xpander-agent-worker -- env | grep API_KEY
    ```

    If missing, re-run the upgrade with the key set (`--set secrets.static.<key>=...`), or check that your secret is correctly mounted through `envFromSecretKeys`: see [Managing LLM API Keys](#managing-llm-api-keys).
  </Accordion>
</AccordionGroup>

### Debug Commands

```bash theme={"dark"}
# Get all resources
kubectl -n xpander get all

# Check events
kubectl -n xpander get events --sort-by=.metadata.creationTimestamp

# Describe problematic pods
kubectl -n xpander describe pod <pod-name>

# Check service endpoints
kubectl -n xpander get endpoints
```

***

## Related

<CardGroup cols={2}>
  <Card title="Hybrid Security Architecture" icon="network-wired" href="/resources/security-compliance/security-architecture-hybrid">
    Network boundary, PrivateLink, and what leaves your VPC
  </Card>

  <Card title="Self-Hosted Overview" icon="server" href="/self-hosted/index">
    Hybrid vs Air-Gapped comparison
  </Card>

  <Card title="Air-gapped deployment" icon="shield-halved" href="/self-hosted/airgap">
    Every component in your environment, no connection to xpander cloud at run time
  </Card>

  <Card title="Access Control" icon="key" href="/resources/security-compliance/access-control">
    Identity by deployment, SSO, API keys, audit logs
  </Card>
</CardGroup>

***

## Uninstalling

```bash theme={"dark"}
# Remove the Helm release
helm uninstall xpander --namespace xpander

# Clean up the namespace (optional)
kubectl delete namespace xpander
```


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.