Troubleshoot Hybrid

Every installer failure prints two lines: FIX:, the action that repairs the condition, and RERUN:, your personalized non-interactive one-liner (secrets typed at prompts are shown as placeholders). The installer is idempotent: fix the condition, paste the rerun line, and it picks up where the failure left off. A secret-scrubbed session log is kept under ~/.xpander/logs/ (latest.log points at the most recent run). For scripted runs, the exit code names the failure class (Automation and CI), and bash install.sh status gives a read-only health verdict.

Installer

Release stuck in pending-install / pending-upgrade / failed

install.sh upgrade requires the release to be in deployed status. Operating on a stuck release (pending-install, pending-upgrade, pending-rollback, failed, uninstalling) would strand it. Recover first:

helm history xpander -n xpander
helm rollback xpander <last-good-revision> -n xpander   # if a good revision exists
helm uninstall xpander -n xpander                        # first install that never converged

Then paste your RERUN: line. A pending-install status usually means a first install was interrupted (Ctrl-C, lost connection) or timed out: helm uninstall is the right recovery for it.

Release already runs the other deployment's chart

The two self-hosted deployments share nothing and cannot be upgraded into each other. If the release name in the namespace already runs the other chart, the installer stops with the collision. Install under a different release or namespace (--release / --namespace), or remove the existing release first (helm uninstall xpander -n xpander).

Piped run exits with 'these required flags are missing'

Piping the installer into bash leaves no stdin to prompt on, so a piped run is non-interactive (the same contract as --non-interactive). Hybrid requires --org, --env, and --dm-api-key. Missing flags exit with code 2 and a completed rerun line. Copy the personalized one-liner from the xpander console, it carries the identity flags, or download the script and run bash install.sh for the interactive wizard.

Cluster and pods

Pods stuck in Init

The init containers wait for agent-controller to be ready. Check its logs:

kubectl logs -n xpander deployment/xpander-agent-controller

Common cause: Cannot reach the deployment manager. If using PrivateLink, verify the DNS and security groups are configured correctly: see PrivateLink.

Pods stuck in Pending

Cause: Storage class or PVC issues.

kubectl -n xpander get pvc
kubectl -n xpander describe pvc

Check that a default StorageClass exists and the EBS CSI driver is running:

kubectl get storageclass  # Should show a default (gp3)
kubectl get pods -n kube-system -l app.kubernetes.io/name=aws-ebs-csi-driver  # Should show Running

If using EKS, ensure the node role has AmazonEBSCSIDriverPolicy attached.

exec format error

You are running on ARM/Graviton nodes. xpander images are amd64 only. Switch to x86 instance types (t3, m5, c5, etc.).

Insufficient CPU

The agent-worker pod requests 2 CPU by default. Two options:

Add more nodes, or use larger instances.
For non-production environments only, lower the CPU request directly on the xpander-agent-worker Deployment in the xpander namespace:

kubectl patch deployment xpander-agent-worker -n xpander \
  --type='json' \
  -p='[{"op":"replace","path":"/spec/template/spec/containers/0/resources/requests/cpu","value":"500m"}]'

Health checks failing

Check application logs (skill calls, including AWS ones, log in agent-controller):

kubectl -n xpander logs deployment/xpander-agent-controller
kubectl -n xpander logs deployment/xpander-agent-worker
kubectl -n xpander logs deployment/xpander-mcp

PrivateLink

PrivateLink InvalidServiceName (cross-region)

When creating a VPC endpoint to the xpander service from any region other than us-west-2, you must include --service-region us-west-2:

aws ec2 create-vpc-endpoint \
  --service-name com.amazonaws.vpce.us-west-2.vpce-svc-0101884b32f655197 \
  --service-region us-west-2 \
  ...

Without --service-region, AWS looks for the service in your local region and fails with InvalidServiceName.

PrivateLink connection timeout (HTTP 000)

Check the security group on the VPC endpoint allows inbound TCP 443 from your VPC CIDR:

aws ec2 authorize-security-group-ingress \
  --group-id <ENDPOINT_SG_ID> \
  --protocol tcp --port 443 --cidr <VPC_CIDR> \
  --region <REGION> --profile <PROFILE>

Also verify the private DNS hosted zone and alias record were created correctly: see PrivateLink → Create private DNS.

Ingress and networking

Ingress not accessible

Verify ingress configuration:

kubectl -n xpander describe ingress
kubectl -n xpander get ingress

Check that the NLB was provisioned and DNS CNAME records point to it:

kubectl get svc -n ingress-nginx ingress-nginx-controller \
  -o jsonpath='{.status.loadBalancer.ingress[0].hostname}'

Load Balancer not provisioning

Symptom: the ingress controller’s Service stays <pending>. Check its events (kubectl -n ingress-nginx describe svc ingress-nginx-controller):

could not find any suitable subnets for creating the ELB: the VPC’s subnets are missing the ELB role tags. Tag public subnets kubernetes.io/role/elb=1 for an internet-facing NLB. For an internal one, tag the private subnets kubernetes.io/role/internal-elb=1 and annotate the Service service.beta.kubernetes.io/aws-load-balancer-internal: "true".
On EKS Auto Mode, ensure the cluster role trust policy includes sts:TagSession:

{
  "Effect": "Allow",
  "Principal": { "Service": "eks.amazonaws.com" },
  "Action": ["sts:AssumeRole", "sts:TagSession"]
}

SSL certificate errors on chat URLs

The ACM certificate must include *.chat.<DOMAIN> as a subject alternative name. The chat UI generates per-thread subdomains (e.g., moccasin-prawn.chat.<DOMAIN>) that are not covered by *.<DOMAIN>.Request a new certificate with:

aws acm request-certificate \
  --domain-name "<DOMAIN>" \
  --subject-alternative-names "*.<DOMAIN>" "*.chat.<DOMAIN>" \
  --validation-method DNS \
  --region <REGION> --profile <PROFILE>

API keys

API key not being picked up after helm upgrade

The xpander-static secret has a Helm resource keep policy, so a helm upgrade does not update it on its own. Patch the key directly on the xpander-static Kubernetes secret in the xpander namespace:

# Set the key directly in the Kubernetes secret
kubectl patch secret xpander-static -n xpander --type=merge \
  -p "{\"data\":{\"ANTHROPIC_API_KEY\":\"$(echo -n '<YOUR_KEY>' | base64)\"}}"

# Restart the worker to pick up the new key
kubectl rollout restart deployment xpander-agent-worker -n xpander

See the secret field name mapping for all key names.

New runs refused: The xpander AI gateway is unavailable right now

New agent runs wait for a healthy LLM Gateway. Check its pods and the reason they are not ready (an image the cluster cannot pull is the usual one):

kubectl -n <namespace> get pods -l app=llm-gw
kubectl -n <namespace> describe pods -l app=llm-gw

See LLM Gateway for the healthy result.If you mirror images, mirror xpanderaihub/llm-gw and point images.llmGw or global.imageRegistry at it. The gateway cannot be turned off, so new runs wait until its pods are ready.

LLM calls fail with a missing API key

A call to a provider with no key, neither in the organization (Platform > AI governance > Models) nor on the install (secrets.static), is refused with a message that names the missing key. Add the key in either place. To check the install’s keys, verify they reach the agent-worker pod:

kubectl -n xpander exec deployment/xpander-agent-worker -- env | grep API_KEY

If missing, re-run the upgrade with the key set (--set secrets.static.<key>=...), or check that your secret is correctly mounted through envFromSecretKeys: see Model keys and Bedrock.

Debug commands

# Get all resources
kubectl -n xpander get all

# Check events
kubectl -n xpander get events --sort-by=.metadata.creationTimestamp

# Describe problematic pods
kubectl -n xpander describe pod <pod-name>

# Check service endpoints
kubectl -n xpander get endpoints

Govern

Company Brain

LLM Gateway

Deploy

Integrations

Licensing and identity

Security and compliance

Installer

Cluster and pods

PrivateLink

Ingress and networking

API keys

Debug commands

​Installer

​Cluster and pods

​PrivateLink

​Ingress and networking

​API keys

​Debug commands

Installer

Cluster and pods

PrivateLink

Ingress and networking

API keys

Debug commands