FIX:, the action that repairs the condition, and RERUN:, your personalized non-interactive one-liner (secrets typed at prompts are shown as placeholders). The installer is idempotent: fix the condition, paste the rerun line, and it picks up where the failure left off. A secret-scrubbed session log is kept under ~/.xpander/logs/ (latest.log points at the most recent run). For scripted runs, the exit code names the failure class (Automation and CI), and bash install.sh status gives a read-only health verdict.
Installer
Release stuck in pending-install / pending-upgrade / failed
Release stuck in pending-install / pending-upgrade / failed
install.sh upgrade requires the release to be in deployed status. Operating on a stuck release (pending-install, pending-upgrade, pending-rollback, failed, uninstalling) would strand it. Recover first:RERUN: line. A pending-install status usually means a first install was interrupted (Ctrl-C, lost connection) or timed out: helm uninstall is the right recovery for it.Release already runs the other deployment's chart
Release already runs the other deployment's chart
The two self-hosted deployments share nothing and cannot be upgraded into each other. If the release name in the namespace already runs the other chart, the installer stops with the collision. Install under a different release or namespace (
--release / --namespace), or remove the existing release first (helm uninstall xpander -n xpander).Piped run exits with 'these required flags are missing'
Piped run exits with 'these required flags are missing'
Piping the installer into
bash leaves no stdin to prompt on, so a piped run is non-interactive (the same contract as --non-interactive). Hybrid requires --org, --env, and --dm-api-key. Missing flags exit with code 2 and a completed rerun line. Copy the personalized one-liner from the xpander console, it carries the identity flags, or download the script and run bash install.sh for the interactive wizard.Cluster and pods
Pods stuck in Init
Pods stuck in Init
The init containers wait for Common cause: Cannot reach the deployment manager. If using PrivateLink, verify the DNS and security groups are configured correctly: see PrivateLink.
agent-controller to be ready. Check its logs:Pods stuck in Pending
Pods stuck in Pending
Cause: Storage class or PVC issues.Check that a default StorageClass exists and the EBS CSI driver is running:If using EKS, ensure the node role has
AmazonEBSCSIDriverPolicy attached.exec format error
exec format error
You are running on ARM/Graviton nodes. xpander images are amd64 only. Switch to x86 instance types (
t3, m5, c5, etc.).Insufficient CPU
Insufficient CPU
The
agent-worker pod requests 2 CPU by default. Two options:- Add more nodes, or use larger instances.
- For non-production environments only, lower the CPU request directly on the
xpander-agent-workerDeployment in thexpandernamespace:
Health checks failing
Health checks failing
Check application logs (skill calls, including AWS ones, log in
agent-controller):PrivateLink
PrivateLink InvalidServiceName (cross-region)
PrivateLink InvalidServiceName (cross-region)
When creating a VPC endpoint to the xpander service from any region other than Without
us-west-2, you must include --service-region us-west-2:--service-region, AWS looks for the service in your local region and fails with InvalidServiceName.PrivateLink connection timeout (HTTP 000)
PrivateLink connection timeout (HTTP 000)
Check the security group on the VPC endpoint allows inbound TCP 443 from your VPC CIDR:Also verify the private DNS hosted zone and alias record were created correctly: see PrivateLink → Create private DNS.
Ingress and networking
Ingress not accessible
Ingress not accessible
Verify ingress configuration:Check that the NLB was provisioned and DNS CNAME records point to it:
Load Balancer not provisioning
Load Balancer not provisioning
Symptom: the ingress controller’s Service stays
<pending>. Check its events (kubectl -n ingress-nginx describe svc ingress-nginx-controller):could not find any suitable subnets for creating the ELB: the VPC’s subnets are missing the ELB role tags. Tag public subnetskubernetes.io/role/elb=1for an internet-facing NLB. For an internal one, tag the private subnetskubernetes.io/role/internal-elb=1and annotate the Serviceservice.beta.kubernetes.io/aws-load-balancer-internal: "true".- On EKS Auto Mode, ensure the cluster role trust policy includes
sts:TagSession:
SSL certificate errors
SSL certificate errors
The ACM certificate must cover
<DOMAIN> and *.<DOMAIN>, the hosts the chart’s ingress publishes (api.<DOMAIN>, agent-controller.<DOMAIN>, mcp.<DOMAIN> and the others in DNS records). With ingress.domainStyle.mode: dash the hosts are <service>-<DOMAIN>, so the certificate needs a wildcard on the zone above <DOMAIN> instead. Since chart 0.23.0 there is no chat.<DOMAIN> host, so a *.chat.<DOMAIN> name on an older certificate is no longer needed.Request a new certificate with:API keys
API key not being picked up after helm upgrade
API key not being picked up after helm upgrade
The See the secret field name mapping for all key names.
xpander-static secret has a Helm resource keep policy, so a helm upgrade does not update it on its own. Patch the key directly on the xpander-static Kubernetes secret in the xpander namespace:LLM calls fail with a missing API key
LLM calls fail with a missing API key
A call to a provider with no key, neither in the organization (Platform > AI governance > Models) nor on the install (If missing, re-run the upgrade with the key set (
secrets.static), is refused with a message that names the missing key. Add the key in either place. To check the install’s keys, verify they reach the agent-worker pod:--set secrets.static.<key>=...), or check that your secret is correctly mounted through envFromSecretKeys: see Model keys and Bedrock.
