> ## Documentation Index > Fetch the complete documentation index at: https://docs.xpander.ai/llms.txt > Use this file to discover all available pages before exploring further. # Troubleshoot Hybrid > Fix a failed Hybrid install, stuck pods, PrivateLink, ingress and model key problems. Every installer failure prints two lines: **`FIX:`**, the action that repairs the condition, and **`RERUN:`**, your personalized non-interactive one-liner (secrets typed at prompts are shown as placeholders). The installer is idempotent: fix the condition, paste the rerun line, and it picks up where the failure left off. A secret-scrubbed session log is kept under `~/.xpander/logs/` (`latest.log` points at the most recent run). For scripted runs, the exit code names the failure class ([Automation and CI](https://docs.xpander.ai/self-hosted/hybrid-upgrade.md#automation-and-ci)), and `bash install.sh status` gives a read-only health verdict. ## Installer > [accordion] Release stuck in pending-install / pending-upgrade / failed > > `install.sh upgrade` requires the release to be in `deployed` status. Operating on a stuck release (`pending-install`, `pending-upgrade`, `pending-rollback`, `failed`, `uninstalling`) would strand it. Recover first: > > ```bash theme={"dark"} > helm history xpander -n xpander > helm rollback xpander -n xpander # if a good revision exists > helm uninstall xpander -n xpander # first install that never converged > ``` > > Then paste your `RERUN:` line. A `pending-install` status usually means a first install was interrupted (Ctrl-C, lost connection) or timed out: `helm uninstall` is the right recovery for it. > [accordion] Release already runs the other deployment's chart > > The two self-hosted deployments share nothing and cannot be upgraded into each other. If the release name in the namespace already runs the other chart, the installer stops with the collision. Install under a different release or namespace (`--release` / `--namespace`), or remove the existing release first (`helm uninstall xpander -n xpander`). > [accordion] Piped run exits with 'these required flags are missing' > > Piping the installer into `bash` leaves no stdin to prompt on, so a piped run is non-interactive (the same contract as `--non-interactive`). Hybrid requires `--org`, `--env`, and `--dm-api-key`. Missing flags exit with code 2 and a completed rerun line. Copy the personalized one-liner from the xpander console, it carries the identity flags, or download the script and run `bash install.sh` for the interactive wizard. ## Cluster and pods > [accordion] Pods stuck in Init > > The init containers wait for `agent-controller` to be ready. Check its logs: > > ```bash theme={"dark"} > kubectl logs -n xpander deployment/xpander-agent-controller > ``` > > **Common cause:** Cannot reach the deployment manager. If using PrivateLink, verify the DNS and security groups are configured correctly: see [PrivateLink](https://docs.xpander.ai/self-hosted/hybrid-privatelink.md). > [accordion] Pods stuck in Pending > > **Cause:** Storage class or PVC issues. > > ```bash theme={"dark"} > kubectl -n xpander get pvc > kubectl -n xpander describe pvc > ``` > > Check that a default StorageClass exists and the EBS CSI driver is running: > > ```bash theme={"dark"} > kubectl get storageclass # Should show a default (gp3) > kubectl get pods -n kube-system -l app.kubernetes.io/name=aws-ebs-csi-driver # Should show Running > ``` > > If using EKS, ensure the node role has `AmazonEBSCSIDriverPolicy` attached. > [accordion] exec format error > > You are running on ARM/Graviton nodes. xpander images are **amd64 only**. Switch to x86 instance types (`t3`, `m5`, `c5`, etc.). > [accordion] Insufficient CPU > > The `agent-worker` pod requests 2 CPU by default. Two options: > > * Add more nodes, or use larger instances. > * For non-production environments only, lower the CPU request directly on the `xpander-agent-worker` Deployment in the `xpander` namespace: > > ```bash theme={"dark"} > kubectl patch deployment xpander-agent-worker -n xpander \ > --type='json' \ > -p='[{"op":"replace","path":"/spec/template/spec/containers/0/resources/requests/cpu","value":"500m"}]' > ``` > [accordion] Health checks failing > > Check application logs (skill calls, including AWS ones, log in `agent-controller`): > > ```bash theme={"dark"} > kubectl -n xpander logs deployment/xpander-agent-controller > kubectl -n xpander logs deployment/xpander-agent-worker > kubectl -n xpander logs deployment/xpander-mcp > ``` ## PrivateLink > [accordion] PrivateLink InvalidServiceName (cross-region) > > When creating a VPC endpoint to the xpander service from any region other than `us-west-2`, you **must** include `--service-region us-west-2`: > > ```bash theme={"dark"} > aws ec2 create-vpc-endpoint \ > --service-name com.amazonaws.vpce.us-west-2.vpce-svc-0101884b32f655197 \ > --service-region us-west-2 \ > ... > ``` > > Without `--service-region`, AWS looks for the service in your local region and fails with `InvalidServiceName`. > [accordion] PrivateLink connection timeout (HTTP 000) > > Check the security group on the VPC endpoint allows inbound TCP 443 from your VPC CIDR: > > ```bash theme={"dark"} > aws ec2 authorize-security-group-ingress \ > --group-id \ > --protocol tcp --port 443 --cidr \ > --region --profile > ``` > > Also verify the private DNS hosted zone and alias record were created correctly: see [PrivateLink → Create private DNS](https://docs.xpander.ai/self-hosted/hybrid-privatelink.md#set-it-up). ## Ingress and networking > [accordion] Ingress not accessible > > Verify ingress configuration: > > ```bash theme={"dark"} > kubectl -n xpander describe ingress > kubectl -n xpander get ingress > ``` > > Check that the NLB was provisioned and DNS CNAME records point to it: > > ```bash theme={"dark"} > kubectl get svc -n ingress-nginx ingress-nginx-controller \ > -o jsonpath='{.status.loadBalancer.ingress[0].hostname}' > ``` > [accordion] Load Balancer not provisioning > > **Symptom:** the ingress controller's Service stays ``. Check its events (`kubectl -n ingress-nginx describe svc ingress-nginx-controller`): > > * `could not find any suitable subnets for creating the ELB`: the VPC's subnets are missing the ELB role tags. Tag public subnets `kubernetes.io/role/elb=1` for an internet-facing NLB. For an internal one, tag the private subnets `kubernetes.io/role/internal-elb=1` and annotate the Service `service.beta.kubernetes.io/aws-load-balancer-internal: "true"`. > * On EKS Auto Mode, ensure the cluster role trust policy includes `sts:TagSession`: > > ```json theme={"dark"} > { > "Effect": "Allow", > "Principal": { "Service": "eks.amazonaws.com" }, > "Action": ["sts:AssumeRole", "sts:TagSession"] > } > ``` > [accordion] SSL certificate errors on chat URLs > > The ACM certificate must include `*.chat.` as a subject alternative name. The chat UI generates per-thread subdomains (e.g., `moccasin-prawn.chat.`) that are not covered by `*.`. > > Request a new certificate with: > > ```bash theme={"dark"} > aws acm request-certificate \ > --domain-name "" \ > --subject-alternative-names "*." "*.chat." \ > --validation-method DNS \ > --region --profile > ``` ## API keys > [accordion] API key not being picked up after helm upgrade > > The `xpander-static` secret has a Helm resource keep policy, so a `helm upgrade` does not update it on its own. Patch the key directly on the `xpander-static` Kubernetes secret in the `xpander` namespace: > > ```bash theme={"dark"} > # Set the key directly in the Kubernetes secret > kubectl patch secret xpander-static -n xpander --type=merge \ > -p "{\"data\":{\"ANTHROPIC_API_KEY\":\"$(echo -n '' | base64)\"}}" > > # Restart the worker to pick up the new key > kubectl rollout restart deployment xpander-agent-worker -n xpander > ``` > > See the [secret field name mapping](https://docs.xpander.ai/self-hosted/hybrid-llm-keys.md#secret-field-names) for all key names. > [accordion] New runs refused: The xpander AI gateway is unavailable right now > > New agent runs wait for a healthy LLM Gateway. Check its pods and the reason they are not ready (an image the cluster cannot pull is the usual one): > > ```bash theme={"dark"} > kubectl -n get pods -l app=llm-gw > kubectl -n describe pods -l app=llm-gw > ``` > > See [LLM Gateway](https://docs.xpander.ai/self-hosted/hybrid-llm-gateway.md#health) for the healthy result. > > If you mirror images, mirror `xpanderaihub/llm-gw` and point `images.llmGw` or `global.imageRegistry` at it. The gateway cannot be turned off, so new runs wait until its pods are ready. > [accordion] LLM calls fail with a missing API key > > A call to a provider with no key, neither in the organization (**Platform > AI governance > Models**) nor on the install (`secrets.static`), is refused with a message that names the missing key. Add the key in either place. To check the install's keys, verify they reach the `agent-worker` pod: > > ```bash theme={"dark"} > kubectl -n xpander exec deployment/xpander-agent-worker -- env | grep API_KEY > ``` > > If missing, re-run the upgrade with the key set (`--set secrets.static.=...`), or check that your secret is correctly mounted through `envFromSecretKeys`: see [Model keys and Bedrock](https://docs.xpander.ai/self-hosted/hybrid-llm-keys.md#using-kubernetes-secrets-recommended-for-production). ## Debug commands ```bash theme={"dark"} # Get all resources kubectl -n xpander get all # Check events kubectl -n xpander get events --sort-by=.metadata.creationTimestamp # Describe problematic pods kubectl -n xpander describe pod # Check service endpoints kubectl -n xpander get endpoints ``` This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.