> ## Documentation Index
> Fetch the complete documentation index at: https://docs.xpander.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Hybrid: LLM Gateway

> What the Hybrid chart deploys for the always-on LLM Gateway, its secrets, health check and network paths.

The LLM Gateway (`llm-gw`) runs in your cluster, always on. Every agent model call goes through it, with your allowed models, budgets and usage reporting. Agents reach it at `http://xpander-llm-gw:4000`, the **Base URL** shown in **Platform > AI governance > Models** while the host is unpublished. Full details: [LLM Gateway on a self-hosted install](https://docs.xpander.ai/run/llm-gateway/self-hosted.md).

| What the chart adds | Default |
| - | - |
| Gateway pods | 2 (one on a single-node cluster installed with `install.sh`), HPA from 2 to 20 on CPU. Each requests 0.5 vCPU / 1 GiB and may use up to 1 vCPU / 4 GiB. No database of its own |
| Secrets | `xpander-llm-gw` (master key) and `xpander-llm-gw-content` (seals saved prompt and response text), generated once and kept on upgrade and on uninstall. Bring your own with `llm-gw.masterKey.existingSecret` / `llm-gw.contentKey.existingSecret` |
| Image | `xpanderaihub/llm-gw`, configured like every other image (`global.imageRegistry`, `images.llmGw`, the chart's pull secrets). Mirror it if you mirror images |
| Service account | `llm-gw-sa`. `llm-gw.serviceAccountAnnotations` carries workload identity (IRSA) for Bedrock or Vertex credentials |

> [!WARNING]
>
> Never delete or rotate `xpander-llm-gw-content`: saved prompt and response text cannot be read without it. With Argo CD, ignore `/data` on both gateway Secrets, or set both `existingSecret` values:
>
> ```yaml theme={"dark"}
> spec:
>   ignoreDifferences:
>     - kind: Secret
>       name: xpander-llm-gw
>       jsonPointers: ["/data"]
>     - kind: Secret
>       name: xpander-llm-gw-content
>       jsonPointers: ["/data"]
> ```

## Model keys

Agents on a Hybrid install never use xpander's keys. Each provider needs a key in the organization (**Platform > AI governance > Models**) or on the install (`secrets.static`), or calls are refused with the missing key named. Set install keys as shown on [Model keys and Bedrock](https://docs.xpander.ai/self-hosted/hybrid-llm-keys.md). Custom OpenAI-compatible endpoints keep working as configured.

## Health

New agent runs wait for a healthy gateway. If its pods cannot start (for example, the image cannot be pulled), new runs are refused with "The xpander AI gateway is unavailable right now" until the pods are ready:

```bash theme={"dark"}
kubectl -n <namespace> get pods -l app=llm-gw
```

A healthy gateway shows two `llm-gw` pods (one on a single-node cluster installed with `install.sh`), each `1/1` and `Running`.

## Network

With `networkPolicy.enabled`, the gateway gets its own policy: ingress on port 4000 only, egress to DNS, agent-controller, Redis and public model providers over HTTPS, never a private or link-local address. Add what else it must reach:

```yaml theme={"dark"}
llm-gw:
  networkPolicy:
    providerCidrs: []        # model endpoints on your own network, any port
    cloudEndpointCidrs: []   # private cloud endpoints such as Bedrock or Azure, port 443
    redisCidrs: []           # an external Redis
  env:                       # the gateway does not read global.env
    HTTPS_PROXY: http://<proxy-host>:<port>
    NO_PROXY: <hosts that bypass the proxy>
```

Your own firewall must let the gateway reach the same model providers agent-controller reaches today.

## Publishing `llm-gw.<domain>`

Only for coding clients and scripts in your network (agents never need it): set `llm-gw.ingress.enabled: true` (it renders only with the chart-wide `ingress.enabled`). You then need a DNS record for the host and a certificate that covers it (the chart reuses `ingress.tls.secretName`). Any load balancer in front needs an idle timeout of at least 1500 seconds (ALB: `idle_timeout.timeout_seconds=1500`). `llm-gw.ingress.internal: true` serves the host on a private load balancer (`internalClassName`, `internalAnnotations`). While no gateway pod is ready, a small `xpander-llm-gw-unavailable` pod answers the published host with a JSON 503 that coding clients and SDKs retry. Set `llm-gw.unavailable.replicas: 2` to keep it through a node loss.

## It cannot be turned off

From chart 0.22.5 the chart ignores `llm-gw.enabled` (0.22.0 to 0.22.4 still read it). Every `helm upgrade`, including `--reuse-values`, starts the gateway.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.