Hybrid: LLM Gateway

The LLM Gateway (llm-gw) runs in your cluster, always on. Every agent model call goes through it, with your allowed models, budgets and usage reporting. Agents reach it at http://xpander-llm-gw:4000, the Base URL shown in Platform > AI governance > Models while the host is unpublished. Full details: LLM Gateway on a self-hosted install.

What the chart adds	Default
Gateway pods	2 (one on a single-node cluster installed with `install.sh`), HPA from 2 to 20 on CPU. Each requests 0.5 vCPU / 1 GiB and may use up to 1 vCPU / 4 GiB. No database of its own
Secrets	`xpander-llm-gw` (master key) and `xpander-llm-gw-content` (seals saved prompt and response text), generated once and kept on upgrade and on uninstall. Bring your own with `llm-gw.masterKey.existingSecret` / `llm-gw.contentKey.existingSecret`
Image	`xpanderaihub/llm-gw`, configured like every other image (`global.imageRegistry`, `images.llmGw`, the chart’s pull secrets). Mirror it if you mirror images
Service account	`llm-gw-sa`. `llm-gw.serviceAccountAnnotations` carries workload identity (IRSA) for Bedrock or Vertex credentials

Never delete or rotate xpander-llm-gw-content: saved prompt and response text cannot be read without it. With Argo CD, ignore /data on both gateway Secrets, or set both existingSecret values:

spec:
  ignoreDifferences:
    - kind: Secret
      name: xpander-llm-gw
      jsonPointers: ["/data"]
    - kind: Secret
      name: xpander-llm-gw-content
      jsonPointers: ["/data"]

Model keys

Agents on a Hybrid install never use xpander’s keys. Each provider needs a key in the organization (Platform > AI governance > Models) or on the install (secrets.static), or calls are refused with the missing key named. Set install keys as shown on Model keys and Bedrock. Custom OpenAI-compatible endpoints keep working as configured.

Health

New agent runs wait for a healthy gateway. If its pods cannot start (for example, the image cannot be pulled), new runs are refused with “The xpander AI gateway is unavailable right now” until the pods are ready:

kubectl -n <namespace> get pods -l app=llm-gw

A healthy gateway shows two llm-gw pods (one on a single-node cluster installed with install.sh), each 1/1 and Running.

Network

With networkPolicy.enabled, the gateway gets its own policy: ingress on port 4000 only, egress to DNS, agent-controller, Redis and public model providers over HTTPS, never a private or link-local address. Add what else it must reach:

llm-gw:
  networkPolicy:
    providerCidrs: []        # model endpoints on your own network, any port
    cloudEndpointCidrs: []   # private cloud endpoints such as Bedrock or Azure, port 443
    redisCidrs: []           # an external Redis
  env:                       # the gateway does not read global.env
    HTTPS_PROXY: http://<proxy-host>:<port>
    NO_PROXY: <hosts that bypass the proxy>

Your own firewall must let the gateway reach the same model providers agent-controller reaches today.

Publishing `llm-gw.<domain>`

Only for coding clients and scripts in your network (agents never need it): set llm-gw.ingress.enabled: true (it renders only with the chart-wide ingress.enabled). You then need a DNS record for the host and a certificate that covers it (the chart reuses ingress.tls.secretName). Any load balancer in front needs an idle timeout of at least 1500 seconds (ALB: idle_timeout.timeout_seconds=1500). llm-gw.ingress.internal: true serves the host on a private load balancer (internalClassName, internalAnnotations). While no gateway pod is ready, a small xpander-llm-gw-unavailable pod answers the published host with a JSON 503 that coding clients and SDKs retry. Set llm-gw.unavailable.replicas: 2 to keep it through a node loss.

It cannot be turned off

From chart 0.22.5 the chart ignores llm-gw.enabled (0.22.0 to 0.22.4 still read it). Every helm upgrade, including --reuse-values, starts the gateway.

Govern

Company Brain

LLM Gateway

Deploy

Integrations

Licensing and identity

Security and compliance

Model keys

Health

Network

Publishing `llm-gw.<domain>`

It cannot be turned off

​Model keys

​Health

​Network

​Publishing llm-gw.<domain>

​It cannot be turned off

Model keys

Health

Network

Publishing `llm-gw.<domain>`

It cannot be turned off