> ## Documentation Index > Fetch the complete documentation index at: https://docs.xpander.ai/llms.txt > Use this file to discover all available pages before exploring further. # Models and local inference > Point an air-gapped install at your LLM endpoint: a local OpenAI-compatible runtime such as vLLM, Amazon Bedrock without a key, or a hosted provider your egress allows. An air-gapped install calls the LLM endpoint you choose: a local OpenAI-compatible runtime, Amazon Bedrock over PrivateLink, or a hosted provider if your egress allows. Declare the defaults at install, then register any local endpoint under **Platform > AI governance > Models**. ## Local LLM inference xpander speaks the OpenAI-compatible API, so any local runtime that exposes it plugs in: | Runtime | Models | Notes | | - | - | - | | vLLM | Llama, Mistral, Qwen | High-throughput inference. OpenAI-compatible API | | Ollama | Llama, Mistral, Phi, Qwen | Easier to operate. Good for smaller deployments | | NVIDIA NIM | Llama, Mistral, others | Optimized GPU runtime. Enterprise support | | AWS Bedrock (private) | Anthropic, Mistral, Llama | Only for VPC deployments with PrivateLink to Bedrock: *not* fully air-gapped | Size the inference pool separately: it is your infrastructure, not part of the chart. Hosted providers also work if your egress allows, with the same [LLM provider keys](https://docs.xpander.ai/self-hosted/hybrid-llm-keys.md#managing-llm-api-keys). Two models are declared at install: the agents' default and the **gateway model** that filters requests first (`global.llm.gateway.{provider,model,apiBase}`). Each install or upgrade converges the organization's defaults to the chart's declaration only when that declaration changed, so defaults an admin edited in the UI survive upgrades. Summaries around agent work (thread titles, context compaction, evaluation judges) use the org default model, backed by the install's other providers. None dials a public endpoint on a sealed install. ### Amazon Bedrock without a key In images built after 27 August 2026, chat, tasks and their summaries use the pods' own AWS identity (EKS Pod Identity or IRSA): leave `global.llm.bedrock.bearerTokenSecret` empty and set `global.llm.bedrock.region`. * The pod role needs `bedrock:InvokeModel`, `bedrock:InvokeModelWithResponseStream`, `bedrock:Converse` and `bedrock:ConverseStream` on `*` (the `global.*` profiles route across regions), bound to both `default` and `agent-controller` ServiceAccounts. No second role is assumed. * The account must be opted in to Bedrock data retention for those models in every region a `global.` profile spans. A sealed network needs VPC endpoints for `bedrock-runtime`, `sts` and `eks-auth`. * A Bedrock API key in `bearerTokenSecret` is the alternative. * The "IAM role" method under **Platform > AI governance > Models** applies to the Claude Code leg only. If such an entry exists and chat fails, remove it and restart `agent-controller`, `agent-worker`, `agents`, `actions` and `functions`. > [accordion] A working keyless declaration, from a customer install on EKS > > ```yaml theme={"dark"} > global: > llm: > defaults: > provider: amazon_bedrock > model: global.anthropic.claude-sonnet-5 > gateway: > provider: amazon_bedrock > model: global.anthropic.claude-sonnet-4-6 > bedrock: > region: > ``` > > Leave `bearerTokenSecret` and `assumeRoleArn` unset, put the Bedrock permissions on the role the pods run as (`xpander-sa-role` there), bind it to `default` and `agent-controller`, and upgrade. No separate Bedrock role is needed. **Claude Code on Bedrock**: the CLI calls the in-cluster LLM Gateway directly, and the gateway gets the Bedrock credential from the Agent Controller, the only place assume-role applies: `global.llm.bedrock.assumeRoleArn` (plus `assumeRoleExternalId`), or with no token and no role, the pod's own credentials. `global.llm.bedrock.baseUrl` points it at your VPC endpoint. Fable- and Mythos-class models also need the per-region data-retention opt-in. See [AI vendors](https://docs.xpander.ai/run/llm-gateway/providers-and-keys.md#amazon-bedrock). ## A worked vLLM setup on EKS Auto Mode Checked end to end on an Air-Gapped install: vLLM serving a small instruct model on a T4, registered as a custom provider. The cluster side is yours. xpander needs only the address. **GPU capacity.** A `NodePool` limited to `g4dn` with a `nvidia.com/gpu` taint gets a T4 node in about two minutes, exposing `nvidia.com/gpu: 1` with no device plugin. It needs `nodeClassRef` `group: eks.amazonaws.com`, `kind: NodeClass`, `name: default`. Keep weights on a PersistentVolumeClaim mounted as `HF_HOME` (pre-loaded on a sealed network). **Five things that stop vLLM:** 1. A Service named `vllm` injects `VLLM_PORT=tcp://...` and vLLM refuses to start (`VLLM_PORT ... appears to be a URI`). Set `enableServiceLinks: false` or rename the Service. 2. A T4 needs `--dtype=half`. vLLM 0.11 crash-loops on a T4. `vllm/vllm-openai:v0.9.2` works. 3. Pass the key as `VLLM_API_KEY`, not `--api-key` (which logs it). It guards only `/v1`. `/health`, `/metrics` and `/tokenize` answer without it, so keep port 8000 on a `ClusterIP` Service, never an ingress or load balancer. 4. Start with `--enable-auto-tool-choice --tool-call-parser=hermes`, or **Add provider** marks the model as unable to run agents. 5. Leave room above the composed prompt. `--max-model-len=8192` failed with `400 This model's maximum context length is 8192 tokens. However, you requested 8298 tokens`, shown as **Unknown model error** (controller log: `Non-retryable model provider error: Unknown model error` next to `[prompt-budget] system_message_tok=...`). `--max-model-len=32768` ran. Custom models also log `unknown model.id ... defaulting to 128,000 window`. > [accordion] The manifests: NodePool, PersistentVolumeClaim, Deployment and Service > > The manifests, with account-specific values replaced by placeholders: > > ```yaml theme={"dark"} > apiVersion: karpenter.sh/v1 > kind: NodePool > metadata: > name: gpu > spec: > template: > metadata: > labels: > workload: gpu > spec: > nodeClassRef: > group: eks.amazonaws.com > kind: NodeClass > name: default > requirements: > - key: eks.amazonaws.com/instance-family > operator: In > values: ["g4dn"] > - key: eks.amazonaws.com/instance-size > operator: In > values: ["xlarge", "2xlarge"] > - key: karpenter.sh/capacity-type > operator: In > values: ["on-demand"] > - key: kubernetes.io/arch > operator: In > values: ["amd64"] > taints: > - key: nvidia.com/gpu > value: "present" > effect: NoSchedule > terminationGracePeriod: 10m > limits: > cpu: "16" > nvidia.com/gpu: "2" > disruption: > consolidationPolicy: WhenEmpty > consolidateAfter: 10m > --- > apiVersion: v1 > kind: PersistentVolumeClaim > metadata: > name: vllm-models > namespace: > spec: > accessModes: ["ReadWriteOnce"] > storageClassName: > resources: > requests: > storage: 40Gi > --- > apiVersion: apps/v1 > kind: Deployment > metadata: > name: vllm > namespace: > labels: { app: vllm } > spec: > replicas: 1 > strategy: { type: Recreate } > selector: { matchLabels: { app: vllm } } > template: > metadata: > labels: { app: vllm } > spec: > enableServiceLinks: false # a Service named vllm would inject VLLM_PORT=tcp://... and stop vLLM > nodeSelector: > workload: gpu > tolerations: > - key: nvidia.com/gpu > operator: Exists > effect: NoSchedule > containers: > - name: vllm > image: docker.io/vllm/vllm-openai:v0.9.2 # from your mirror; 0.11 does not run on a T4 > args: > - --model=Qwen/Qwen2.5-1.5B-Instruct > - --served-model-name=qwen2.5-1.5b-instruct > - --dtype=half > - --max-model-len=32768 # 8192 fails the first turn: the composed prompt is larger > - --gpu-memory-utilization=0.90 > - --host=0.0.0.0 > - --port=8000 > - --enable-auto-tool-choice # xpander's Add provider dialog probes for this > - --tool-call-parser=hermes > env: > - name: HF_HOME > value: /models > - name: VLLM_API_KEY # the variable, not --api-key: the flag logs the key > valueFrom: > secretKeyRef: { name: vllm-api-key, key: key } > ports: > - containerPort: 8000 > resources: > limits: > nvidia.com/gpu: 1 > memory: 12Gi > requests: > cpu: "2" > memory: 8Gi > volumeMounts: > - { name: models, mountPath: /models } > - { name: shm, mountPath: /dev/shm } > readinessProbe: > httpGet: { path: /health, port: 8000 } > initialDelaySeconds: 60 > periodSeconds: 15 > failureThreshold: 60 > volumes: > - name: models > persistentVolumeClaim: { claimName: vllm-models } > - name: shm > emptyDir: { medium: Memory, sizeLimit: 2Gi } > --- > apiVersion: v1 > kind: Service > metadata: > name: vllm > namespace: > spec: > selector: { app: vllm } > ports: > - port: 8000 > targetPort: 8000 > ``` ## Register the endpoint in xpander 1. **Platform > AI governance > Models**, **Add provider**. 2. **Name** is free text, and **ID (used in API calls)** derives from it and is fixed. **Protocol**: **OpenAI-compatible**. **Base URL**: the address the pods reach, for example `http://vllm..svc.cluster.local:8000/v1`. In-cluster addresses work on Hybrid and Air-Gapped. xpander cloud accepts only a public `https` address. 3. **API key**: **No key (local or unauthenticated server)**, **Use an existing key**, or **Create a new key** with a **Key name** and the **API key** (the `VLLM_API_KEY` value). 4. **Test connection** shows **Reachable** and the served models. A `tools` badge marks models that accept function calls. **Add** the model, with an optional **Display name**. 5. Click **Add provider**. 6. In the provider's **Keys and models…**, tick the models the organization may use. Prices (USD per 1M tokens, via **Edit model**) track spend and are never charged. **Who may use it…** sets access. New models wait under **New models** unless **Approve new models automatically** is on. **Agent readiness** shows, from the last **Test model**, which harnesses work with each model. ![Models page with the xpander.ai vllm provider panel open, showing Keys, LLM gateway, New models with Approve new models automatically, a Models list with muse-glimmer ticked and its prices](https://mintcdn.com/xpanderai-099931d1/wqIIXSsh0kqA2hZ_/images/verify/lab-llm-model-cost.png?fit=max\&auto=format\&n=wqIIXSsh0kqA2hZ_\&q=85\&s=8d73c56a36da4da694142831c00a1983) Keys and models on a custom provider: the key in use, New models waiting for approval, the models the organization may use with their prices. Shown with sample data and a different model than the one deployed above. `POST /v1/agents` accepts only built-in providers in `model_provider` (a custom ID gets `422`). Pick a custom provider per conversation in **Harness and model**. The header then reads, for example, **OpenCode · Qwen2.5 1.5B (lab T4)**, the controller logs `gateway turn start` with `model=:/`, and vLLM logs `POST /v1/chat/completions 200`. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.