Skip to content

Use an LLM Gateway (LiteLLM)

An LLM gateway such as LiteLLM sits between your AI tools and the model providers. It holds the provider keys, gives each person or project its own key with a budget, and logs usage in one place.

This page shows how to let AI tools in the container reach a gateway, including tools other than Claude Code. The image does not change: you only edit the project's .devcontainer/ files. Claude Code stays installed; the claude CLI makes no network calls unless you start it.

Every step uses commented-out placeholders that ship in the examples/ templates. Search for llm-gateway.example.com and OPENAI_ to find them.

What the gateway needs

  • HTTPS with a publicly trusted certificate. All container traffic passes through the inspection proxy (mitmproxy), which checks the gateway's certificate against the image's system CA bundle. Only root can change that bundle, so a gateway behind a private CA fails TLS. Supporting a private CA needs an image change. Any port works.
  • An OpenAI-compatible endpoint: /v1/chat/completions, and /v1/responses for tools that use it. LiteLLM serves both.
  • Models that support tool calling (function calling). Coding agents don't work without it.
  • One key per person or project. In LiteLLM, that is a virtual key limited to the models the project needs, with a budget and an expiry. Store it in 1Password.

A minimal LiteLLM setup looks like this. See the LiteLLM docs for every option.

# LiteLLM config.yaml (runs on the gateway host, not in this container)
model_list:
  - model_name: coding-model              # the name your tools ask for
    litellm_params:
      model: <provider>/<model-id>        # an Anthropic, OpenAI, Bedrock, Azure, ... model
      api_key: os.environ/PROVIDER_API_KEY
general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY
  database_url: os.environ/DATABASE_URL   # required for virtual keys

# Create a virtual key for one project (run by the gateway admin)
curl -s https://llm-gateway.example.com/key/generate \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{"models": ["coding-model"], "max_budget": 50, "duration": "30d"}'

Step 1: allow the gateway

Add the gateway hostname to both egress layers, as for any other host (see Manage Egress).

.devcontainer/egress-allowlist.txt:

# --- LLM gateway (e.g. LiteLLM) ---
llm-gateway.example.com

.devcontainer/proxy.yaml. The # tag:llm comment makes the dashboard mark the traffic as LLM calls:

allowed:
  - llm-gateway.example.com   # tag:llm

The firewall resolves hostnames once, at container start. For a gateway on an internal network, connect the VPN before starting the container. Otherwise start-up prints WARN: Could not resolve: llm-gateway.example.com and the gateway stays blocked. Reload either layer without a restart:

sudo /usr/local/bin/firewall.sh apply
PROXY_YAML=.devcontainer/proxy.yaml generate-proxy-filter

Hosted gateway: OpenRouter

OpenRouter is a hosted gateway with an OpenAI-compatible API at https://openrouter.ai/api/v1, so there is nothing to run yourself. The templates ship a commented block for it, next to the one for llm-gateway.example.com. Uncomment openrouter.ai in both .devcontainer/egress-allowlist.txt and .devcontainer/proxy.yaml, reload the layers as above, then choose "OpenRouter" as the provider in the Cline extension (the templates install it). Cline takes the key in its own settings, so Step 2 does not apply to it.

OpenRouter sits behind Cloudflare, so its IPs can change; if calls start failing, run sudo /usr/local/bin/firewall.sh apply again. Prompts and code you send leave your network and pass through OpenRouter to the model provider, so check that this fits your data policy first. Use a separate key per person or project, and keep it in 1Password.

Step 2: pass the URL and key

Uncomment the LLM gateway block in .devcontainer/env.sh, and point it at your gateway and your 1Password item:

export "MY_PROJECT_OPENAI_BASE_URL"="https://llm-gateway.example.com/v1"
export "MY_PROJECT_OPENAI_API_KEY"="$(op read "op://${OP_VAULT}/${PROJECT_PREFIX_LOWER}_llm_gateway/api_key")"

Then uncomment the matching lines in .devcontainer/devcontainer.json:

"OPENAI_BASE_URL": "${localEnv:MY_PROJECT_OPENAI_BASE_URL}",
"OPENAI_API_KEY":  "${localEnv:MY_PROJECT_OPENAI_API_KEY}",

Source env.sh again and reopen the container from that same terminal.

Many tools read OPENAI_BASE_URL and OPENAI_API_KEY. Others, including most VS Code extensions, take the URL and key in their own settings instead. Enter the same values there.

Step 3: add an AI tool

VS Code extension (simplest)

Add the extension's Marketplace ID to customizations.vscode.extensions in devcontainer.json, then rebuild the container. The Marketplace is not in the egress allowlist, but VS Code still installs the listed extensions. This was checked in this container: the server's own Marketplace requests are refused, and the extensions still install.

The extension runs inside the container, so its requests go through the proxy chain like everything else.

Command-line tool

The image ships two command-line agents: Claude Code (claude) and the GitHub Copilot CLI (copilot). The Copilot CLI talks to GitHub's own Copilot service, not to a gateway, so it needs the "GitHub Copilot" allowlist entries. You can install another agent into your home directory at runtime if you first add its package registry (for example registry.npmjs.org or pypi.org) to both allowlists. It is lost when the container is rebuilt. To have one permanently, open an issue asking for it to be added to the image.

Before you pick a tool

Check four things:

  1. Custom endpoint. The tool accepts a custom OpenAI-compatible base URL.
  2. AGENTS.md. The tool reads AGENTS.md, or you can point it at that file. The safety rules every agent must follow live in AGENTS.md ## Agent Guardrails. CLAUDE.md only repeats them for Claude Code.
  3. System CA bundle. The tool trusts the system certificates. The container sets SSL_CERT_FILE, CURL_CA_BUNDLE, and NODE_EXTRA_CA_CERTS, which covers Node.js, curl, and most Python tools. A tool that ships its own list of root certificates fails every HTTPS request through the inspection proxy.
  4. Approval mode. The tool can ask before it runs commands. Keep that on, so the confirmations required by the guardrails reach you.

Step 4: verify

# Gateway reachable through both layers (LiteLLM health check)
curl -s -o /dev/null -w '%{http_code}\n' https://llm-gateway.example.com/health/liveliness

# Key accepted; lists the models this key may use
curl -s https://llm-gateway.example.com/v1/models \
  -H "Authorization: Bearer $OPENAI_API_KEY" | jq -r '.data[].id'

Then use the tool once and open the dashboard (http://localhost:9090). The gateway host should appear in the Proxy Feed with the LLM badge. If it does not appear, the tool is not using the proxy.

Optional: send all AI traffic through the gateway

Leave the Anthropic / Claude Code entries commented out in both allowlists. That is the default in the examples/ templates. The firewall then rejects direct calls to Anthropic, so the gateway is the only way to reach a model. Other providers' APIs (such as api.openai.com) are not in the templates at all.

Limitations

  • No token counts. The dashboard's token and cost counters only read Anthropic and GitHub Copilot traffic (mitm/addon.py). Gateway calls appear in the Proxy Feed, without token counts. Use LiteLLM's own spend tracking.
  • No private CA without an image change; see What the gateway needs.
  • Changing gateway IPs. The firewall keeps the IP addresses it resolved at start-up. If the gateway's IPs change (common behind cloud load balancers), run sudo /usr/local/bin/firewall.sh apply again.
  • Claude Code through the gateway is a separate setup and is not covered here. It uses the Anthropic-format endpoint and the ANTHROPIC_BASE_URL and ANTHROPIC_AUTH_TOKEN variables.

See ADR 0018 for why this is configuration-only.