LiteLLM: an in-house AI gateway with virtual keys and budgets

Once three or more teams in a company use LLMs, these problems show up almost guaranteed: root API keys pasted across repos, nobody knows which team spends how much, every team writes its own retry/fallback, and switching providers means editing code in ten places. LiteLLM (repo BerriAI/litellm) is a popular answer: an open-source, self-hosted AI Gateway that lets you call 100+ LLM providers (Anthropic, OpenAI, Gemini, Bedrock, Azure…) through one OpenAI-style format.

Two ways to use it

SDK: one function for every provider

from litellm import completion

resp = completion(
    model="anthropic/claude-sonnet-5-5",   # switching provider = changing this string
    messages=[{"role": "user", "content": "Summarise the following ticket..."}],
)
print(resp.choices[0].message.content)

Responses come back in OpenAI format, so downstream code doesn't need to know which model was called.

Proxy: minimal configuration

# config.yaml
model_list:
  - model_name: claude-main
    litellm_params:
      model: anthropic/claude-sonnet-5-5
      api_key: os.environ/ANTHROPIC_API_KEY

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY
  database_url: os.environ/DATABASE_URL   # Postgres, required for virtual keys

litellm --config config.yaml     # listens on port 4000 by default

Apps just point an OpenAI client at http://<gateway>:4000 and use the name claude-main. To change model, edit only config.yaml, not the app code.

Virtual keys: per-team keys with budgets

Instead of handing out the root API key, an admin uses the master key to generate child keys for each team/app:

curl http://gateway:4000/key/generate \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "models": ["claude-main"],
    "max_budget": 50,
    "duration": "30d",
    "rpm_limit": 60,
    "tpm_limit": 100000,
    "metadata": {"team": "support-bot"}
  }'

• max_budget (USD), rpm_limit/tpm_limit and duration are set directly on the key.
• Attach keys to a user_id or team_id to track spend; check via /key/info, /user/info, /team/info.
• Block/unblock keys instantly with /key/block, /key/unblock. Model aliases let you upgrade/downgrade models without clients knowing.

Practical benefits

• No exposed root keys: provider keys live only in the gateway; if a child key leaks, block just that key.
• Clear chargeback: know exactly how much each team/feature spends.
• Fallback and load balancing: when one provider is overloaded or failing, switch to another deployment.
• Central observability: send logs to Langfuse, MLflow… from one place.

Risks and caveats

• Single point of failure: if the gateway goes down, every AI app goes down. Run multiple replicas, use the -stable image tag (the README says it passed a 12-hour load test), add health checks.
• The gateway holds every key and sees every prompt: protect it like a secrets system: internal network, restricted dashboard access, regular patching.
• Advanced features may be commercial: SSO and some features such as key rotation with grace periods are in the Enterprise tier. Check before designing around them.
• Losing provider-specific features: normalising to OpenAI format is convenient but can hide specific parameters (thinking, caching…). Check the /messages endpoint or pass-through when you need Claude-specific features.
• Installation: install only one of litellm or litellm-core in an environment, as they overwrite each other's files.

How to apply it

• Dev: Use the SDK if a single app needs to switch between a few models. Don't stand up a proxy before you need one.
• Tech Lead / Platform: Once three or more teams use LLMs, run the proxy as the single gateway: per-team virtual keys, monthly budgets, central logs, and root keys removed from repos.
• PM: The gateway gives cost data per feature: the basis for deciding which features deserve the expensive model.