LiteLLM Admin Guide

This guide is for the operators, platform engineers, and architects who deploy and run the LiteLLM AI Gateway (Proxy Server) for a team or organization. It covers architecture, deployment topology, the full configuration reference, authentication and access control, spend tracking, observability, guardrails, security hardening, operations, and the internals you need to extend or troubleshoot LiteLLM.

For end-user SDK and client usage, see the LiteLLM User Guide.

• Project: BerriAI/litellm — Open Source AI Gateway for 100+ LLMs
• Version documented: v1.88.0 (commit 9196098)
• License/Tier: Open source (MIT-style) core, with an optional Enterprise tier (litellm[enterprise] / enterprise/)

1. Executive Summary

LiteLLM is two cooperating products built on one routing engine:

1. The Python SDK (litellm) — a library that translates OpenAI-format calls to 100+ provider APIs.
2. The AI Gateway / Proxy Server (litellm[proxy]) — a FastAPI service that wraps the SDK and adds the controls an organization needs to safely expose many models to many consumers: virtual API keys, budgets and rate limits, spend tracking, load balancing and fallbacks, guardrails, observability, secret management, and an admin dashboard.

As an operator you run the gateway, define a model_list mapping public model names to upstream deployments, issue scoped virtual keys to teams and users, and wire up logging, caching, and a database for persistence.

Note

The gateway is OpenAI-compatible: consumers use the standard OpenAI SDK pointed at your gateway URL with a virtual key. Provider credentials never leave the gateway.

2. Architecture

                          ┌─────────────────────────────────────────────┐
   OpenAI SDK / curl ───▶ │             LiteLLM Proxy (FastAPI)          │
   (virtual key sk-…)     │                                             │
                          │  ┌── Auth & RBAC ──┐  ┌── Hooks/Middleware ─┐│
                          │  │ master key,     │  │ guardrails,         ││
                          │  │ virtual keys,   │  │ logging callbacks,  ││
                          │  │ teams, users,   │  │ spend tracking,     ││
                          │  │ orgs, budgets,  │  │ caching             ││
                          │  │ SSO / JWT       │  └─────────────────────┘│
                          │  └─────────────────┘                         │
                          │            │                                 │
                          │            ▼                                 │
                          │     ┌──────────────┐   routing strategy,     │
                          │     │   Router     │   retries, fallbacks,   │
                          │     │  (engine)    │   tpm/rpm, cooldowns    │
                          │     └──────┬───────┘                         │
                          │            │ dispatch                        │
                          │            ▼                                 │
                          │   ┌────────────────────────────────────┐    │
                          │   │  Provider adapters (litellm/llms/*) │    │
                          │   │  openai│anthropic│gemini│bedrock│…  │    │
                          │   └────────────────────────────────────┘    │
                          └───────┬───────────────────┬─────────────────┘
                                  │                   │
                     ┌────────────▼──────┐   ┌────────▼─────────┐
                     │  PostgreSQL        │   │  Redis (optional) │
                     │  (Prisma: keys,    │   │  cache + cross-   │
                     │  spend, teams,     │   │  worker counters  │
                     │  models, configs)  │   └──────────────────┘
                     └────────────────────┘
                                  │
                        ┌─────────▼─────────┐
                        │  Logging sinks     │  Langfuse, Datadog,
                        │  (callbacks)       │  Prometheus, OTel, …
                        └────────────────────┘

Component inventory

Important

The provider adapter layer is the heart of LiteLLM's value and the part you almost never need to touch. Each provider directory implements a common interface that maps the OpenAI request shape to that provider's native API and back. Treat providers as pluggable backends — configure and select them; don't re-document or modify their internals.

3. Requirements

4. Deployment

4.1 Running the server — the litellm CLI

The server is started by litellm (defined in litellm/proxy/proxy_cli.py). The most important flags:

# Production-style start
litellm --config /app/config.yaml --port 4000 --num_workers 4 --run_gunicorn

Caution

--reload is dev only and is incompatible with --num_workers > 1, --run_gunicorn, and --run_hypercorn. Never enable it in production.

4.2 Docker

docker run -p 4000:4000 \
  -e OPENAI_API_KEY=$OPENAI_API_KEY \
  -e DATABASE_URL=postgresql://user:pass@host:5432/litellm \
  -e LITELLM_MASTER_KEY=sk-1234 \
  -v $(pwd)/config.yaml:/app/config.yaml \
  ghcr.io/berriai/litellm:main-latest \
  --config /app/config.yaml --port 4000

4.3 Helm / Kubernetes

Two Helm assets ship in the repo:

• deploy/charts/litellm-helm/ — the primary Helm chart (templates, CI, tests).
• helm/litellm/ — chart with backend, gateway, and ui template groups.

These template the proxy Deployment, Service, Secret (for DATABASE_URL/MASTER_KEY/provider keys), and an optional PostgreSQL dependency. Provide your config.yaml via a ConfigMap and your secrets via a Kubernetes Secret.

4.4 Infrastructure-as-code

4.5 Scaling topology

Warning

In multi-worker / multi-pod deployments, Redis is required for correct budget and rate-limit enforcement. Without a shared counter store, each worker tracks spend independently and limits are under-enforced. (See Known Issues for historical Redis spend-counter bugs to watch for.)

5. Configuration Mastery (config.yaml)

The gateway is driven by a single YAML file with these top-level sections. Example skeletons live in litellm/proxy/example_config_yaml/.

model_list:            # which models exist and where they route
litellm_settings:      # SDK-level behavior (caching, callbacks, retries, params)
general_settings:      # proxy/server-level behavior (auth, DB, master key)
router_settings:       # load-balancing & fallback policy
mcp_servers:           # registered MCP tool servers
environment_variables: # values injected into the process env

5.1 model_list — the model catalog

Each entry maps a public model_name to an upstream litellm_params.model:

model_list:
  - model_name: gpt-4o                       # public alias clients request
    litellm_params:
      model: openai/gpt-4o                    # real upstream model
      api_key: os.environ/OPENAI_API_KEY      # os.environ/ indirection (never inline secrets)
    model_info:
      id: "openai-gpt4o-primary"

  - model_name: gpt-4o                        # second deployment of same alias → load-balanced
    litellm_params:
      model: azure/my-gpt4o-deployment
      api_key: os.environ/AZURE_API_KEY
      api_base: os.environ/AZURE_API_BASE
      api_version: "2024-02-01"
    tpm: 100000                               # per-deployment rate caps for routing
    rpm: 1000

  - model_name: bedrock-claude
    litellm_params:
      model: "bedrock/us.anthropic.claude-sonnet-4-20250514-v1:0"
      aws_region_name: "us-east-1"

Tip

Always reference secrets with the os.environ/VAR_NAME form rather than pasting keys into the YAML. This keeps the config safe to commit and lets a secret manager supply the actual value.

5.2 general_settings — server & auth

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY
  database_url: os.environ/DATABASE_URL
  store_model_in_db: true

Caution

The master_key is all-powerful. Store it in a secret manager, never hand it to application clients, and issue scoped virtual keys for actual traffic. Rotate it if exposed.

5.3 litellm_settings — request behavior

litellm_settings:
  drop_params: true
  cache: true
  cache_params:
    type: redis
    host: os.environ/REDIS_HOST
    port: os.environ/REDIS_PORT
    password: os.environ/REDIS_PASSWORD
  success_callback: ["langfuse"]
  callbacks: ["prometheus"]
  num_retries: 2

5.4 router_settings — load balancing & fallbacks

Routing strategies (litellm/router_strategy/):

LiteLLM also ships smart routers under router_strategy/ — adaptive_router, complexity_router, quality_router — that route based on request complexity/quality signals (see litellm/proxy/example_config_yaml/adaptive_router_example.yaml).

router_settings:
  routing_strategy: usage-based-routing-v2
  redis_host: os.environ/REDIS_HOST
  fallbacks: [{"gpt-4o": ["bedrock-claude"]}]
  enable_tag_filtering: true

5.5 mcp_servers — MCP tool gateway

mcp_servers:
  github:
    transport: "http"
    url: "https://api.githubcopilot.com/mcp/"
  wikipedia:
    transport: "stdio"
    command: "uvx"
    args: ["mcp-server-fetch"]

Registered MCP servers become callable via /chat/completions (tools: [{type: mcp, server_url: litellm_proxy/mcp/<name>}]) and via the gateway's /mcp/ endpoint.

5.6 Secret managers

Instead of env vars, the gateway can pull provider keys from a vault (litellm/secret_managers/):

Enable via general_settings/env; the gateway resolves os.environ/-style references and configured vault paths at startup.

6. Authentication & Access Control

LiteLLM provides a full RBAC model on top of provider credentials (litellm/proxy/auth/).

6.1 The key hierarchy

master_key (root admin, sk-…)
   └── virtual keys (scoped: models, budget, rpm/tpm, expiry)
         ├── attached to a user
         ├── attached to a team  (team budget & model scope)
         └── attached to an organization (org-level rollup)

6.2 Issuing a virtual key

curl -X POST http://your-gateway:4000/key/generate \
  -H "Authorization: Bearer $MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "models": ["gpt-4o", "bedrock-claude"],
    "max_budget": 100,
    "duration": "30d",
    "team_id": "team-eng"
  }'

Or via the CLI: litellm-proxy keys generate --models gpt-4o --spend 100 --duration 30d.

6.3 SSO and JWT

The gateway supports SSO login (/sso/callback, /sso/key/generate) and JWT-based auth (/.well-known/jwks.json, JWT key mapping). This lets you federate identity to Google/Microsoft/OIDC providers and mint keys per authenticated user. Azure managed-identity DB auth and IAM-token RDS auth are supported for credential-free database connections.

Important

Set general_settings.master_key and a database_url before enabling keys/teams — the RBAC model is persisted in PostgreSQL. Without a DB, the gateway runs in a stateless single-key mode only.

7. Spend Tracking & Budgets

Every request through the gateway is costed and attributed to a key/user/team/org. Spend data powers budgets, dashboards, and reporting.

Budgets can reset on an interval and block requests when exceeded. Cost data is also exported via the Prometheus and other logging callbacks.

Warning

Accurate spend enforcement across multiple workers/pods depends on a shared Redis counter. Historically there have been Redis spend-counter inflation/reseed bugs (see Known Issues) — monitor reported spend against provider invoices, and pin to a stable release for budget-critical deployments.

8. Observability & Logging

Logging is callback-driven. Add sinks to litellm_settings.callbacks / success_callback / failure_callback. Available integrations (litellm/integrations/):

litellm_settings:
  success_callback: ["langfuse", "datadog"]
  failure_callback: ["langfuse"]
  callbacks: ["prometheus"]

Prometheus metrics are exposed at /metrics. Health and readiness probes are at /health, /health/liveliness (alias /health/liveness), /health/readiness, and /health/services.

9. Guardrails & Content Security

LiteLLM runs guardrails as pre/post-call hooks (litellm/proxy/guardrails/guardrail_hooks/), with ~45 integrations:

Guardrails are registered and managed via /guardrails/* endpoints (/guardrails/list, /guardrails/register, /guardrails/{guardrail_id}, plus a submission/approval workflow and per-guardrail usage analytics). Apply ad-hoc checks with /apply_guardrail.

Note

Some guardrails are recompiled on update. Be aware that certain guardrails historically required a reinit to pick up rule changes made via PUT /guardrails — verify enforcement after editing rules.

10. Caching

Caching (litellm/caching/) avoids re-hitting providers for identical requests and backs cross-worker counters.

Cache management endpoints: /cache/ping, /cache/delete, /cache/flushall, /cache/redis/info, /cache/settings.

litellm_settings:
  cache: true
  cache_params:
    type: redis
    host: os.environ/REDIS_HOST
    port: os.environ/REDIS_PORT
    password: os.environ/REDIS_PASSWORD
    ttl: 600

11. Database & Migrations

Persistence uses PostgreSQL via Prisma. Migrations live in litellm-proxy-extras/litellm_proxy_extras/migrations/ (timestamped directories), covering keys, spend logs, daily aggregate tables, teams, MCP servers, guardrails, prompts, policies, and more.

• On startup the gateway applies migrations (unless disable_prisma_schema_update / --use_prisma_db_push is set).
• Use --enforce_prisma_migration_check to fail fast if a migration fails instead of starting in a degraded state.
• Use --use_v2_migration_resolver during rolling upgrades where two LiteLLM versions briefly share the DB, to avoid schema thrashing.
• db_scripts/ and ci_cd/run_migration.py provide migration tooling (the latter refuses destructive DROP diffs unless --allow-destructive is passed).

Caution

Run database migrations as a controlled step in CI/CD, not implicitly during a high-traffic rolling deploy. Back up before destructive migrations.

12. Operations

12.1 Health checks

Enable background_health_checks and health_check_interval in litellm_settings to poll model health continuously rather than on-request.

12.2 Config and model management at runtime

With store_model_in_db: true, admins add/update/remove models without restarts:

• Models: /model/new, /model/update, /model/delete, /model/info, /model/metrics.
• Config fields: /config/update, /config/field/update, /config/list, /config/yaml.

12.3 Routine procedures

• Start/stop: systemd unit or container orchestration around litellm --config ….
• Logs: stdout/stderr; raise verbosity with --detailed_debug.
• Worker recycling: --max_requests_before_restart to bound memory growth.
• Backups: snapshot PostgreSQL (keys, spend, configs) regularly.

13. Management API Reference (grouped)

The proxy exposes ~700 routes. The OpenAI-compatible inference routes are covered in the User Guide; the management/admin surface (master-key or admin-role) is grouped below.

Note

Inference routes (/chat/completions, /embeddings, /responses, /messages, /rerank, /audio/*, /images/*, /moderations, /models) and their /v1/... aliases accept any valid virtual key; management routes require the master key or an admin-role key.

14. Security Analysis

Caution

Client-supplied fields can, in some configurations, influence model_info/extra_body. Validate and constrain client metadata, and keep the gateway updated — provider/parameter-handling regressions are fixed frequently (see Known Issues).

15. Internals (for contributors & extenders)

Repository structure

Adding a provider adapter

A new provider is a directory under litellm/llms/<provider>/ implementing the shared completion/embedding interface (request transform → upstream call → response transform), registered so the provider/ prefix resolves to it. Because every provider conforms to the same OpenAI-shaped contract, the router, auth, logging, and gateway layers work with it automatically — you implement only the translation.

Extending behavior

• Custom auth: point general_settings.custom_auth at a function that validates keys your way.
• Custom logging: implement a CustomLogger callback and add it to callbacks.
• Custom guardrail: drop a hook into guardrail_hooks/custom_code or implement the guardrail interface.
• Custom secret manager: use custom_secret_manager_loader.

16. Known Issues & Operational Gotchas

These are community-reported issues relevant to operators at/around this version. Track them upstream before relying on the affected feature.

Suggested hardening checklist

• [ ] Master key in a secret manager; metrics endpoint authenticated.
• [ ] All traffic on scoped virtual keys with budgets and expiry.
• [ ] Shared Redis for multi-worker budget/rate correctness; reconcile spend vs. invoices.
• [ ] PostgreSQL with TLS/IAM auth and regular backups.
• [ ] Migrations gated in CI/CD with --enforce_prisma_migration_check.
• [ ] Guardrails enabled for inbound content and MCP tool use.
• [ ] L7 proxy configured to not buffer streaming responses.
• [ ] Pinned, tested release; smoke-test routing/fallbacks and provider quirks after every upgrade.

17. Quick Reference