# LiteLLM Admin Guide

This guide is for the operators, platform engineers, and architects who **deploy and run the LiteLLM AI Gateway (Proxy Server)** for a team or organization. It covers architecture, deployment topology, the full configuration reference, authentication and access control, spend tracking, observability, guardrails, security hardening, operations, and the internals you need to extend or troubleshoot LiteLLM.

For end-user SDK and client usage, see the **LiteLLM User Guide**.

- **Project:** [BerriAI/litellm](https://github.com/BerriAI/litellm) — Open Source AI Gateway for 100+ LLMs
- **Version documented:** v1.88.0 (commit `9196098`)
- **License/Tier:** Open source (MIT-style) core, with an optional Enterprise tier (`litellm[enterprise]` / `enterprise/`)

---

## 1. Executive Summary

LiteLLM is two cooperating products built on one routing engine:

1. **The Python SDK** (`litellm`) — a library that translates OpenAI-format calls to 100+ provider APIs.
2. **The AI Gateway / Proxy Server** (`litellm[proxy]`) — a FastAPI service that wraps the SDK and adds the controls an organization needs to safely expose many models to many consumers: **virtual API keys, budgets and rate limits, spend tracking, load balancing and fallbacks, guardrails, observability, secret management, and an admin dashboard**.

As an operator you run the gateway, define a `model_list` mapping public model names to upstream deployments, issue scoped virtual keys to teams and users, and wire up logging, caching, and a database for persistence.

> [!NOTE]
> The gateway is **OpenAI-compatible**: consumers use the standard OpenAI SDK pointed at your gateway URL with a virtual key. Provider credentials never leave the gateway.

---

## 2. Architecture

```
                          ┌─────────────────────────────────────────────┐
   OpenAI SDK / curl ───▶ │             LiteLLM Proxy (FastAPI)          │
   (virtual key sk-…)     │                                             │
                          │  ┌── Auth & RBAC ──┐  ┌── Hooks/Middleware ─┐│
                          │  │ master key,     │  │ guardrails,         ││
                          │  │ virtual keys,   │  │ logging callbacks,  ││
                          │  │ teams, users,   │  │ spend tracking,     ││
                          │  │ orgs, budgets,  │  │ caching             ││
                          │  │ SSO / JWT       │  └─────────────────────┘│
                          │  └─────────────────┘                         │
                          │            │                                 │
                          │            ▼                                 │
                          │     ┌──────────────┐   routing strategy,     │
                          │     │   Router     │   retries, fallbacks,   │
                          │     │  (engine)    │   tpm/rpm, cooldowns    │
                          │     └──────┬───────┘                         │
                          │            │ dispatch                        │
                          │            ▼                                 │
                          │   ┌────────────────────────────────────┐    │
                          │   │  Provider adapters (litellm/llms/*) │    │
                          │   │  openai│anthropic│gemini│bedrock│…  │    │
                          │   └────────────────────────────────────┘    │
                          └───────┬───────────────────┬─────────────────┘
                                  │                   │
                     ┌────────────▼──────┐   ┌────────▼─────────┐
                     │  PostgreSQL        │   │  Redis (optional) │
                     │  (Prisma: keys,    │   │  cache + cross-   │
                     │  spend, teams,     │   │  worker counters  │
                     │  models, configs)  │   └──────────────────┘
                     └────────────────────┘
                                  │
                        ┌─────────▼─────────┐
                        │  Logging sinks     │  Langfuse, Datadog,
                        │  (callbacks)       │  Prometheus, OTel, …
                        └────────────────────┘
```

### Component inventory

| Component | Source location | Responsibility |
|---|---|---|
| Proxy server (FastAPI app) | `litellm/proxy/proxy_server.py` | HTTP entry point, route registration, request lifecycle. |
| Proxy CLI (server launcher) | `litellm/proxy/proxy_cli.py` | `litellm …` command that boots the server (uvicorn/gunicorn/hypercorn/granian). |
| Router (routing engine) | `litellm/router.py`, `litellm/router_strategy/`, `litellm/router_utils/` | Load balancing, retries, fallbacks, cooldowns, rate limits. |
| Provider adapters | `litellm/llms/<provider>/` | Per-provider request/response translation (123 integrations). |
| Auth & RBAC | `litellm/proxy/auth/` | Virtual keys, master key, teams, users, orgs, SSO, JWT. |
| Caching | `litellm/caching/` | In-memory, Redis, S3, disk caches; cross-worker counters. |
| Integrations (logging) | `litellm/integrations/` | Observability/logging callbacks (Langfuse, Datadog, Prometheus, OTel, …). |
| Secret managers | `litellm/secret_managers/` | Fetch credentials from AWS/GCP/Azure/HashiCorp/CyberArk vaults. |
| Guardrails | `litellm/proxy/guardrails/guardrail_hooks/` | Pre/post-call content and policy enforcement (~45 integrations). |
| DB layer | `litellm-proxy-extras/` (Prisma migrations), `litellm/proxy/db/` | PostgreSQL schema and migrations. |
| Admin dashboard | `ui/litellm-dashboard/` (Next.js) | Web UI for keys, models, spend, teams, settings. |
| Core utils | `litellm/litellm_core_utils/` | Token counting, cost calc, prompt templating, streaming helpers. |

> [!IMPORTANT]
> The **provider adapter layer is the heart of LiteLLM's value** and the part you almost never need to touch. Each provider directory implements a common interface that maps the OpenAI request shape to that provider's native API and back. Treat providers as pluggable backends — configure and select them; don't re-document or modify their internals.

---

## 3. Requirements

| Concern | Requirement |
|---|---|
| Runtime | Python 3.8+ (3.10+ recommended; required for `--run_granian`). |
| Install | `pip install 'litellm[proxy]'` or the official Docker image. |
| Database | PostgreSQL — required for virtual keys, teams, spend tracking, and storing models in the DB. Managed via Prisma. |
| Cache (optional) | Redis — required for cross-worker caching and accurate multi-pod budget/rate-limit counters. |
| Reverse proxy (optional) | Any L7 proxy in front for TLS termination; must **not buffer SSE** (streaming). |
| Provider credentials | Per-provider API keys, supplied via env vars or a secret manager. |

---

## 4. Deployment

### 4.1 Running the server — the `litellm` CLI

The server is started by `litellm` (defined in `litellm/proxy/proxy_cli.py`). The most important flags:

| Flag | Default | Purpose |
|---|---|---|
| `--config`, `-c` | none | Path to `config.yaml` (the primary deployment input). |
| `--host` | `0.0.0.0` | Bind address. |
| `--port` | `4000` | Bind port. |
| `--num_workers` | `1` | Worker processes (uvicorn/gunicorn/granian). |
| `--model`, `-m` | none | Run a single model without a config (dev/quick start). |
| `--detailed_debug` | `False` | Verbose request/response logging. |
| `--debug` | `False` | Debug-level logging. |
| `--run_gunicorn` | `False` | Serve via gunicorn (multi-worker management). |
| `--run_hypercorn` | `False` | Serve via hypercorn (HTTP/2 support). |
| `--run_granian` | `False` | Serve via Granian (Rust ASGI server; needs Python 3.10+). |
| `--granian_threads` | none | Runtime threads per worker under Granian. |
| `--ssl_keyfile_path` / `--ssl_certfile_path` | none | TLS cert/key for direct HTTPS. |
| `--ciphers` | none | TLS ciphers. |
| `--request_timeout` | none | Default completion timeout (seconds). |
| `--drop_params` | off | Drop provider-unsupported params globally. |
| `--telemetry` | `True` | Anonymous usage telemetry; disable with `--telemetry False`. |
| `--use_prisma_db_push` | `False` | Use `prisma db push` instead of `migrate` for schema updates. |
| `--enforce_prisma_migration_check` | `False` | Exit on startup if a DB migration fails. |
| `--use_v2_migration_resolver` | `False` | Opt into the v2 migration resolver (avoids schema thrash during rolling deploys). |
| `--iam_token_db_auth` | `False` | Connect to RDS using an IAM token. |
| `--max_requests_before_restart` | none | Recycle a worker after N requests (memory hygiene). |
| `--keepalive_timeout` | none | uvicorn keep-alive timeout. |
| `--reload` | `False` | Hot reload on code/config change (dev only; incompatible with multi-worker). |
| `--health` | — | Send a chat/completion to every model in the config and report health. |
| `--setup` | `False` | Interactive setup wizard to generate a config and provider keys. |

```bash
# Production-style start
litellm --config /app/config.yaml --port 4000 --num_workers 4 --run_gunicorn
```

> [!CAUTION]
> `--reload` is **dev only** and is incompatible with `--num_workers > 1`, `--run_gunicorn`, and `--run_hypercorn`. Never enable it in production.

### 4.2 Docker

```bash
docker run -p 4000:4000 \
  -e OPENAI_API_KEY=$OPENAI_API_KEY \
  -e DATABASE_URL=postgresql://user:pass@host:5432/litellm \
  -e LITELLM_MASTER_KEY=sk-1234 \
  -v $(pwd)/config.yaml:/app/config.yaml \
  ghcr.io/berriai/litellm:main-latest \
  --config /app/config.yaml --port 4000
```

### 4.3 Helm / Kubernetes

Two Helm assets ship in the repo:

- `deploy/charts/litellm-helm/` — the primary Helm chart (templates, CI, tests).
- `helm/litellm/` — chart with `backend`, `gateway`, and `ui` template groups.

These template the proxy Deployment, Service, Secret (for `DATABASE_URL`/`MASTER_KEY`/provider keys), and an optional PostgreSQL dependency. Provide your `config.yaml` via a ConfigMap and your secrets via a Kubernetes Secret.

### 4.4 Infrastructure-as-code

| Asset | Location | Target |
|---|---|---|
| Terraform (AWS) | `terraform/litellm/aws/` | LiteLLM on AWS. |
| Terraform (GCP) | `terraform/litellm/gcp/` | LiteLLM on GCP. |
| Azure Resource Manager | `deploy/azure_resource_manager/` | Azure Marketplace deployment. |
| CloudFormation | `enterprise/cloudformation_stack/` | Enterprise AWS stack. |

### 4.5 Scaling topology

| Stage | Setup |
|---|---|
| Single node | One container, `--num_workers 1`, SQLite-free (Postgres) or no DB for stateless single-model. |
| Multi-worker | One node, `--num_workers N` (+ `--run_gunicorn`). Add Redis so workers share cache and budget/rate-limit counters. |
| Multi-node | Several replicas behind a load balancer, all pointing at one **shared PostgreSQL** and one **shared Redis**. Use `--use_v2_migration_resolver` during rolling upgrades. |

> [!WARNING]
> In multi-worker / multi-pod deployments, **Redis is required for correct budget and rate-limit enforcement**. Without a shared counter store, each worker tracks spend independently and limits are under-enforced. (See Known Issues for historical Redis spend-counter bugs to watch for.)

---

## 5. Configuration Mastery (`config.yaml`)

The gateway is driven by a single YAML file with these top-level sections. Example skeletons live in `litellm/proxy/example_config_yaml/`.

```yaml
model_list:            # which models exist and where they route
litellm_settings:      # SDK-level behavior (caching, callbacks, retries, params)
general_settings:      # proxy/server-level behavior (auth, DB, master key)
router_settings:       # load-balancing & fallback policy
mcp_servers:           # registered MCP tool servers
environment_variables: # values injected into the process env
```

### 5.1 `model_list` — the model catalog

Each entry maps a **public `model_name`** to an upstream **`litellm_params.model`**:

```yaml
model_list:
  - model_name: gpt-4o                       # public alias clients request
    litellm_params:
      model: openai/gpt-4o                    # real upstream model
      api_key: os.environ/OPENAI_API_KEY      # os.environ/ indirection (never inline secrets)
    model_info:
      id: "openai-gpt4o-primary"

  - model_name: gpt-4o                        # second deployment of same alias → load-balanced
    litellm_params:
      model: azure/my-gpt4o-deployment
      api_key: os.environ/AZURE_API_KEY
      api_base: os.environ/AZURE_API_BASE
      api_version: "2024-02-01"
    tpm: 100000                               # per-deployment rate caps for routing
    rpm: 1000

  - model_name: bedrock-claude
    litellm_params:
      model: "bedrock/us.anthropic.claude-sonnet-4-20250514-v1:0"
      aws_region_name: "us-east-1"
```

| Field | Purpose |
|---|---|
| `model_name` | Public alias; repeat it across entries to create a load-balanced group. |
| `litellm_params.model` | Upstream `provider/model` string. |
| `litellm_params.api_key` / `api_base` / `api_version` | Credentials/endpoint; use `os.environ/VAR` to reference env. |
| `tpm` / `rpm` | Per-deployment tokens/requests-per-minute used by the router. |
| `model_info.id` | Stable deployment ID (for metrics, targeted updates). |
| `model_info.access_groups` | Tag deployments for key/team scoping. |

> [!TIP]
> Always reference secrets with the `os.environ/VAR_NAME` form rather than pasting keys into the YAML. This keeps the config safe to commit and lets a secret manager supply the actual value.

### 5.2 `general_settings` — server & auth

| Key | Purpose |
|---|---|
| `master_key` | The root admin key (`sk-...`). Required to manage keys/teams/models. |
| `database_url` | PostgreSQL connection string (or set `DATABASE_URL` env). |
| `store_model_in_db` | If `true`, models added via API/UI persist in the DB (dynamic catalog). |
| `disable_prisma_schema_update` | Prevent automatic schema migration on startup. |
| `custom_auth` | Path to a custom auth function for bespoke key validation. |
| `require_auth_for_metrics_endpoint` | Require auth on `/metrics`. |
| `pass_through_endpoints` | Define raw pass-through routes to upstream providers. |

```yaml
general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY
  database_url: os.environ/DATABASE_URL
  store_model_in_db: true
```

> [!CAUTION]
> The `master_key` is all-powerful. Store it in a secret manager, never hand it to application clients, and issue **scoped virtual keys** for actual traffic. Rotate it if exposed.

### 5.3 `litellm_settings` — request behavior

| Key | Purpose |
|---|---|
| `drop_params` | Drop provider-unsupported params globally. |
| `cache` / `cache_params` | Enable and configure response caching (e.g. Redis). |
| `callbacks` | Logging/guardrail callbacks run on every request (e.g. `["prometheus"]`). |
| `success_callback` / `failure_callback` | Sinks fired on success/failure (e.g. `["langfuse"]`). |
| `num_retries` | Default retry count. |
| `request_timeout` | Default per-request timeout. |
| `set_verbose` | Verbose SDK logging. |
| `telemetry` | Toggle anonymous telemetry. |
| `context_window_fallbacks` | Fallback models when context window is exceeded. |
| `default_team_settings` | Defaults applied to all teams. |

```yaml
litellm_settings:
  drop_params: true
  cache: true
  cache_params:
    type: redis
    host: os.environ/REDIS_HOST
    port: os.environ/REDIS_PORT
    password: os.environ/REDIS_PASSWORD
  success_callback: ["langfuse"]
  callbacks: ["prometheus"]
  num_retries: 2
```

### 5.4 `router_settings` — load balancing & fallbacks

| Key | Purpose |
|---|---|
| `routing_strategy` | How to pick among deployments (see table below). |
| `fallbacks` | Per-model fallback chains. |
| `context_window_fallbacks` | Context-overflow fallback chains. |
| `enable_tag_filtering` | Route by request tags (e.g. team-scoped models). |
| `redis_host` / `redis_port` / `redis_password` | Shared state for cross-worker routing. |
| `cooldown_time` | How long to bench a failing deployment. |
| `allowed_fails` | Failures before a deployment is cooled down. |

**Routing strategies** (`litellm/router_strategy/`):

| Strategy | Behavior |
|---|---|
| `simple-shuffle` | Random/weighted shuffle (default, honors `rpm`/`weight`). |
| `least-busy` | Send to the deployment with the fewest in-flight requests. |
| `usage-based-routing` | Route by current TPM/RPM usage. |
| `usage-based-routing-v2` | Improved usage-based routing (Redis-backed, recommended for scale). |
| `latency-based-routing` | Prefer the lowest-latency deployment. |
| `cost-based-routing` | Prefer the cheapest deployment for the request. |
| `priority` | Strict priority ordering of deployments. |

LiteLLM also ships **smart routers** under `router_strategy/` — `adaptive_router`, `complexity_router`, `quality_router` — that route based on request complexity/quality signals (see `litellm/proxy/example_config_yaml/adaptive_router_example.yaml`).

```yaml
router_settings:
  routing_strategy: usage-based-routing-v2
  redis_host: os.environ/REDIS_HOST
  fallbacks: [{"gpt-4o": ["bedrock-claude"]}]
  enable_tag_filtering: true
```

### 5.5 `mcp_servers` — MCP tool gateway

```yaml
mcp_servers:
  github:
    transport: "http"
    url: "https://api.githubcopilot.com/mcp/"
  wikipedia:
    transport: "stdio"
    command: "uvx"
    args: ["mcp-server-fetch"]
```

Registered MCP servers become callable via `/chat/completions` (`tools: [{type: mcp, server_url: litellm_proxy/mcp/<name>}]`) and via the gateway's `/mcp/` endpoint.

### 5.6 Secret managers

Instead of env vars, the gateway can pull provider keys from a vault (`litellm/secret_managers/`):

| Secret manager | Module |
|---|---|
| AWS Secrets Manager | `aws_secret_manager`, `aws_secret_manager_v2` |
| Google Secret Manager / KMS | `google_secret_manager`, `google_kms` |
| HashiCorp Vault | `hashicorp_secret_manager` |
| CyberArk | `cyberark_secret_manager` |
| Azure AD token provider | `get_azure_ad_token_provider` |
| Custom | `custom_secret_manager_loader` |

Enable via `general_settings`/env; the gateway resolves `os.environ/`-style references and configured vault paths at startup.

---

## 6. Authentication & Access Control

LiteLLM provides a full RBAC model on top of provider credentials (`litellm/proxy/auth/`).

### 6.1 The key hierarchy

```
master_key (root admin, sk-…)
   └── virtual keys (scoped: models, budget, rpm/tpm, expiry)
         ├── attached to a user
         ├── attached to a team  (team budget & model scope)
         └── attached to an organization (org-level rollup)
```

| Concept | Description |
|---|---|
| **Master key** | Root credential; can call all management endpoints. |
| **Virtual key** | Scoped `sk-...` token issued to consumers; limits models, spend, rate, and lifetime. |
| **User** | An identity (often via SSO) that owns keys and accrues spend. |
| **Team** | A group with shared budget and allowed-model scope; keys inherit team limits. |
| **Organization** | Top-level grouping of teams for budget rollups. |
| **Budget** | A spend cap (with optional reset interval) attachable to keys/users/teams/orgs. |

### 6.2 Issuing a virtual key

```bash
curl -X POST http://your-gateway:4000/key/generate \
  -H "Authorization: Bearer $MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "models": ["gpt-4o", "bedrock-claude"],
    "max_budget": 100,
    "duration": "30d",
    "team_id": "team-eng"
  }'
```

Or via the CLI: `litellm-proxy keys generate --models gpt-4o --spend 100 --duration 30d`.

### 6.3 SSO and JWT

The gateway supports SSO login (`/sso/callback`, `/sso/key/generate`) and JWT-based auth (`/.well-known/jwks.json`, JWT key mapping). This lets you federate identity to Google/Microsoft/OIDC providers and mint keys per authenticated user. Azure managed-identity DB auth and IAM-token RDS auth are supported for credential-free database connections.

> [!IMPORTANT]
> Set `general_settings.master_key` and a `database_url` before enabling keys/teams — the RBAC model is persisted in PostgreSQL. Without a DB, the gateway runs in a stateless single-key mode only.

---

## 7. Spend Tracking & Budgets

Every request through the gateway is costed and attributed to a key/user/team/org. Spend data powers budgets, dashboards, and reporting.

| Surface | Endpoints |
|---|---|
| Per-entity spend | `/spend/logs`, `/spend/calculate`, `/spend/tags`, `/spend/users` |
| Global rollups | `/global/spend`, `/global/spend/report`, `/global/spend/keys`, `/global/spend/models`, `/global/spend/teams`, `/global/spend/tags`, `/global/activity` |
| Daily activity | `/customer/daily/activity`, `/team/daily/activity`, `/organization/daily/activity` |
| Budgets | `/budget/new`, `/budget/update`, `/budget/info`, `/budget/list`, `/budget/delete`, `/budget/settings` |

Budgets can reset on an interval and block requests when exceeded. Cost data is also exported via the Prometheus and other logging callbacks.

> [!WARNING]
> Accurate spend enforcement across multiple workers/pods depends on a **shared Redis** counter. Historically there have been Redis spend-counter inflation/reseed bugs (see Known Issues) — monitor reported spend against provider invoices, and pin to a stable release for budget-critical deployments.

---

## 8. Observability & Logging

Logging is callback-driven. Add sinks to `litellm_settings.callbacks` / `success_callback` / `failure_callback`. Available integrations (`litellm/integrations/`):

| Category | Integrations |
|---|---|
| LLM observability | `langfuse`, `arize`, `opik`, `deepeval`, `weave`, `agentops`, `levo` |
| Metrics / APM | `prometheus`, `datadog`, OpenTelemetry (`otel`, `opentelemetry_utils`) |
| Cloud log sinks | `gcs_bucket`, `gcs_pubsub`, `azure_storage`, `azure_sentinel` |
| Alerting | `SlackAlerting` |
| Cost / FinOps | `cloudzero`, `vantage` |
| Generic / custom | `generic_api`, `generic_prompt_management`, `dotprompt`, `bitbucket`, `gitlab` |

```yaml
litellm_settings:
  success_callback: ["langfuse", "datadog"]
  failure_callback: ["langfuse"]
  callbacks: ["prometheus"]
```

Prometheus metrics are exposed at `/metrics`. Health and readiness probes are at `/health`, `/health/liveliness` (alias `/health/liveness`), `/health/readiness`, and `/health/services`.

---

## 9. Guardrails & Content Security

LiteLLM runs **guardrails** as pre/post-call hooks (`litellm/proxy/guardrails/guardrail_hooks/`), with ~45 integrations:

| Type | Integrations (examples) |
|---|---|
| Dedicated guardrail vendors | `aporia_ai`, `aim`, `lasso`, `pangea`, `pillar`, `noma`, `javelin`, `qualifire`, `enkryptai`, `grayswan`, `hiddenlayer` |
| Cloud / enterprise | `azure` (Content Safety), `microsoft_purview`, `ibm_guardrails`, `model_armor`, `panw_prisma_airs`, `crowdstrike_aidr`, `zscaler_ai_guard`, `cato_networks` |
| Built-in | `litellm_content_filter`, `llm_as_a_judge`, `semantic_guard`, `promptguard`, `prompt_security`, `block_code_execution`, `tool_policy` |
| MCP-specific | `mcp_security`, `mcp_jwt_signer`, `mcp_end_user_permission` |
| Custom | `custom_code`, `generic_guardrail_api`, `unified_guardrail` |

Guardrails are registered and managed via `/guardrails/*` endpoints (`/guardrails/list`, `/guardrails/register`, `/guardrails/{guardrail_id}`, plus a submission/approval workflow and per-guardrail usage analytics). Apply ad-hoc checks with `/apply_guardrail`.

> [!NOTE]
> Some guardrails are recompiled on update. Be aware that certain guardrails historically required a reinit to pick up rule changes made via `PUT /guardrails` — verify enforcement after editing rules.

---

## 10. Caching

Caching (`litellm/caching/`) avoids re-hitting providers for identical requests and backs cross-worker counters.

| Backend | Use |
|---|---|
| In-memory | Single-worker dev/test. |
| Redis | Production; shared across workers/pods. Also required for accurate budgets and usage-based routing. |
| S3 / GCS | Durable response cache. |
| Disk / dual | Local persistence; layered caches. |

Cache management endpoints: `/cache/ping`, `/cache/delete`, `/cache/flushall`, `/cache/redis/info`, `/cache/settings`.

```yaml
litellm_settings:
  cache: true
  cache_params:
    type: redis
    host: os.environ/REDIS_HOST
    port: os.environ/REDIS_PORT
    password: os.environ/REDIS_PASSWORD
    ttl: 600
```

---

## 11. Database & Migrations

Persistence uses **PostgreSQL via Prisma**. Migrations live in `litellm-proxy-extras/litellm_proxy_extras/migrations/` (timestamped directories), covering keys, spend logs, daily aggregate tables, teams, MCP servers, guardrails, prompts, policies, and more.

- On startup the gateway applies migrations (unless `disable_prisma_schema_update` / `--use_prisma_db_push` is set).
- Use `--enforce_prisma_migration_check` to **fail fast** if a migration fails instead of starting in a degraded state.
- Use `--use_v2_migration_resolver` during rolling upgrades where two LiteLLM versions briefly share the DB, to avoid schema thrashing.
- `db_scripts/` and `ci_cd/run_migration.py` provide migration tooling (the latter refuses destructive `DROP` diffs unless `--allow-destructive` is passed).

> [!CAUTION]
> Run database migrations as a controlled step in CI/CD, not implicitly during a high-traffic rolling deploy. Back up before destructive migrations.

---

## 12. Operations

### 12.1 Health checks

| Endpoint | Purpose |
|---|---|
| `/health` | Runs a real request against configured models (gated; can be heavy). |
| `/health/liveliness` (`/health/liveness`) | Process is alive. |
| `/health/readiness` | Ready to serve (DB/cache reachable). |
| `/health/services` | Status of dependent services. |
| `/health/latest`, `/health/history` | Recent health-check results. |

Enable `background_health_checks` and `health_check_interval` in `litellm_settings` to poll model health continuously rather than on-request.

### 12.2 Config and model management at runtime

With `store_model_in_db: true`, admins add/update/remove models without restarts:

- Models: `/model/new`, `/model/update`, `/model/delete`, `/model/info`, `/model/metrics`.
- Config fields: `/config/update`, `/config/field/update`, `/config/list`, `/config/yaml`.

### 12.3 Routine procedures

- **Start/stop:** systemd unit or container orchestration around `litellm --config …`.
- **Logs:** stdout/stderr; raise verbosity with `--detailed_debug`.
- **Worker recycling:** `--max_requests_before_restart` to bound memory growth.
- **Backups:** snapshot PostgreSQL (keys, spend, configs) regularly.

---

## 13. Management API Reference (grouped)

The proxy exposes ~700 routes. The OpenAI-compatible inference routes are covered in the User Guide; the **management/admin** surface (master-key or admin-role) is grouped below.

| Group | Representative endpoints |
|---|---|
| **Keys** | `/key/generate`, `/key/info`, `/key/update`, `/key/delete`, `/key/list`, `/key/block`, `/key/unblock`, `/key/regenerate`, `/key/health`, `/key/bulk_update` |
| **Users** | `/user/new`, `/user/info`, `/user/update`, `/user/delete`, `/user/list` |
| **Teams** | `/team/new`, `/team/info`, `/team/update`, `/team/delete`, `/team/list`, `/team/member_add`, `/team/member_delete`, `/team/available` |
| **Organizations** | `/organization/new`, `/organization/info`, `/organization/update`, `/organization/delete`, `/organization/list`, `/organization/member_add` |
| **Customers** | `/customer/new`, `/customer/info`, `/customer/list`, `/customer/update`, `/customer/block`, `/customer/unblock` |
| **Budgets** | `/budget/new`, `/budget/update`, `/budget/info`, `/budget/list`, `/budget/delete`, `/budget/settings` |
| **Models** | `/model/new`, `/model/update`, `/model/delete`, `/model/info`, `/model/metrics`, `/model_group/info` |
| **Spend** | `/spend/logs`, `/spend/calculate`, `/spend/tags`, `/global/spend/*` |
| **Guardrails** | `/guardrails/list`, `/guardrails/register`, `/guardrails/{id}`, `/apply_guardrail` |
| **Config** | `/config/update`, `/config/field/update`, `/config/list`, `/config/yaml`, `/config/pass_through_endpoint` |
| **Cache** | `/cache/ping`, `/cache/delete`, `/cache/flushall`, `/cache/redis/info` |
| **Health** | `/health`, `/health/readiness`, `/health/liveliness`, `/health/services` |
| **Auth/SSO** | `/sso/key/generate`, `/sso/callback`, `/.well-known/jwks.json`, `/jwt/key/mapping/delete` |

> [!NOTE]
> Inference routes (`/chat/completions`, `/embeddings`, `/responses`, `/messages`, `/rerank`, `/audio/*`, `/images/*`, `/moderations`, `/models`) and their `/v1/...` aliases accept any valid virtual key; management routes require the master key or an admin-role key.

---

## 14. Security Analysis

| Surface | Risk | Mitigation |
|---|---|---|
| Master key | Full control of keys, models, spend. | Store in a secret manager; never expose; rotate on exposure; restrict the management API by network policy. |
| Virtual keys | Over-broad scope leaks model access/budget. | Scope every key to specific models, teams, budgets, and expiry. Block/regenerate compromised keys. |
| Provider credentials | Leak grants direct provider billing. | Use `os.environ/`/secret-manager references; never commit keys; rotate regularly. |
| Database | Holds keys, spend, configs. | Use TLS + strong auth (or IAM/Managed Identity); restrict network access; back up. |
| Metrics endpoint | `/metrics` may expose usage detail. | Set `require_auth_for_metrics_endpoint: true`. |
| Pass-through endpoints | Raw proxying can bypass controls. | Define `pass_through_endpoints` narrowly; apply guardrails. |
| MCP / tool calling | Tools can execute actions. | Use MCP guardrails (`mcp_security`, `mcp_end_user_permission`) and `require_approval`. |
| Streaming through L7 proxies | Buffering breaks SSE; misconfig can leak. | Disable response buffering for the gateway path. |

> [!CAUTION]
> Client-supplied fields can, in some configurations, influence `model_info`/`extra_body`. Validate and constrain client metadata, and keep the gateway updated — provider/parameter-handling regressions are fixed frequently (see Known Issues).

---

## 15. Internals (for contributors & extenders)

### Repository structure

| Path | Contents |
|---|---|
| `litellm/` | Core SDK: `main.py` (public functions), `router.py`, `__init__.py` (exports/config). |
| `litellm/llms/<provider>/` | The 123 provider adapters — the pluggable backend layer. |
| `litellm/litellm_core_utils/` | Token counting, cost calc (`llm_cost_calc/`), prompt templating, streaming. |
| `litellm/proxy/` | The FastAPI gateway: `proxy_server.py`, `proxy_cli.py`, `auth/`, `hooks/`, `guardrails/`, `*_endpoints.py`, `client/cli/` (the `litellm-proxy` client). |
| `litellm/router_strategy/`, `router_utils/` | Routing engine and strategies. |
| `litellm/integrations/` | Logging/observability callbacks. |
| `litellm/secret_managers/` | Vault integrations. |
| `litellm/types/` | Pydantic types for requests, responses, providers. |
| `litellm-proxy-extras/` | Prisma schema + DB migrations. |
| `enterprise/` | Enterprise-tier hooks, callbacks, and UI. |
| `ui/litellm-dashboard/` | Next.js admin dashboard. |
| `tests/` | Extensive test suite (unit, LLM translation, proxy, router, guardrails, e2e). |
| `cookbook/` | Runnable examples (router load tests, proxy servers, agent SDKs). |
| `deploy/`, `helm/`, `terraform/` | Deployment assets. |

### Adding a provider adapter

A new provider is a directory under `litellm/llms/<provider>/` implementing the shared completion/embedding interface (request transform → upstream call → response transform), registered so the `provider/` prefix resolves to it. Because every provider conforms to the same OpenAI-shaped contract, the router, auth, logging, and gateway layers work with it automatically — you implement only the translation.

### Extending behavior

- **Custom auth:** point `general_settings.custom_auth` at a function that validates keys your way.
- **Custom logging:** implement a `CustomLogger` callback and add it to `callbacks`.
- **Custom guardrail:** drop a hook into `guardrail_hooks/custom_code` or implement the guardrail interface.
- **Custom secret manager:** use `custom_secret_manager_loader`.

---

## 16. Known Issues & Operational Gotchas

These are community-reported issues relevant to operators at/around this version. Track them upstream before relying on the affected feature.

| Area | Issue |
|---|---|
| Redis spend tracking | Spend-counter inflation (3–6× real spend) and non-idempotent reseed across pods have been reported (#28283, #28247). Reconcile reported spend against provider invoices; pin a stable release for budget-critical use. |
| Bedrock model groups | A regression broke Bedrock model groups/aliases in a prior release (#29618). Test Bedrock group routing after upgrades. |
| Gemini/Vertex caching | `tool_choice` could be silently dropped on cached Gemini/Vertex requests (#29088). Verify tool-calling behavior with caching enabled. |
| Strict JSON schema | Some providers (e.g. Fireworks) silently downgraded strict `json_schema` to `json_object` (#29604). Verify schema enforcement per provider. |
| Guardrail reload | `ToolPermissionGuardrail` rule changes via `PUT /guardrails` weren't enforced until reinit (#29592). Confirm enforcement after editing rules. |
| Anthropic streaming | A proxied Anthropic response was reported buffered rather than streamed (#28384). Verify SSE pass-through end-to-end. |
| `model_info` poisoning | Client requests could poison `model_info` when a model config had `extra_body` (#29615). Constrain client-supplied params. |

### Suggested hardening checklist

- [ ] Master key in a secret manager; metrics endpoint authenticated.
- [ ] All traffic on scoped virtual keys with budgets and expiry.
- [ ] Shared Redis for multi-worker budget/rate correctness; reconcile spend vs. invoices.
- [ ] PostgreSQL with TLS/IAM auth and regular backups.
- [ ] Migrations gated in CI/CD with `--enforce_prisma_migration_check`.
- [ ] Guardrails enabled for inbound content and MCP tool use.
- [ ] L7 proxy configured to **not** buffer streaming responses.
- [ ] Pinned, tested release; smoke-test routing/fallbacks and provider quirks after every upgrade.

---

## 17. Quick Reference

| Need | Where |
|---|---|
| Start gateway | `litellm --config config.yaml` |
| Single-model dev | `litellm --model gpt-4o` |
| Health smoke test | `litellm --config config.yaml --health` |
| Setup wizard | `litellm --setup` |
| Define models | `config.yaml` → `model_list` |
| Auth/DB/master key | `config.yaml` → `general_settings` |
| Caching/callbacks/retries | `config.yaml` → `litellm_settings` |
| Load balancing/fallbacks | `config.yaml` → `router_settings` |
| Issue a key | `POST /key/generate` or `litellm-proxy keys generate` |
| View spend | `/global/spend/report`, dashboard |
| Liveness/readiness | `/health/liveliness`, `/health/readiness` |
| Metrics | `/metrics` (Prometheus) |
| Example configs | `litellm/proxy/example_config_yaml/` |

---

_Generated June 2026 from BerriAI/litellm at 9196098. AI-generated documentation. The library is checked by sampling (three repositories so far); this guide was not individually reviewed. Verify against the project before relying on it. Unofficial; not affiliated with or reviewed by the project._