# LiteLLM User Guide

LiteLLM is an open-source **AI Gateway** that gives you one unified, OpenAI-format interface to **100+ large language model providers** — OpenAI, Anthropic, Google Gemini, AWS Bedrock, Azure, and many more. You use it in one of two ways:

- **Python SDK** — `pip install litellm`, then call `completion(...)` from your code. One function, every provider.
- **AI Gateway (Proxy Server)** — run `litellm` as a standalone OpenAI-compatible HTTP service for your whole team, with virtual API keys, spend tracking, load balancing, guardrails, and an admin dashboard.

Both speak the **OpenAI request/response format**, so you can swap providers — or move from the SDK to the gateway — without rewriting your application code.

> [!NOTE]
> This guide covers everyday usage of both the SDK and the gateway. For architecture, deployment topology, the full configuration reference, authentication internals, and operations, see the **LiteLLM Admin Guide**.

---

## 1. Quick Start

### 1.1 Python SDK in 30 seconds

```bash
pip install litellm
```

```python
from litellm import completion
import os

os.environ["OPENAI_API_KEY"] = "sk-your-openai-key"

response = completion(
    model="openai/gpt-4o",
    messages=[{"role": "user", "content": "Hello, world!"}],
)

print(response.choices[0].message.content)
```

The return value is an OpenAI-style `ModelResponse` object: `response.choices[0].message.content` holds the text, and `response.usage` holds token counts. Switching to a different provider is a one-line change to the `model` string:

```python
os.environ["ANTHROPIC_API_KEY"] = "sk-ant-your-key"

response = completion(
    model="anthropic/claude-sonnet-4-20250514",
    messages=[{"role": "user", "content": "Hello, world!"}],
)
```

> [!TIP]
> The `model` string follows the pattern `provider/model-name`. The prefix before the slash (`openai/`, `anthropic/`, `gemini/`, `bedrock/`, …) tells LiteLLM which provider to route to and which API key to read. See [§5 Supported Providers](#5-supported-providers).

### 1.2 AI Gateway in 60 seconds

```bash
pip install 'litellm[proxy]'
litellm --model gpt-4o
```

This starts an OpenAI-compatible server on `http://0.0.0.0:4000`. Point any OpenAI client at it:

```python
import openai

client = openai.OpenAI(api_key="anything", base_url="http://0.0.0.0:4000")

response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Hello!"}],
)
print(response.choices[0].message.content)
```

For real deployments you start the gateway from a `config.yaml` instead of `--model`:

```bash
litellm --config config.yaml
```

A minimal `config.yaml`:

```yaml
model_list:
  - model_name: gpt-4o
    litellm_params:
      model: openai/gpt-4o
      api_key: os.environ/OPENAI_API_KEY

general_settings:
  master_key: sk-1234
```

> [!IMPORTANT]
> `model_name` is the **public alias** clients request. `litellm_params.model` is the **real upstream model** LiteLLM calls. This indirection is what lets you load-balance, rename, and swap models without touching client code. The full config reference lives in the **Admin Guide**.

---

## 2. Installation

LiteLLM is published on PyPI as [`litellm`](https://pypi.org/project/litellm/). The current version documented here is **v1.88.0**.

=== "pip (SDK only)"

    ```bash
    pip install litellm
    ```

=== "pip (with proxy server)"

    ```bash
    pip install 'litellm[proxy]'
    ```

=== "uv"

    ```bash
    uv add litellm            # SDK in a project
    uv tool install 'litellm[proxy]'   # proxy as a CLI tool
    ```

=== "Docker (gateway)"

    ```bash
    docker run -p 4000:4000 \
      -e OPENAI_API_KEY=$OPENAI_API_KEY \
      -v $(pwd)/config.yaml:/app/config.yaml \
      ghcr.io/berriai/litellm:main-latest \
      --config /app/config.yaml
    ```

| Install target | When to use |
|---|---|
| `litellm` | You only need the Python SDK inside your own application. |
| `litellm[proxy]` | You want to run the gateway / proxy server (`litellm --config …`). |
| Docker image | You want to deploy the gateway as a container. See the Admin Guide. |

> [!NOTE]
> Requirements: **Python 3.8+** for the SDK. The proxy server adds FastAPI/uvicorn and (optionally) a PostgreSQL database when you enable virtual keys, teams, and spend tracking. Database setup is covered in the Admin Guide.

---

## 3. Core Concepts

**LiteLLM (`litellm`)**
: The Python library. Import functions like `completion`, `embedding`, and `acompletion` directly. No server required.

**AI Gateway / Proxy Server**
: A long-running HTTP service (`litellm --config config.yaml`) that exposes OpenAI-compatible endpoints, manages virtual keys and budgets, and routes to many upstream models. Also called the **LiteLLM Proxy**.

**The model string** (`provider/model`)
: Every call identifies a model with a `provider/model-name` string. The provider prefix selects the adapter and the environment variable that holds its credentials. Examples: `openai/gpt-4o`, `anthropic/claude-sonnet-4-20250514`, `gemini/gemini-2.5-pro`, `bedrock/anthropic.claude-3-5-sonnet-20240620-v1:0`, `azure/<your-deployment>`.

**OpenAI format**
: Requests use OpenAI's `messages` array and parameters (`temperature`, `max_tokens`, `tools`, `stream`, …); responses use OpenAI's `choices`/`usage` shape — regardless of the underlying provider. LiteLLM translates to and from each provider's native API for you.

**Router**
: The SDK component (and the engine inside the gateway) that load-balances across multiple deployments of the same model, applies retries, fallbacks, and rate limits. See [§6](#6-reliability-routing-retries-and-fallbacks).

---

## 4. Using the Python SDK

All SDK functions accept the OpenAI parameter set and return OpenAI-shaped objects. Every function has an `a`-prefixed **async** twin (`acompletion`, `aembedding`, …) usable with `await`.

### 4.1 Chat completions

```python
from litellm import completion

response = completion(
    model="openai/gpt-4o",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Explain async I/O in one sentence."},
    ],
    temperature=0.7,
    max_tokens=200,
)
print(response.choices[0].message.content)
print(response.usage)   # prompt_tokens, completion_tokens, total_tokens
```

```text
ModelResponse(... choices=[Choices(message=Message(content="Async I/O lets a single thread ..."))],
usage=Usage(prompt_tokens=24, completion_tokens=18, total_tokens=42))
```

### 4.2 Streaming

Set `stream=True` to receive incremental OpenAI-style chunks:

```python
from litellm import completion

stream = completion(
    model="anthropic/claude-sonnet-4-20250514",
    messages=[{"role": "user", "content": "Write a haiku about the sea."}],
    stream=True,
)

for chunk in stream:
    print(chunk.choices[0].delta.content or "", end="", flush=True)
```

To reassemble a streamed response into a single object, use `stream_chunk_builder`:

```python
from litellm import completion, stream_chunk_builder

chunks = list(completion(model="openai/gpt-4o", messages=messages, stream=True))
final = stream_chunk_builder(chunks, messages=messages)
print(final.choices[0].message.content)
```

### 4.3 Async usage

```python
import asyncio
from litellm import acompletion

async def main():
    response = await acompletion(
        model="gemini/gemini-2.5-pro",
        messages=[{"role": "user", "content": "Hello!"}],
    )
    print(response.choices[0].message.content)

asyncio.run(main())
```

### 4.4 Embeddings

```python
from litellm import embedding

response = embedding(
    model="openai/text-embedding-3-small",
    input=["The quick brown fox", "jumps over the lazy dog"],
)
print(len(response.data))            # 2 vectors
print(len(response.data[0]["embedding"]))  # vector dimensions
```

Async: `from litellm import aembedding`.

### 4.5 Image generation

```python
from litellm import image_generation

response = image_generation(
    model="openai/dall-e-3",
    prompt="A red panda coding at a laptop, watercolor",
    n=1,
    size="1024x1024",
)
print(response.data[0]["url"])
```

### 4.6 Audio — transcription and speech

```python
from litellm import transcription, speech

# Speech-to-text (Whisper-compatible)
with open("audio.mp3", "rb") as f:
    result = transcription(model="openai/whisper-1", file=f)
print(result.text)

# Text-to-speech
tts = speech(model="openai/tts-1", input="Hello from LiteLLM", voice="alloy")
tts.stream_to_file("speech.mp3")
```

Async twins: `atranscription`, `aspeech`.

### 4.7 Moderation

```python
from litellm import moderation

result = moderation(model="openai/omni-moderation-latest", input="some text to check")
print(result.results[0].flagged)
```

### 4.8 Reranking

```python
from litellm import rerank

result = rerank(
    model="cohere/rerank-english-v3.0",
    query="What is the capital of the US?",
    documents=["Paris is in France.", "Washington, D.C. is the US capital."],
    top_n=2,
)
print(result.results)
```

Async: `arerank`.

### 4.9 The Responses API

LiteLLM mirrors OpenAI's newer **Responses API** (`/responses`) across providers:

```python
from litellm import responses

resp = responses(
    model="openai/gpt-4o",
    input="Summarize the theory of relativity in two sentences.",
)
print(resp.output_text)
```

### 4.10 Token counting

```python
from litellm import token_counter

n = token_counter(model="openai/gpt-4o", messages=[{"role": "user", "content": "hi"}])
print(n)
```

### 4.11 Setting provider credentials

LiteLLM reads provider keys from **environment variables** by default. The variable name is provider-specific (e.g. `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, `GEMINI_API_KEY`, `COHERE_API_KEY`; AWS uses the standard `AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY` / `AWS_REGION_NAME`). You can also pass credentials inline per call:

```python
response = completion(
    model="openai/gpt-4o",
    messages=messages,
    api_key="sk-...",          # overrides the env var
    api_base="https://my-proxy.example.com/v1",  # custom endpoint
)
```

> [!TIP]
> Provider-specific parameters that aren't part of the OpenAI schema can be passed straight through (e.g. `top_k`, `reasoning_effort`). If a provider doesn't support a parameter you sent, set `drop_params=True` to have LiteLLM silently drop unsupported params instead of erroring.

### 4.12 Handling exceptions

LiteLLM maps every provider's errors onto the **OpenAI exception types**, so you write one set of handlers:

```python
from litellm import completion
from litellm.exceptions import (
    AuthenticationError, RateLimitError, ContextWindowExceededError,
    BadRequestError, Timeout, APIConnectionError, ServiceUnavailableError,
)

try:
    response = completion(model="openai/gpt-4o", messages=messages)
except RateLimitError:
    ...   # back off and retry
except ContextWindowExceededError:
    ...   # trim the prompt
except AuthenticationError:
    ...   # bad/missing API key
```

| Exception | Meaning |
|---|---|
| `AuthenticationError` | Missing or invalid API key for the provider. |
| `RateLimitError` | Upstream provider rate limit (HTTP 429). |
| `ContextWindowExceededError` | Prompt + max_tokens exceed the model's context window. |
| `BadRequestError` | Malformed request or unsupported parameter. |
| `Timeout` | The request exceeded the configured timeout. |
| `APIConnectionError` | Network failure reaching the provider. |
| `ServiceUnavailableError` | Provider returned 5xx / is down. |

---

## 5. Supported Providers

LiteLLM ships adapters for **100+ providers** (123 provider integrations in this version). You select one with the model-string prefix; the full, always-current matrix of which providers support which endpoints lives at **[models.litellm.ai](https://models.litellm.ai/)** and in the [provider docs](https://docs.litellm.ai/docs/providers).

A selection of commonly used providers and their model-string prefixes:

| Provider | Prefix | Common credentials |
|---|---|---|
| OpenAI | `openai/` | `OPENAI_API_KEY` |
| Anthropic | `anthropic/` | `ANTHROPIC_API_KEY` |
| Google AI Studio (Gemini) | `gemini/` | `GEMINI_API_KEY` |
| Google Vertex AI | `vertex_ai/` | GCP service-account / ADC |
| Azure OpenAI | `azure/` | `AZURE_API_KEY`, `AZURE_API_BASE`, `AZURE_API_VERSION` |
| AWS Bedrock | `bedrock/` | `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_REGION_NAME` |
| AWS SageMaker | `sagemaker/` | AWS credentials |
| Cohere | `cohere/` | `COHERE_API_KEY` |
| Mistral | `mistral/` | `MISTRAL_API_KEY` |
| Groq | `groq/` | `GROQ_API_KEY` |
| DeepSeek | `deepseek/` | `DEEPSEEK_API_KEY` |
| Together AI | `together_ai/` | `TOGETHERAI_API_KEY` |
| Fireworks AI | `fireworks_ai/` | `FIREWORKS_AI_API_KEY` |
| OpenRouter | `openrouter/` | `OPENROUTER_API_KEY` |
| Databricks | `databricks/` | Databricks token |
| IBM watsonx | `watsonx/` | watsonx credentials |
| Hugging Face | `huggingface/` | `HUGGINGFACE_API_KEY` |
| Ollama (local) | `ollama/` | — (local server) |
| vLLM (self-hosted) | `hosted_vllm/` | — (your endpoint) |
| xAI (Grok) | `xai/` | `XAI_API_KEY` |
| Perplexity | `perplexity/` | `PERPLEXITYAI_API_KEY` |
| NVIDIA NIM | `nvidia_nim/` | NVIDIA credentials |
| GitHub Models / Copilot | `github/`, `github_copilot/` | GitHub token |

> [!NOTE]
> Not every provider supports every capability. Bedrock and Cohere support `/rerank`; Azure and Vertex support image and audio endpoints; many OpenAI-compatible providers support only `/chat/completions`. Check the capability matrix on [models.litellm.ai](https://models.litellm.ai/) before relying on a less common endpoint for a given provider.

### Calling any OpenAI-compatible endpoint

If a provider speaks the OpenAI API but isn't a built-in integration, use the generic `openai/` prefix with a custom `api_base`:

```python
response = completion(
    model="openai/my-model",
    messages=messages,
    api_base="https://my-openai-compatible-host/v1",
    api_key="sk-...",
)
```

---

## 6. Reliability: Routing, Retries, and Fallbacks

For production workloads you usually want more than a single model call: load balancing across several API keys or regions, automatic retries, and fallbacks to a backup model. The **`Router`** provides this in the SDK (the gateway uses the same engine — configure it in `config.yaml`, covered in the Admin Guide).

### 6.1 Load balancing across deployments

```python
from litellm import Router

router = Router(
    model_list=[
        {
            "model_name": "gpt-4o",                       # public alias
            "litellm_params": {"model": "openai/gpt-4o", "api_key": "sk-key-1"},
            "tpm": 100000, "rpm": 1000,
        },
        {
            "model_name": "gpt-4o",                       # same alias, 2nd deployment
            "litellm_params": {"model": "azure/gpt-4o-deployment",
                                "api_key": "azure-key", "api_base": "https://...azure.com"},
        },
    ],
    routing_strategy="usage-based-routing-v2",
)

response = router.completion(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Hello!"}],
)
```

The router spreads requests across both deployments of `gpt-4o` according to the chosen strategy and the per-deployment `tpm`/`rpm` limits.

### 6.2 Retries and fallbacks

```python
router = Router(
    model_list=model_list,
    num_retries=2,                       # retry transient failures
    fallbacks=[{"gpt-4o": ["claude-sonnet"]}],   # if gpt-4o fails, try claude-sonnet
    context_window_fallbacks=[{"gpt-4o": ["gpt-4o-large-context"]}],
)
```

| Mechanism | What it does |
|---|---|
| `num_retries` | Retries the same deployment on transient errors (timeouts, 5xx, rate limits). |
| `fallbacks` | On exhausting retries for a model, fails over to a different model. |
| `context_window_fallbacks` | On a context-window error, routes to a larger-context model. |
| `tpm` / `rpm` | Per-deployment tokens-per-minute / requests-per-minute caps used for load balancing. |

> [!TIP]
> Without the `Router`, `completion_with_retries(...)` gives you simple retry behavior for a single model. Use the `Router` when you have multiple deployments or want fallbacks.

### 6.3 One-off retries without a router

```python
from litellm import completion_with_retries

response = completion_with_retries(
    model="openai/gpt-4o",
    messages=messages,
    num_retries=3,
)
```

---

## 7. Caching (SDK)

LiteLLM can cache responses so identical requests don't re-hit the provider:

```python
import litellm
from litellm import completion
from litellm.caching.caching import Cache

litellm.cache = Cache()   # in-memory by default; supports Redis, S3, disk, etc.

# First call hits the provider; the second identical call is served from cache.
completion(model="openai/gpt-4o", messages=messages, caching=True)
completion(model="openai/gpt-4o", messages=messages, caching=True)
```

> [!NOTE]
> For Redis-backed caching, pass `Cache(type="redis", host=..., port=..., password=...)`. The gateway exposes cache configuration in `config.yaml` and cache-management endpoints — see the Admin Guide.

---

## 8. Using the AI Gateway (as a client)

Once an admin has deployed the LiteLLM gateway (see the Admin Guide), you interact with it as an OpenAI-compatible service. You authenticate with a **virtual key** (a `sk-...` token your admin issues), not the raw provider keys.

### 8.1 Making requests

=== "OpenAI Python SDK"

    ```python
    import openai

    client = openai.OpenAI(
        api_key="sk-your-virtual-key",
        base_url="http://your-gateway:4000",
    )
    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": "Hello!"}],
    )
    ```

=== "curl"

    ```bash
    curl http://your-gateway:4000/v1/chat/completions \
      -H "Authorization: Bearer sk-your-virtual-key" \
      -H "Content-Type: application/json" \
      -d '{
        "model": "gpt-4o",
        "messages": [{"role": "user", "content": "Hello!"}]
      }'
    ```

=== "LiteLLM SDK → gateway"

    ```python
    from litellm import completion

    response = completion(
        model="litellm_proxy/gpt-4o",
        messages=[{"role": "user", "content": "Hello!"}],
        api_key="sk-your-virtual-key",
        api_base="http://your-gateway:4000",
    )
    ```

The gateway exposes the standard OpenAI surface — `/chat/completions`, `/completions`, `/embeddings`, `/images/generations`, `/audio/transcriptions`, `/audio/speech`, `/moderations`, `/rerank`, `/responses`, `/messages` (Anthropic format), and `/models` — plus `/v1/...` equivalents.

> [!IMPORTANT]
> You request the **public model name** your admin configured (e.g. `gpt-4o`), not the upstream model string. Run `GET /models` (or `litellm-proxy models list`) to see what's available to your key.

### 8.2 The `litellm-proxy` CLI

Installing `litellm[proxy]` also gives you the **`litellm-proxy`** command-line client for talking to a running gateway. Point it at your gateway and key:

```bash
export LITELLM_PROXY_URL=http://your-gateway:4000
export LITELLM_PROXY_API_KEY=sk-your-virtual-key
```

| Command group | What it does |
|---|---|
| `litellm-proxy chat <model>` | Start an interactive chat against a model (`-t/--temperature`, `--max-tokens`, `-s/--system`). |
| `litellm-proxy models list` | List models available on the gateway. |
| `litellm-proxy keys list` / `generate` | List or create virtual keys (admin). |
| `litellm-proxy users …` | List/create/delete users (admin). |
| `litellm-proxy teams …` | Manage teams (admin). |
| `litellm-proxy credentials …` | Manage stored provider credentials (admin). |
| `litellm-proxy http request <METHOD> <URI>` | Make a raw authenticated request to any gateway route. |
| `litellm-proxy login` / `logout` / `whoami` | Authenticate the CLI session (SSO) and check identity. |

```bash
# Quick interactive chat
litellm-proxy chat gpt-4o --system "You are concise." --temperature 0.3

# See which models you can call
litellm-proxy models list
```

> [!NOTE]
> Key, user, team, and credential management commands require an admin (master-key or admin-role) token. As a regular user you'll mostly use `chat`, `models list`, and `http request`.

---

## 9. Advanced Features

### 9.1 Tool / function calling

Tool calling works in the OpenAI format across every provider that supports it:

```python
tools = [{
    "type": "function",
    "function": {
        "name": "get_weather",
        "description": "Get the weather for a city",
        "parameters": {"type": "object", "properties": {"city": {"type": "string"}}},
    },
}]

response = completion(model="openai/gpt-4o", messages=messages, tools=tools)
tool_calls = response.choices[0].message.tool_calls
```

### 9.2 MCP tools (Model Context Protocol)

LiteLLM can load tools from an **MCP server** and expose them to any model. Via the SDK bridge:

```python
from litellm import experimental_mcp_client
import litellm

tools = await experimental_mcp_client.load_mcp_tools(session=session, format="openai")
response = await litellm.acompletion(model="gpt-4o", messages=messages, tools=tools)
```

Via the gateway, reference a registered MCP server directly in the request:

```bash
curl -X POST 'http://your-gateway:4000/v1/chat/completions' \
  -H 'Authorization: Bearer sk-your-virtual-key' \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "gpt-4o",
    "messages": [{"role": "user", "content": "Summarize the latest open PR"}],
    "tools": [{
      "type": "mcp",
      "server_url": "litellm_proxy/mcp/github",
      "server_label": "github_mcp",
      "require_approval": "never"
    }]
  }'
```

The gateway can also be used as an MCP endpoint directly from tools like Cursor by pointing them at `http://your-gateway:4000/mcp/` with an `x-litellm-api-key` header.

### 9.3 A2A agents

LiteLLM exposes an **Agent-to-Agent (A2A)** gateway for invoking agents (LangGraph, Vertex AI Agent Engine, Azure AI Foundry, Bedrock AgentCore, Pydantic AI) behind a virtual key:

```python
base_url = "http://your-gateway:4000/a2a/my-agent"   # gateway + agent name
headers = {"Authorization": "Bearer sk-your-virtual-key"}
# ...use the a2a-sdk A2AClient to send messages (see README for the full example)
```

### 9.4 Batch completions

Send the same prompt to multiple models, or run many prompts, in one call:

```python
from litellm import batch_completion

responses = batch_completion(
    model="openai/gpt-4o",
    messages=[[{"role": "user", "content": q}] for q in questions],
)
```

---

## 10. Common Tasks (Recipes)

**Swap a provider without code changes**
: Change only the `model` string: `openai/gpt-4o` → `anthropic/claude-sonnet-4-20250514`. Set the matching API key env var. Everything else stays the same.

**Drop unsupported parameters automatically**
: Pass `drop_params=True` (or set `litellm.drop_params = True` globally) so provider-incompatible params are dropped instead of raising `BadRequestError`.

**Get consistent token usage across providers**
: Read `response.usage.total_tokens` — LiteLLM normalizes usage reporting even when providers report it differently.

**Use a local model**
: Run Ollama locally and call `completion(model="ollama/llama3", messages=...)`. No API key needed.

**Point the SDK at your team's gateway**
: Prefix the model with `litellm_proxy/` and pass `api_base`/`api_key` for the gateway, so the SDK routes through your gateway's keys, budgets, and logging.

**Verbose debugging**
: Set `litellm.set_verbose = True` (SDK) or start the gateway with `--detailed_debug` to see the exact upstream request/response.

---

## 11. Troubleshooting

### Error: `AuthenticationError - Missing API Key`

**Cause:** The environment variable for the provider isn't set, or the wrong variable name is used.
**Solution:** Set the provider's key (e.g. `export OPENAI_API_KEY=sk-...`). Confirm the variable name matches the provider (`ANTHROPIC_API_KEY`, `GEMINI_API_KEY`, etc.).
**Prevention:** Load keys from a `.env` file at startup, or pass `api_key=...` explicitly per call.

### Error: `BadRequestError - <param> is not supported`

**Cause:** You passed a parameter the target provider doesn't accept.
**Solution:** Set `drop_params=True` so LiteLLM drops unsupported params, or remove the parameter.
**Prevention:** Set `litellm.drop_params = True` globally when calling many heterogeneous providers.

### Error: `ContextWindowExceededError`

**Cause:** Prompt tokens plus `max_tokens` exceed the model's context window.
**Solution:** Trim the prompt, lower `max_tokens`, or configure `context_window_fallbacks` to a larger-context model.
**Prevention:** Use `token_counter(...)` to check size before sending.

### Error: `RateLimitError` (HTTP 429)

**Cause:** The upstream provider throttled you.
**Solution:** Use a `Router` with `num_retries` and multiple deployments to spread load; add `fallbacks` to a secondary provider.
**Prevention:** Set per-deployment `tpm`/`rpm` so the router paces requests.

### Error: `Timeout`

**Cause:** The provider didn't respond within the timeout.
**Solution:** Increase `timeout=` on the call (or `--request_timeout` on the gateway). Add retries.
**Prevention:** Set a sensible default timeout and retry policy for production traffic.

### Gateway: `Invalid proxy server token passed`

**Cause:** The `Authorization: Bearer` token isn't a valid virtual key (or you used the master key where a virtual key is expected).
**Solution:** Generate a virtual key (`litellm-proxy keys generate` or the dashboard) and use it. Confirm `base_url` points at the gateway, not a provider.
**Prevention:** Never ship the master key to clients — issue scoped virtual keys.

### Streaming returns nothing / buffers

**Cause:** A client or intermediary is buffering the response, or the provider's streaming isn't enabled.
**Solution:** Confirm `stream=True` and that your HTTP client reads incrementally. When going through the gateway, ensure no proxy in front buffers SSE responses.

> [!WARNING]
> Some providers silently degrade unsupported features rather than erroring — e.g. strict JSON-schema response formats may fall back to looser JSON on certain providers, and `tool_choice` can be dropped on cached requests for some providers. If strict behavior matters, verify the provider supports it on [models.litellm.ai](https://models.litellm.ai/) and test end-to-end.

---

## 12. FAQ

**Q: Do I have to use the OpenAI format?**
Yes — that's the point. You always send OpenAI-format requests; LiteLLM translates to each provider. It also accepts and exposes the Anthropic `/messages` format through a compatibility layer if you prefer it.

**Q: What's the difference between the SDK and the gateway?**
The SDK is a library you import into one application. The gateway is a shared HTTP service for many applications/users, adding virtual keys, budgets, spend tracking, load balancing, guardrails, and a dashboard. They share the same routing engine.

**Q: Can I use LiteLLM with the official OpenAI client?**
Yes. Point `openai.OpenAI(base_url="http://your-gateway:4000", api_key="sk-virtual-key")` at the gateway. No code changes beyond the base URL and key.

**Q: How do I add a provider that isn't built in?**
If it's OpenAI-compatible, use the `openai/` prefix with a custom `api_base`. Otherwise LiteLLM supports custom provider handlers (see the Admin Guide / provider docs).

**Q: Does LiteLLM support local models?**
Yes — Ollama (`ollama/...`), self-hosted vLLM (`hosted_vllm/...`), LM Studio, llamafile, and others. No API key needed for local servers.

**Q: How do I control which models a key can use?**
On the gateway, virtual keys can be scoped to specific models, teams, and budgets. Admins manage this via the dashboard, the management API, or `litellm-proxy keys generate --models ...`. See the Admin Guide.

**Q: How do I see how many tokens / how much I'm spending?**
Through the gateway: spend is tracked per key/user/team and viewable in the dashboard and via `/spend/*` endpoints. In the SDK, read `response.usage` and use LiteLLM's cost helpers.

**Q: Is streaming supported everywhere?**
Streaming is supported for chat completions across providers that offer it. Use `stream=True` and iterate the response; use `stream_chunk_builder` to reassemble.

**Q: Which Python version do I need?**
Python 3.8 or newer for the SDK.

**Q: Where's the full, always-current provider/model list?**
[models.litellm.ai](https://models.litellm.ai/) and [docs.litellm.ai/docs/providers](https://docs.litellm.ai/docs/providers).

---

## 13. Quick Reference

| Task | Call |
|---|---|
| Chat completion | `completion(model="provider/model", messages=[...])` |
| Async chat | `await acompletion(...)` |
| Streaming | `completion(..., stream=True)` |
| Embeddings | `embedding(model=..., input=[...])` |
| Image generation | `image_generation(model=..., prompt=...)` |
| Transcribe audio | `transcription(model=..., file=f)` |
| Text-to-speech | `speech(model=..., input=..., voice=...)` |
| Moderation | `moderation(model=..., input=...)` |
| Rerank | `rerank(model=..., query=..., documents=[...])` |
| Responses API | `responses(model=..., input=...)` |
| Count tokens | `token_counter(model=..., messages=[...])` |
| Load balance / fallbacks | `Router(model_list=[...], fallbacks=[...])` |
| Retries (single model) | `completion_with_retries(..., num_retries=3)` |
| Start the gateway | `litellm --config config.yaml` |
| Call the gateway | `openai.OpenAI(base_url="http://host:4000", api_key="sk-...")` |
| Gateway CLI chat | `litellm-proxy chat <model>` |

---

_Generated June 2026 from BerriAI/litellm at 9196098. AI-generated documentation. The library is checked by sampling (three repositories so far); this guide was not individually reviewed. Verify against the project before relying on it. Unofficial; not affiliated with or reviewed by the project._