# vLLM — User Guide

vLLM is a high-throughput, memory-efficient inference and serving engine for large language models. Originally developed at UC Berkeley's Sky Computing Lab, vLLM enables production-grade LLM serving with PagedAttention, continuous batching, and a fully OpenAI-compatible API — all on your own hardware.

This guide covers everything you need to deploy, configure, and use vLLM: from running your first model in minutes to advanced multi-GPU parallelism, quantization, LoRA adapters, multimodal inference, and beyond. For architecture internals, deployment operations, and configuration deep-dives, see the **vLLM Admin Guide**.

---

## Table of Contents

1. Quick Start
2. Installation
3. Configuration
4. Core Usage
5. Common Tasks & Workflow Recipes
6. Troubleshooting
7. FAQ

---

## 1. Quick Start

Get vLLM serving a model in under five minutes.

### 1.1 Prerequisites

- Python 3.9 or later (3.12 recommended)
- NVIDIA GPU with CUDA 12.1+ **or** AMD GPU with ROCm 6.x **or** a CPU-only environment
- At least 16 GiB of VRAM for a 7B parameter model (more for larger models)
- Internet access to download models from HuggingFace Hub

> [!TIP]
> For your very first test, the `facebook/opt-125m` model runs on any GPU with 2+ GiB VRAM and downloads in seconds. Use it to verify your installation before pulling a multi-gigabyte model.

### 1.2 Install vLLM

```bash
pip install vllm
```

### 1.3 Start a Server

```bash
vllm serve meta-llama/Llama-3.1-8B-Instruct
```

This starts an OpenAI-compatible HTTP server on `http://localhost:8000`. The first run downloads model weights from HuggingFace Hub.

### 1.4 Send Your First Request

In a second terminal, send a chat completion:

```bash
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Llama-3.1-8B-Instruct",
    "messages": [
      {"role": "user", "content": "What is PagedAttention?"}
    ]
  }'
```

Or use the built-in interactive chat CLI:

```bash
vllm chat
```

> [!TIP]
> The `vllm chat` and `vllm complete` commands connect to a running server at `localhost:8000` by default. Start the server first, then open these in a second terminal for a quick interactive test without writing any code.

### 1.5 Quick Offline Inference (Python)

No server needed for batch offline workloads:

```python
from vllm import LLM, SamplingParams

llm = LLM(model="meta-llama/Llama-3.1-8B-Instruct")
outputs = llm.generate(
    ["Explain quantum computing in one paragraph."],
    SamplingParams(temperature=0.8, max_tokens=256)
)
for output in outputs:
    print(output.outputs[0].text)
```

### 1.6 CLI at a Glance

| Command | Purpose |
|---|---|
| `vllm serve <model>` | Start OpenAI-compatible API server |
| `vllm chat` | Interactive chat with a running server |
| `vllm complete` | Interactive completion with a running server |
| `vllm bench serve` | Benchmark serving throughput |
| `vllm bench latency` | Benchmark single-request latency |
| `vllm bench throughput` | Benchmark offline throughput |
| `vllm bench mm-processor` | Benchmark multimodal preprocessing |
| `vllm bench startup` | Benchmark server startup time |
| `vllm bench sweep` | Parameter sweep benchmarking |
| `vllm collect-env` | Collect environment info for bug reports |
| `vllm launch render` | Start GPU-less rendering server |
| `vllm -v` / `vllm --version` | Show version |

---

## 2. Installation

### 2.1 Platform Support Matrix

| Platform | Method | Notes |
|---|---|---|
| NVIDIA CUDA (12.1+) | `pip install vllm` | Primary, fully supported |
| NVIDIA CUDA (Docker) | `docker pull vllm/vllm-openai` | Recommended for production |
| AMD ROCm | `pip install vllm` (ROCm build) | ROCm 6.x required |
| Intel XPU | `pip install vllm` (XPU build) | Intel Extension for PyTorch |
| CPU (Intel/AMD x86) | `pip install vllm --extra-index-url ...` | No GPU required |
| CPU (ARM AArch64) | `pip install vllm` | ARM-native build |
| CPU (Apple Silicon) | `pip install vllm` | MPS backend |
| CPU (IBM Z) | `pip install vllm` | s390x support |
| Google TPU | Hardware plugin | See TPU plugin docs |
| Intel Gaudi | Hardware plugin | See Gaudi plugin docs |
| IBM Spyre | Hardware plugin | See Spyre plugin docs |
| Huawei Ascend | Hardware plugin | See Ascend plugin docs |

> [!IMPORTANT]
> The default `pip install vllm` installs the CUDA build. For ROCm, XPU, or CPU targets, you must use the appropriate index URL or build from source. Installing the CUDA wheel on a non-CUDA system will fail at runtime.

### 2.2 NVIDIA CUDA — pip (Recommended for Development)

```bash
# Create a virtual environment (strongly recommended)
python -m venv vllm-env
source vllm-env/bin/activate   # Linux/macOS
# vllm-env\Scripts\activate    # Windows

# Install
pip install vllm
```

Verify the installation:

```bash
python -c "import vllm; print(vllm.__version__)"
vllm --version
```

> [!WARNING]
> vLLM requires PyTorch 2.4+ and is sensitive to CUDA/PyTorch version mismatches. If you already have PyTorch installed, ensure it matches vLLM's requirements. Installing into a fresh virtual environment avoids version conflicts.

### 2.3 NVIDIA CUDA — Docker (Recommended for Production)

The official Docker image bundles CUDA, FlashInfer, and all dependencies:

```bash
# Pull the latest image
docker pull vllm/vllm-openai:latest

# Run with GPU access
docker run --runtime nvidia --gpus all \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -p 8000:8000 \
  vllm/vllm-openai:latest \
  --model meta-llama/Llama-3.1-8B-Instruct
```

Default image base: `nvidia/cuda:13.0.0-devel-ubuntu22.04`, Python 3.12, FlashInfer 0.6.7.

Pass a HuggingFace token for gated models:

```bash
docker run --runtime nvidia --gpus all \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -e HUGGING_FACE_HUB_TOKEN=hf_your_token_here \
  -p 8000:8000 \
  vllm/vllm-openai:latest \
  --model meta-llama/Meta-Llama-3-70B-Instruct \
  --tensor-parallel-size 4
```

> [!IMPORTANT]
> Mount the HuggingFace cache directory (`~/.cache/huggingface`) into the container so model weights persist between container restarts. Without this mount, the container re-downloads weights on every run.

### 2.4 AMD ROCm

```bash
# Install the ROCm-specific build
pip install vllm --extra-index-url https://download.pytorch.org/whl/rocm6.2

# Set the target device
export VLLM_TARGET_DEVICE=rocm

# Verify GPU visibility
export HIP_VISIBLE_DEVICES=0,1  # adjust as needed
```

### 2.5 Intel XPU

```bash
export VLLM_TARGET_DEVICE=xpu
pip install vllm  # XPU build from appropriate index
```

### 2.6 CPU-Only Installation

CPU mode works on Intel/AMD x86, ARM AArch64, Apple Silicon, and IBM Z. It is slower than GPU but requires no specialized hardware:

```bash
export VLLM_TARGET_DEVICE=cpu
pip install vllm
```

For CPU inference, also configure KV cache space:

```bash
export VLLM_CPU_KVCACHE_SPACE=8   # GiB of RAM for KV cache
```

### 2.7 Build Parallelism (Speeding Up Source Builds)

If building from source, control compilation parallelism:

```bash
export MAX_JOBS=8       # parallel compile jobs
export NVCC_THREADS=4  # NVCC threads per job
pip install -e .
```

### 2.8 Verifying Installation

```bash
# Check version
vllm --version

# Collect full environment info (share when filing bug reports)
vllm collect-env

# Quick smoke test (tiny model, minimal VRAM)
vllm serve facebook/opt-125m --port 8001 &
curl http://localhost:8001/health
curl http://localhost:8001/v1/models
kill %1
```

---

## 3. Configuration

vLLM can be configured via CLI flags, a YAML config file, and environment variables. All three can be combined; CLI flags take highest precedence, then YAML, then environment variables.

### 3.1 YAML Config File

Create a `vllm_config.yaml` to avoid long command lines:

```yaml
model: meta-llama/Llama-3.1-8B-Instruct
host: 0.0.0.0
port: 8000
tensor-parallel-size: 2
gpu-memory-utilization: 0.90
max-model-len: 8192
enable-prefix-caching: true
dtype: bfloat16
```

Load it with:

```bash
vllm serve --config vllm_config.yaml
```

CLI flags override YAML values when both are specified.

### 3.2 Complete `vllm serve` Flag Reference

#### Model & Tokenizer

| Flag | Default | Description |
|---|---|---|
| `<model>` (positional) | — | Model name or local path (required) |
| `--model` | — | Model name or path (alternative to positional) |
| `--served-model-name` | — | Name returned by `/v1/models` (alias for API use) |
| `--tokenizer-mode` | `auto` | `auto`, `hf`, `slow`, `mistral`, `deepseek_v32`, `qwen_vl` |
| `--trust-remote-code` | false | Allow remote code execution from HuggingFace Hub |
| `--max-model-len` | (from config) | Maximum sequence length (input + output tokens) |
| `--dtype` | `auto` | `auto`, `half`, `float16`, `bfloat16`, `float32` |
| `--kv-cache-dtype` | `auto` | Data type for KV cache |

#### Networking

| Flag | Default | Description |
|---|---|---|
| `--host` | `localhost` | Server bind address |
| `--port` | `8000` | Server port |
| `--ssl-keyfile` | — | Path to SSL private key |
| `--ssl-certfile` | — | Path to SSL certificate |

#### Memory & Performance

| Flag | Default | Description |
|---|---|---|
| `--gpu-memory-utilization` | `0.9` | Fraction of GPU memory to use for model + KV cache |
| `--max-num-seqs` | (auto) | Maximum concurrent sequences in a batch |
| `--max-num-batched-tokens` | (auto) | Maximum tokens processed per scheduler step |
| `--block-size` | (auto) | Block size for paged attention KV cache |
| `--enable-prefix-caching` | false | Cache KV for repeated prompt prefixes |
| `--enable-chunked-prefill` | false | Process long prefills in chunks for better batching |
| `--enforce-eager` | false | Disable CUDA graphs (use eager PyTorch execution) |

#### Parallelism

| Flag | Short | Description |
|---|---|---|
| `--tensor-parallel-size` | `-tp` | Number of GPUs for tensor parallelism (intra-layer) |
| `--pipeline-parallel-size` | `-pp` | Number of GPUs for pipeline parallelism (inter-layer) |
| `--data-parallel-size` | `-dp` | Number of data-parallel replicas |
| `--enable-expert-parallel` | `-ep` | Enable expert parallelism for MoE models |

#### Quantization

| Flag | Short | Description |
|---|---|---|
| `--quantization` | `-q` | Quantization method (see §4.5 for full list) |

#### LoRA

| Flag | Description |
|---|---|
| `--enable-lora` | Enable LoRA adapter loading |
| `--max-loras` | Maximum number of LoRA adapters loaded simultaneously |

#### Advanced Features

| Flag | Short | Description |
|---|---|---|
| `--speculative-config` | `-sc` | JSON/YAML speculative decoding config |
| `--compilation-config` | `-cc` | torch.compile configuration |
| `--structured-outputs-config` | — | Structured output (xgrammar/guidance) config |
| `--attention-config` | `-ac` | Attention backend selection config |
| `--reasoning-config` | — | Reasoning parser configuration |
| `--kv-transfer-config` | — | Disaggregated prefill/decode config |

#### Observability

| Flag | Description |
|---|---|
| `--disable-log-stats` | Disable per-request and aggregate statistics logging |
| `--config` | Path to YAML config file |

### 3.3 Environment Variable Reference

| Variable | Default | Description |
|---|---|---|
| `VLLM_TARGET_DEVICE` | `cuda` | Target device: `cuda`, `rocm`, `cpu`, `xpu`, `empty` |
| `VLLM_API_KEY` | — | API key for server authentication (if set, all requests must include it) |
| `VLLM_HOST_IP` | — | Explicit host IP override |
| `VLLM_PORT` | `0` | Default port override |
| `VLLM_CONFIGURE_LOGGING` | `1` | Enable vLLM logging (`0` to disable) |
| `VLLM_LOGGING_LEVEL` | `INFO` | Log level: `DEBUG`, `INFO`, `WARNING`, `ERROR` |
| `VLLM_LOGGING_CONFIG_PATH` | — | Path to Python logging config file |
| `VLLM_NO_USAGE_STATS` | — | Set any value to disable usage statistics reporting |
| `VLLM_DO_NOT_TRACK` | — | Alternative to `VLLM_NO_USAGE_STATS` |
| `VLLM_CPU_KVCACHE_SPACE` | `0` | CPU KV cache space in GiB (CPU inference only) |
| `VLLM_IMAGE_FETCH_TIMEOUT` | `5` | Timeout in seconds for fetching images |
| `VLLM_VIDEO_FETCH_TIMEOUT` | `30` | Timeout in seconds for fetching video |
| `VLLM_AUDIO_FETCH_TIMEOUT` | `10` | Timeout in seconds for fetching audio |
| `VLLM_MEDIA_CACHE_MAX_SIZE_MB` | `5120` | Maximum media (image/video/audio) cache size in MiB |
| `CUDA_VISIBLE_DEVICES` | — | Comma-separated CUDA device indices to expose |
| `HIP_VISIBLE_DEVICES` | — | Comma-separated ROCm device indices to expose |
| `MAX_JOBS` | — | Build parallelism for source compilation |
| `NVCC_THREADS` | — | NVCC threads per job for source compilation |

### 3.4 API Key Authentication

To require an API key for all server requests:

```bash
export VLLM_API_KEY=my-secret-key
vllm serve meta-llama/Llama-3.1-8B-Instruct
```

Clients must then pass the key:

```bash
curl http://localhost:8000/v1/chat/completions \
  -H "Authorization: Bearer my-secret-key" \
  -H "Content-Type: application/json" \
  -d '{"model": "meta-llama/Llama-3.1-8B-Instruct", "messages": [{"role": "user", "content": "Hello"}]}'
```

### 3.5 Disabling Usage Statistics

vLLM may collect anonymous usage statistics. To opt out:

```bash
export VLLM_NO_USAGE_STATS=1
# or equivalently:
export VLLM_DO_NOT_TRACK=1
```

---

## 4. Core Usage

### 4.1 OpenAI-Compatible API

vLLM's server is a drop-in replacement for the OpenAI API. Any client that works with OpenAI works with vLLM — point it at your vLLM host and port.

#### Chat Completions

```bash
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Llama-3.1-8B-Instruct",
    "messages": [
      {"role": "system", "content": "You are a helpful assistant."},
      {"role": "user", "content": "Summarize the French Revolution."}
    ],
    "temperature": 0.7,
    "max_tokens": 512
  }'
```

#### Text Completions

```bash
curl http://localhost:8000/v1/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Llama-3.1-8B-Instruct",
    "prompt": "The capital of France is",
    "max_tokens": 20,
    "temperature": 0
  }'
```

#### Streaming Responses

Add `"stream": true` to any request for Server-Sent Events:

```bash
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Llama-3.1-8B-Instruct",
    "messages": [{"role": "user", "content": "Write a haiku about the ocean."}],
    "stream": true
  }'
```

> [!TIP]
> Streaming dramatically improves perceived latency for long responses. Enable it in any production user-facing application — the first token arrives as soon as it is generated rather than waiting for the complete response.

#### List Available Models

```bash
curl http://localhost:8000/v1/models
```

#### Health Check

```bash
curl http://localhost:8000/health
```

### 4.2 Complete API Endpoint Reference

| Method | Endpoint | Description |
|---|---|---|
| `POST` | `/v1/chat/completions` | OpenAI-compatible chat completion |
| `POST` | `/v1/completions` | OpenAI-compatible text completion |
| `GET` | `/v1/models` | List loaded models |
| `GET` | `/health` | Server health check |
| `POST` | `/v1/embeddings` | Generate text embeddings |
| `POST` | `/v2/embed` | Cohere-compatible embeddings |
| `POST` | `/classify` | Text classification |
| `POST` | `/score` | Relevance scoring |
| `POST` | `/tokenize` | Tokenize text |
| `POST` | `/detokenize` | Detokenize token IDs |

vLLM also supports:
- **Anthropic Messages API** — use with Anthropic-format clients
- **Responses API** — stateful multi-turn via the Responses protocol

### 4.3 Python Client (OpenAI SDK)

```python
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="not-used"  # or your VLLM_API_KEY value
)

# Chat completion
response = client.chat.completions.create(
    model="meta-llama/Llama-3.1-8B-Instruct",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "What is PagedAttention?"}
    ],
    temperature=0.7,
    max_tokens=256
)
print(response.choices[0].message.content)

# Streaming
stream = client.chat.completions.create(
    model="meta-llama/Llama-3.1-8B-Instruct",
    messages=[{"role": "user", "content": "List 5 programming languages."}],
    stream=True
)
for chunk in stream:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)
```

### 4.4 Offline Python API

The `LLM` class provides a high-level interface for batch inference without a server:

```python
from vllm import LLM, SamplingParams

# Initialize (downloads model on first run)
llm = LLM(model="meta-llama/Llama-3.1-8B-Instruct")

# Define sampling parameters
params = SamplingParams(
    temperature=0.8,
    top_p=0.95,
    max_tokens=512
)

# Batch generation
prompts = [
    "Explain the Pythagorean theorem.",
    "What is the capital of Japan?",
    "Write a limerick about Python programming."
]
outputs = llm.generate(prompts, params)

for output in outputs:
    print(f"Prompt: {output.prompt!r}")
    print(f"Output: {output.outputs[0].text!r}")
    print("---")
```

#### Chat Interface (Offline)

```python
from vllm import LLM, SamplingParams

llm = LLM(model="meta-llama/Llama-3.1-8B-Instruct")

conversation = [
    {"role": "system", "content": "You are a concise assistant."},
    {"role": "user", "content": "What is 2 + 2?"},
]

outputs = llm.chat(conversation, SamplingParams(temperature=0, max_tokens=50))
print(outputs[0].outputs[0].text)
```

#### Embeddings (Offline)

```python
from vllm import LLM

llm = LLM(model="intfloat/e5-mistral-7b-instruct", task="embed")
outputs = llm.embed(["Hello, world!", "vLLM is fast."])
for output in outputs:
    print(output.outputs.embedding[:5])  # first 5 dimensions
```

#### Classification (Offline)

```python
from vllm import LLM

llm = LLM(model="your-classification-model", task="classify")
outputs = llm.classify(["This movie is great!", "Terrible experience."])
for output in outputs:
    print(output.outputs.probs)
```

#### Scoring / Reranking (Offline)

```python
from vllm import LLM

llm = LLM(model="your-reranker-model", task="score")
outputs = llm.score(
    query="What is PagedAttention?",
    texts=["PagedAttention is...", "Paris is the capital..."]
)
for output in outputs:
    print(output.outputs.score)
```

### 4.5 Quantization Methods

Quantization reduces VRAM usage and often increases throughput, at a small quality cost:

| Method | Flag value | Notes |
|---|---|---|
| FP8 | `fp8` | NVIDIA H100/H200, AMD MI300X natively; emulated elsewhere |
| MXFP8 | `mxfp8` | Microscaling FP8 |
| MXFP4 | `mxfp4` | Microscaling FP4 |
| NVFP4 | `nvfp4` | NVIDIA FP4 |
| INT8 (SmoothQuant) | `int8` | Post-training quantization |
| INT4 (GPTQ) | `gptq` | GPTQ-quantized HuggingFace models |
| AWQ | `awq` | Activation-aware Weight Quantization |
| GGUF | `gguf` | GGUF-format models (llama.cpp compat) |
| compressed-tensors | `compressed-tensors` | Neural Magic format |
| ModelOpt | `modelopt` | NVIDIA ModelOpt format |
| TorchAO | `torchao` | PyTorch native quantization |
| BitsAndBytes | `bitsandbytes` | 4-bit/8-bit NF4/INT8 |

Apply quantization:

```bash
vllm serve TheBloke/Llama-2-7B-GPTQ --quantization gptq
vllm serve your-fp8-model --quantization fp8
```

> [!NOTE]
> Online quantization (quantizing at load time) is supported for some methods. Pre-quantized models from HuggingFace Hub load faster. Check the model card to see if a pre-quantized variant exists before quantizing at load time.

### 4.6 Model Types Supported

| Category | Examples |
|---|---|
| Decoder-only LLMs | Llama 3, Qwen 2/3, Gemma 2/3 |
| Mixture-of-Experts | Mixtral, DeepSeek-V3, Qwen-MoE, GPT-OSS |
| Hybrid Attention/SSM | Mamba, Qwen3.5 |
| Multimodal (vision) | LLaVA, Qwen-VL, Pixtral |
| Embedding/Retrieval | E5-Mistral, GTE, ColBERT |
| Reward/Classification | Qwen-Math |

vLLM supports 200+ model architectures. If your model is not listed, check the official vLLM docs for the full architecture support matrix.

---

## 5. Common Tasks & Workflow Recipes

### 5.1 Single-GPU Serving (Default)

```bash
# Serve Llama 3.1 8B on a single GPU
vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --dtype bfloat16 \
  --gpu-memory-utilization 0.90 \
  --max-model-len 8192
```

### 5.2 Multi-GPU with Tensor Parallelism

Tensor parallelism splits individual layers across GPUs. Use it when a model does not fit on one GPU, or to increase throughput:

```bash
# 4-GPU tensor parallel for Llama 3.1 70B
vllm serve meta-llama/Llama-3.1-70B-Instruct \
  --tensor-parallel-size 4 \
  --dtype bfloat16
```

> [!IMPORTANT]
> Tensor parallel size must divide evenly into the number of attention heads in the model. For example, a model with 32 attention heads supports `-tp 1`, `-tp 2`, `-tp 4`, `-tp 8`, and `-tp 16`, but not `-tp 3` or `-tp 5`. Check the model config if you get a validation error.

### 5.3 Multi-Node with Pipeline Parallelism

Pipeline parallelism splits layers across nodes. Combine with tensor parallelism for very large models:

```bash
# 8 GPUs across 2 nodes: 4-way TP, 2-way PP
vllm serve meta-llama/Llama-3.1-405B-Instruct \
  --tensor-parallel-size 4 \
  --pipeline-parallel-size 2 \
  --dtype bfloat16
```

See the Admin Guide for multi-node Ray cluster setup.

### 5.4 Data Parallelism

Run multiple full model replicas for higher throughput:

```bash
vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --data-parallel-size 4 \
  --tensor-parallel-size 1
```

### 5.5 Expert Parallelism for MoE Models

For Mixture-of-Experts models like Mixtral or DeepSeek-V3:

```bash
vllm serve deepseek-ai/DeepSeek-V3 \
  --tensor-parallel-size 4 \
  --enable-expert-parallel \
  --dtype bfloat16
```

### 5.6 Serving a Quantized Model

```bash
# AWQ-quantized Mistral 7B
vllm serve TheBloke/Mistral-7B-Instruct-v0.2-AWQ --quantization awq

# GPTQ-quantized model
vllm serve TheBloke/Llama-2-13B-GPTQ --quantization gptq

# FP8
vllm serve neuralmagic/Meta-Llama-3-8B-Instruct-FP8 --quantization fp8
```

### 5.7 LoRA Adapter Serving

Enable multi-LoRA serving to dynamically load adapters per request:

```bash
vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --enable-lora \
  --max-loras 4
```

Then specify a LoRA in your request:

```bash
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Llama-3.1-8B-Instruct",
    "messages": [{"role": "user", "content": "Translate to Spanish: Hello world"}],
    "lora_request": {
      "lora_name": "my-spanish-lora",
      "lora_int_id": 1,
      "lora_path": "/path/to/lora/adapter"
    }
  }'
```

### 5.8 Prefix Caching

Enable prefix caching to speed up requests that share a common prompt prefix (system prompts, few-shot examples, RAG context):

```bash
vllm serve meta-llama/Llama-3.1-8B-Instruct --enable-prefix-caching
```

> [!TIP]
> Prefix caching is particularly effective for RAG applications where many requests share the same large retrieved context, and for chat applications where the system prompt is identical across sessions. Time-to-first-token drops dramatically for repeated prefixes.

### 5.9 Chunked Prefill

Chunked prefill allows long prompts to be processed incrementally alongside generation, preventing latency spikes for other requests:

```bash
vllm serve meta-llama/Llama-3.1-8B-Instruct --enable-chunked-prefill
```

### 5.10 Speculative Decoding

Speculative decoding accelerates generation by using a smaller draft model to propose tokens, then verifying with the main model in parallel. Several strategies are supported:

#### Draft Model

```bash
vllm serve meta-llama/Llama-3.1-70B-Instruct \
  --speculative-config '{"model": "meta-llama/Llama-3.1-8B-Instruct", "num_speculative_tokens": 5}'
```

#### EAGLE (Efficient Autoregressive Language Modeling)

```bash
vllm serve meta-llama/Llama-3.1-70B-Instruct \
  --speculative-config '{"method": "eagle", "model": "path/to/eagle-head"}'
```

#### N-gram Speculative Decoding

Uses n-gram matching in the prompt for speculation — no draft model required:

```bash
vllm serve meta-llama/Llama-3.1-70B-Instruct \
  --speculative-config '{"method": "ngram", "num_speculative_tokens": 5}'
```

> [!NOTE]
> Speculative decoding reduces latency when generation is memory-bandwidth bound and draft acceptance rates are high. It has no effect (or may slightly reduce throughput) at high request rates. Best suited for low-concurrency, latency-sensitive workloads. Known issue: thinking token budgets may not be enforced when MTP speculative decoding is active (see GitHub issue #39573).

### 5.11 Structured Output Generation

Force the model to produce valid JSON, matching a schema, or following a grammar:

```bash
vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --structured-outputs-config '{"backend": "xgrammar"}'
```

Then in your request:

```bash
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Llama-3.1-8B-Instruct",
    "messages": [{"role": "user", "content": "Extract: John Smith, age 30, engineer"}],
    "guided_json": {
      "type": "object",
      "properties": {
        "name": {"type": "string"},
        "age": {"type": "integer"},
        "occupation": {"type": "string"}
      },
      "required": ["name", "age", "occupation"]
    }
  }'
```

### 5.12 Multimodal Inference (Vision)

For vision-language models like LLaVA or Qwen-VL:

```bash
vllm serve llava-hf/llava-1.5-7b-hf --tokenizer-mode auto
```

Send images in your request:

```bash
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llava-hf/llava-1.5-7b-hf",
    "messages": [
      {
        "role": "user",
        "content": [
          {"type": "image_url", "image_url": {"url": "https://example.com/image.jpg"}},
          {"type": "text", "text": "What is in this image?"}
        ]
      }
    ],
    "max_tokens": 256
  }'
```

For video and audio, use the same multi-content format with appropriate media types.

Control media fetch timeouts:

```bash
export VLLM_IMAGE_FETCH_TIMEOUT=10   # seconds
export VLLM_VIDEO_FETCH_TIMEOUT=60   # seconds
export VLLM_AUDIO_FETCH_TIMEOUT=20   # seconds
export VLLM_MEDIA_CACHE_MAX_SIZE_MB=10240  # 10 GiB cache
```

### 5.13 Embeddings Server

```bash
vllm serve intfloat/e5-mistral-7b-instruct
```

Generate embeddings:

```bash
curl http://localhost:8000/v1/embeddings \
  -H "Content-Type: application/json" \
  -d '{
    "model": "intfloat/e5-mistral-7b-instruct",
    "input": ["Hello, world!", "vLLM embeddings are fast."]
  }'
```

For Cohere-compatible clients:

```bash
curl http://localhost:8000/v2/embed \
  -H "Content-Type: application/json" \
  -d '{
    "model": "intfloat/e5-mistral-7b-instruct",
    "texts": ["Hello", "World"]
  }'
```

### 5.14 Tokenization Endpoints

Inspect tokens without generating:

```bash
# Tokenize
curl http://localhost:8000/tokenize \
  -H "Content-Type: application/json" \
  -d '{"model": "meta-llama/Llama-3.1-8B-Instruct", "prompt": "Hello, world!"}'

# Detokenize
curl http://localhost:8000/detokenize \
  -H "Content-Type: application/json" \
  -d '{"model": "meta-llama/Llama-3.1-8B-Instruct", "tokens": [15496, 11, 995, 0]}'
```

### 5.15 Tool Calling

vLLM supports OpenAI-format tool calling:

```bash
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Llama-3.1-8B-Instruct",
    "messages": [{"role": "user", "content": "What is the weather in Paris?"}],
    "tools": [
      {
        "type": "function",
        "function": {
          "name": "get_weather",
          "description": "Get current weather for a city",
          "parameters": {
            "type": "object",
            "properties": {
              "city": {"type": "string"}
            },
            "required": ["city"]
          }
        }
      }
    ],
    "tool_choice": "auto"
  }'
```

### 5.16 Reasoning Models

Enable reasoning output parsing for thinking models:

```bash
vllm serve your-reasoning-model \
  --reasoning-config '{"backend": "deepseek_r1"}'
```

> [!WARNING]
> Known limitation: when using MTP speculative decoding alongside reasoning models, thinking token budgets may not be enforced correctly (GitHub issue #39573). If you need strict thinking budget control, disable speculative decoding with these models.

### 5.17 Serving with SSL/TLS

```bash
vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --ssl-keyfile /path/to/server.key \
  --ssl-certfile /path/to/server.crt \
  --host 0.0.0.0 \
  --port 443
```

### 5.18 Limiting GPU Selection

Use environment variables to restrict which GPUs vLLM sees:

```bash
# NVIDIA — use only GPUs 0 and 1
CUDA_VISIBLE_DEVICES=0,1 vllm serve meta-llama/Llama-3.1-8B-Instruct --tensor-parallel-size 2

# AMD ROCm — use only GPU 0
HIP_VISIBLE_DEVICES=0 vllm serve meta-llama/Llama-3.1-8B-Instruct
```

### 5.19 Named API Model Alias

Expose a model under a different name for API compatibility:

```bash
vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --served-model-name gpt-3.5-turbo
```

Clients can then use `"model": "gpt-3.5-turbo"` in requests without knowing the actual model path.

### 5.20 Benchmarking

vLLM includes a comprehensive benchmarking suite:

```bash
# Benchmark serving throughput (requires running server)
vllm bench serve \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --dataset-name sharegpt \
  --num-prompts 100

# Benchmark per-request latency (standalone, no server needed)
vllm bench latency \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --input-len 512 \
  --output-len 128 \
  --num-iters 20

# Benchmark offline throughput
vllm bench throughput \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --input-len 512 \
  --output-len 128 \
  --num-prompts 200

# Benchmark multimodal preprocessor
vllm bench mm-processor --model llava-hf/llava-1.5-7b-hf

# Benchmark server startup time
vllm bench startup --model meta-llama/Llama-3.1-8B-Instruct

# Parameter sweep
vllm bench sweep --model meta-llama/Llama-3.1-8B-Instruct
```

### 5.21 Disabling CUDA Graphs (Eager Mode)

CUDA graphs improve throughput but increase startup time and VRAM. Disable them for debugging or on older GPUs:

```bash
vllm serve meta-llama/Llama-3.1-8B-Instruct --enforce-eager
```

### 5.22 Custom Tokenizer Mode

Select the tokenizer backend explicitly:

```bash
# Default — auto-detect
vllm serve your-model --tokenizer-mode auto

# For Mistral models (use official Mistral tokenizer)
vllm serve mistralai/Mistral-7B-Instruct-v0.3 --tokenizer-mode mistral

# For DeepSeek V3.2
vllm serve deepseek-ai/DeepSeek-V3 --tokenizer-mode deepseek_v32

# For Qwen VL models
vllm serve Qwen/Qwen-VL-Chat --tokenizer-mode qwen_vl
```

### 5.23 torch.compile Integration

Enable torch.compile for potential throughput improvements on supported hardware:

```bash
vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --compilation-config '{"level": 3}'
```

### 5.24 Disaggregated Prefill/Decode

Separate prefill and decode phases across different instances for latency optimization:

```bash
# Configure KV transfer for disaggregated serving
vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --kv-transfer-config '{"kv_connector": "PyNcclConnector", "kv_role": "kv_producer"}'
```

> [!WARNING]
> Known issue: the `OffloadingConnector` for GPU-to-CPU KV offload may crash in certain configurations (GitHub issue #39491). If you need KV offloading, test thoroughly before deploying to production and monitor for crashes.

See the Admin Guide for full disaggregated prefill/decode architecture.

### 5.25 CPU Inference

For environments without GPUs:

```bash
export VLLM_TARGET_DEVICE=cpu
export VLLM_CPU_KVCACHE_SPACE=16  # GiB of RAM for KV cache

vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --dtype bfloat16 \
  --max-model-len 4096
```

> [!NOTE]
> CPU inference is significantly slower than GPU inference for most workloads. It is best suited for development, testing, or low-concurrency use cases where GPU hardware is unavailable. For Apple Silicon Macs, performance is improved by using the MPS backend.

### 5.26 Collect Environment Info for Bug Reports

Before filing a GitHub issue, collect environment information:

```bash
vllm collect-env
```

This outputs CUDA version, PyTorch version, GPU info, installed packages, and other diagnostic data. Include this output in all bug reports.

### 5.27 Rendering Server (GPU-less)

Start a rendering server that does not require a GPU:

```bash
vllm launch render
```

### 5.28 Logging Configuration

Control log verbosity:

```bash
# Set log level
export VLLM_LOGGING_LEVEL=DEBUG

# Use a custom Python logging config
export VLLM_LOGGING_CONFIG_PATH=/path/to/logging.json

# Disable internal stats logging
vllm serve meta-llama/Llama-3.1-8B-Instruct --disable-log-stats

# Disable all vLLM logging
export VLLM_CONFIGURE_LOGGING=0
```

### 5.29 Serving a Local Model

Use a local directory path instead of a HuggingFace model name:

```bash
vllm serve /models/my-fine-tuned-llama \
  --trust-remote-code \
  --dtype bfloat16
```

> [!TIP]
> Point `--model` at any directory containing a `config.json` and model weights in HuggingFace format. This works with fine-tuned models, merged LoRA adapters, and models downloaded with `huggingface-cli download`.

### 5.30 Serving a Gated HuggingFace Model

```bash
# Authenticate with HuggingFace CLI first
huggingface-cli login

# Then serve normally
vllm serve meta-llama/Meta-Llama-3-70B-Instruct --tensor-parallel-size 4
```

Or pass the token via environment variable:

```bash
HUGGING_FACE_HUB_TOKEN=hf_your_token vllm serve meta-llama/Meta-Llama-3-70B-Instruct
```

---

## 6. Troubleshooting

### Error: `CUDA out of memory`

**Cause:** The model plus KV cache exceeds available GPU VRAM.

**Solution:**
1. Reduce `--gpu-memory-utilization` (try `0.80` or `0.75`).
2. Reduce `--max-model-len` to limit KV cache size.
3. Enable quantization: `--quantization awq` or `--quantization gptq`.
4. Reduce `--max-num-seqs` to limit concurrent sequences.
5. Use tensor parallelism across more GPUs: `--tensor-parallel-size 2`.
6. Enable chunked prefill: `--enable-chunked-prefill`.

**Prevention:** Leave at least 10% GPU memory headroom. Always set `--gpu-memory-utilization 0.90` or lower for production deployments.

---

### Error: `The model's max seq len (X) is larger than the maximum number of tokens that can be stored in KV cache`

**Cause:** `--max-model-len` is set higher than what fits in GPU memory given current `--gpu-memory-utilization`.

**Solution:**
1. Reduce `--max-model-len` to match your actual use case.
2. Increase `--gpu-memory-utilization` (up to ~0.95 on a dedicated GPU).
3. Add more GPUs and increase `--tensor-parallel-size`.
4. Use quantization to reduce memory per parameter.

**Prevention:** Set `--max-model-len` explicitly to the maximum context length your application needs, rather than using the model's full supported length.

---

### Error: `ValueError: tensor parallel size (X) does not divide attention head count (Y)`

**Cause:** The `--tensor-parallel-size` value does not evenly divide the number of attention heads.

**Solution:**
1. Check the model's `config.json` for `num_attention_heads`.
2. Choose a `-tp` value that divides that number evenly (power-of-2 values typically work: 1, 2, 4, 8).

**Prevention:** Always verify that `-tp N` divides the model's attention head count before starting the server.

---

### Error: `ImportError: No module named 'vllm'`

**Cause:** vLLM is not installed in the current Python environment.

**Solution:**
1. Activate the correct virtual environment: `source vllm-env/bin/activate`.
2. Verify installation: `pip show vllm`.
3. Reinstall if needed: `pip install vllm`.

**Prevention:** Always activate your virtual environment before running vLLM commands.

---

### Error: `requests.exceptions.ConnectionError: Failed to establish a new connection`

**Cause:** The vLLM server is not running, or the client is pointed at the wrong host/port.

**Solution:**
1. Check that `vllm serve` is running and did not exit with an error.
2. Verify the server address and port match (`--host` and `--port`).
3. Check health: `curl http://localhost:8000/health`.
4. Check for firewall rules blocking the port.

**Prevention:** Always run a health check after starting the server before sending inference requests.

---

### Error: `Model is not found` (404 from `/v1/chat/completions`)

**Cause:** The `"model"` field in your request does not match the model name vLLM loaded.

**Solution:**
1. Call `GET /v1/models` to see the exact model name the server reports.
2. Use `--served-model-name your-alias` if you want to use a custom name.
3. Update the `"model"` field in your request to match.

**Prevention:** Use `GET /v1/models` programmatically in your client to discover the active model name rather than hardcoding it.

---

### Error: `NCCL error` during multi-GPU startup

**Cause:** NCCL communication initialization failed, usually due to network interface configuration or firewall rules between GPUs/nodes.

**Solution:**
1. Set `NCCL_DEBUG=INFO` to get detailed NCCL logs.
2. Verify all GPUs are visible: `nvidia-smi`.
3. For multi-node: ensure all nodes can reach each other on NCCL ports.
4. Try `--enforce-eager` to bypass CUDA graph issues.

**Prevention:** Test single-GPU serving before scaling to multi-GPU. For multi-node, test network connectivity between nodes before starting vLLM.

---

### Error: `RuntimeError: Expected all tensors to be on the same device`

**Cause:** Version mismatch between vLLM, PyTorch, and CUDA, or a bug in a specific model's code path.

**Solution:**
1. Run `vllm collect-env` and check for version mismatches.
2. Reinstall vLLM in a fresh virtual environment.
3. Try `--enforce-eager` to disable CUDA graphs.
4. File a bug report with the output of `vllm collect-env`.

**Prevention:** Always install vLLM in an isolated virtual environment. Check the release notes for supported PyTorch versions.

---

### Error: `ValueError: trust_remote_code must be set to True`

**Cause:** The model requires executing custom modeling code from HuggingFace Hub, but `--trust-remote-code` was not passed.

**Solution:**
1. Add `--trust-remote-code` to your `vllm serve` command.

**Prevention:** Review the model card before trusting remote code. Only use this flag for models from sources you trust.

---

### Error: `httpx.ReadTimeout` or client timeout

**Cause:** The model is taking longer to respond than the client's configured timeout.

**Solution:**
1. Increase the client timeout (e.g., `httpx.Client(timeout=120.0)`).
2. Reduce `max_tokens` in your request.
3. Enable streaming so partial output arrives before the timeout.
4. Check GPU utilization — if the GPU is not being used, there may be a larger problem.

**Prevention:** Set client timeouts appropriate to your expected `max_tokens`. For long outputs, always use streaming.

---

### Error: `TokenizerMismatch` or garbled output

**Cause:** The tokenizer mode does not match the model's expected tokenizer.

**Solution:**
1. Specify the correct `--tokenizer-mode` for your model family.
2. For Mistral models: `--tokenizer-mode mistral`.
3. For DeepSeek V3.2: `--tokenizer-mode deepseek_v32`.
4. For Qwen-VL: `--tokenizer-mode qwen_vl`.

**Prevention:** Always check the model card for tokenizer requirements before serving.

---

### Error: MoE kernel config fails on GH200

**Cause:** Benchmark tuning for MoE kernels fails on NVIDIA GH200 hardware (GitHub issue #37191).

**Solution:**
1. Skip benchmark tuning when running on GH200.
2. Monitor the GitHub issue for a fix.

**Prevention:** Check hardware compatibility in the vLLM issue tracker when using bleeding-edge NVIDIA hardware.

---

### Error: Ngram speculative decoding produces corrupted output on hybrid models

**Cause:** N-gram speculative decoding produces incorrect output on hybrid GDN (gated-dense-network) models (GitHub issue #39273).

**Solution:**
1. Disable ngram speculative decoding for hybrid GDN models.
2. Use a draft model or EAGLE instead.

**Prevention:** Do not use ngram speculative decoding on Mamba or hybrid SSM/attention architectures.

---

### Error: Mamba prefix caching + MTP crashes on NemotronH

**Cause:** Combining Mamba prefix caching with MTP (multi-token prediction) speculative decoding on NemotronH models causes a crash (GitHub issue #39809).

**Solution:**
1. Disable prefix caching when using MTP with Mamba-based models: remove `--enable-prefix-caching`.
2. Or disable MTP speculative decoding.

**Prevention:** Do not combine prefix caching and MTP speculative decoding on Mamba/NemotronH architectures until this issue is resolved.

---

### Error: `OffloadingConnector` GPU-to-CPU KV crash

**Cause:** The `OffloadingConnector` for disaggregated prefill/decode GPU-to-CPU KV transfer crashes in certain configurations (GitHub issue #39491).

**Solution:**
1. Do not use `OffloadingConnector` in production until the issue is resolved.
2. Use an alternative KV connector (`PyNcclConnector` if applicable).

**Prevention:** Monitor GitHub issue #39491 for status and patches.

---

### Error: Thinking token budget not enforced with MTP speculative decoding

**Cause:** When using MTP (multi-token prediction) speculative decoding with reasoning models, the configured thinking token budget is not respected (GitHub issue #39573).

**Solution:**
1. Disable speculative decoding when strict thinking budget enforcement is required.
2. Set longer budgets as a workaround.

**Prevention:** Do not combine MTP speculative decoding with thinking-budget-sensitive reasoning workflows.

---

### Server starts but requests hang indefinitely

**Cause:** The model is still loading (can take several minutes for large models), or there is a deadlock in the scheduler.

**Solution:**
1. Check server logs for progress messages — model loading is normal and can take 1-10 minutes.
2. Poll `GET /health` — it returns 200 only when the server is ready.
3. If the server is stuck after model loading completes, restart it.

**Prevention:** Always wait for the server to report healthy before sending requests. Implement a health check loop in your startup scripts.

---

### High latency under load

**Cause:** GPU memory is full causing KV cache eviction, or `--max-num-batched-tokens` is too low.

**Solution:**
1. Enable chunked prefill: `--enable-chunked-prefill`.
2. Enable prefix caching: `--enable-prefix-caching`.
3. Increase `--max-num-batched-tokens`.
4. Reduce `--max-model-len` to free KV cache space.
5. Add more GPUs or replicas.

**Prevention:** Benchmark with `vllm bench serve` at your expected concurrency level before going to production.

---

### Model loads but generates empty or nonsensical output

**Cause:** Wrong dtype, quantization mismatch, or the prompt format does not match the model's training.

**Solution:**
1. Ensure you are using the model's expected chat template format.
2. Try `--dtype bfloat16` explicitly.
3. Verify the quantization method matches the model's format.
4. Use `--enforce-eager` to rule out CUDA graph issues.

**Prevention:** Check the model card for the recommended `dtype` and prompt format.

---

### Out of CPU memory during model loading

**Cause:** Model is too large for system RAM during weight loading and conversion.

**Solution:**
1. Reduce the number of worker processes.
2. Use a pre-quantized model to reduce weight size.
3. Close other applications consuming RAM.
4. Use a machine with more RAM.

**Prevention:** Ensure system RAM is at least 2x the model's disk size to accommodate loading and conversion buffers.

---

### Image/video/audio fetch fails or times out in multimodal requests

**Cause:** Default fetch timeouts are too short for large media files or slow network connections.

**Solution:**
1. Increase timeouts:
   ```bash
   export VLLM_IMAGE_FETCH_TIMEOUT=30
   export VLLM_VIDEO_FETCH_TIMEOUT=120
   export VLLM_AUDIO_FETCH_TIMEOUT=60
   ```
2. Pre-download media and serve from localhost.
3. Increase `VLLM_MEDIA_CACHE_MAX_SIZE_MB` to cache more media.

**Prevention:** For production multimodal workloads, serve media from a local or fast CDN rather than fetching from arbitrary URLs at inference time.

---

### Port already in use

**Cause:** Another process (or a previous vLLM instance) is already using port 8000.

**Solution:**
1. Find the process: `lsof -i :8000` (Linux/macOS) or `netstat -ano | findstr :8000` (Windows).
2. Kill the old process or use a different port: `vllm serve <model> --port 8001`.

**Prevention:** Use a process manager (systemd, supervisor) to ensure only one vLLM instance runs per port.

---

### ROCm: Model not running on GPU

**Cause:** `VLLM_TARGET_DEVICE` not set, or `HIP_VISIBLE_DEVICES` not configured correctly.

**Solution:**
1. Set `export VLLM_TARGET_DEVICE=rocm`.
2. Verify GPU visibility: `rocm-smi`.
3. Set `HIP_VISIBLE_DEVICES=0,1,...` explicitly.

**Prevention:** Always set `VLLM_TARGET_DEVICE=rocm` in your environment when running on AMD GPUs.

---

### Structured output backend errors

**Cause:** The `xgrammar` or `guidance` backend may not be installed or may conflict with your Python environment.

**Solution:**
1. Install the appropriate backend: `pip install xgrammar` or `pip install guidance`.
2. Ensure no version conflicts with existing packages.

**Prevention:** Install structured output dependencies before enabling structured output configs.

---

### `WARNING: Your system has insufficient swap space`

**Cause:** vLLM tries to use swap for KV cache overflows, but swap is unavailable.

**Solution:**
1. Add swap space on Linux: `sudo fallocate -l 32G /swapfile && sudo mkswap /swapfile && sudo swapon /swapfile`.
2. Reduce `--gpu-memory-utilization` to leave more headroom.

**Prevention:** Provision servers with adequate RAM and swap for your workload.

---

### Server exits immediately after starting

**Cause:** Missing required flags, unsupported GPU, or dependency error.

**Solution:**
1. Check the terminal output for the specific error message.
2. Run `vllm collect-env` to verify environment compatibility.
3. Ensure `--model` points to a valid model name or path.
4. Try with `--enforce-eager` and minimal flags to isolate the issue.

**Prevention:** Always test with a small known-good model (e.g., `facebook/opt-125m`) before serving your target model.

---

### `AttributeError` or `ModuleNotFoundError` after upgrading vLLM

**Cause:** Breaking API changes between vLLM versions, or partially upgraded dependencies.

**Solution:**
1. Reinstall vLLM in a clean virtual environment: `pip install --force-reinstall vllm`.
2. Check the vLLM changelog for breaking changes between versions.
3. Update any vLLM-specific code to match the new API.

**Prevention:** Pin vLLM to a specific version in production and upgrade intentionally with testing.

---

> [!CAUTION]
> Never expose the vLLM server on a public IP without authentication (`VLLM_API_KEY`) and rate limiting. An unauthenticated vLLM server allows anyone to consume your GPU resources and generate arbitrary text. Always place the server behind a reverse proxy or firewall in production.

> [!WARNING]
> Using `--trust-remote-code` executes Python code downloaded from HuggingFace Hub. Only use this flag with models from organizations you fully trust. Malicious model code could exfiltrate data, install backdoors, or compromise your server.

---

## 7. FAQ

**Q: What is PagedAttention?**

PagedAttention is vLLM's core memory management innovation. It manages the KV cache in fixed-size "pages" (similar to OS virtual memory) rather than allocating contiguous blocks. This eliminates internal fragmentation and allows much higher GPU memory utilization, enabling more concurrent requests than systems that pre-allocate per-sequence contiguous memory.

---

**Q: How does vLLM compare to other serving frameworks?**

vLLM focuses on high throughput and memory efficiency via PagedAttention and continuous batching. It provides a fully OpenAI-compatible API, making it a drop-in replacement. For most multi-user serving scenarios, vLLM's continuous batching provides best-in-class GPU utilization.

---

**Q: What is continuous batching?**

Continuous batching (also called in-flight batching) processes new requests as soon as GPU capacity is available, rather than waiting for an entire batch to finish. This means each GPU step can contain tokens from different requests at different stages of generation, maximizing utilization and reducing average latency.

---

**Q: Can I run vLLM without a GPU?**

Yes. Set `VLLM_TARGET_DEVICE=cpu` and use `VLLM_CPU_KVCACHE_SPACE` to allocate RAM for the KV cache. CPU inference is significantly slower than GPU inference but works on any modern Intel/AMD x86 machine, ARM AArch64, Apple Silicon, or IBM Z.

---

**Q: What models does vLLM support?**

vLLM supports 200+ model architectures, including decoder-only LLMs (Llama, Qwen, Gemma), MoE models (Mixtral, DeepSeek-V3), multimodal models (LLaVA, Qwen-VL, Pixtral), embedding models (E5-Mistral, GTE, ColBERT), and classification/reward models. Check the official vLLM documentation for the full, up-to-date list.

---

**Q: What is the difference between tensor parallelism and pipeline parallelism?**

Tensor parallelism (`-tp`) splits individual model layers across GPUs — each GPU computes part of every layer in every step. It reduces per-layer latency but requires fast GPU interconnect (NVLink preferred). Pipeline parallelism (`-pp`) assigns different layers to different GPUs. It works better over slower interconnects but introduces pipeline bubble overhead. They can be combined.

---

**Q: Can I serve multiple models at once?**

A single `vllm serve` instance serves one model. To serve multiple models, run multiple `vllm serve` instances on different ports and route requests with a load balancer or API gateway.

---

**Q: What is the difference between LoRA adapters and full model serving?**

A base model plus LoRA adapters allows dynamic switching of fine-tuned behaviors without loading separate full models. With `--enable-lora --max-loras 4`, vLLM keeps up to 4 adapters hot in GPU memory and applies the requested one per request. This is far more memory-efficient than loading multiple full fine-tunes when the adapters share the same base model.

---

**Q: Does vLLM support streaming responses?**

Yes. Add `"stream": true` to any `/v1/chat/completions` or `/v1/completions` request to receive Server-Sent Events as tokens are generated.

---

**Q: How do I serve a fine-tuned model?**

Point `--model` at the local directory containing your fine-tuned weights in HuggingFace format. If your fine-tune merges the LoRA into the base model, serve it like any other model. If it is a separate LoRA adapter, use `--enable-lora` and specify the adapter path per request.

---

**Q: What quantization method should I use?**

For NVIDIA H100/H200 or AMD MI300X, FP8 is the best tradeoff of quality and speed with native hardware support. For older NVIDIA GPUs, AWQ or GPTQ offer good quality at 4-bit. BitsAndBytes is the easiest to set up and works broadly. See §4.5 for the full list.

---

**Q: How do I check if my model loaded successfully?**

Poll `GET /health` — it returns HTTP 200 when the server is ready. Also call `GET /v1/models` to confirm the model name is registered.

---

**Q: What is prefix caching and when should I enable it?**

Prefix caching stores the KV activations for the shared prefix of many requests (e.g., a system prompt or RAG context) so they are not recomputed for each request. Enable it with `--enable-prefix-caching`. It is most beneficial when many requests share a long common prefix. It has negligible overhead when prefixes are not repeated.

---

**Q: What is chunked prefill?**

Chunked prefill processes long prompt prefills incrementally instead of all at once. This prevents a single long request from monopolizing the GPU and stalling shorter requests waiting in the queue. Enable with `--enable-chunked-prefill`. It generally improves latency fairness at the cost of slightly lower peak throughput.

---

**Q: How do I use speculative decoding?**

Pass a `--speculative-config` JSON string with the method and parameters. Options include `draft_model` (small draft model), `eagle`, `ngram`, `suffix`, `pard`, and `mlp`. Speculative decoding reduces latency when the draft model's token predictions are accepted at high rates by the main model. See §5.10 for examples.

---

**Q: Is vLLM's API compatible with the OpenAI Python SDK?**

Yes. Set `base_url="http://your-vllm-host:8000/v1"` in the `OpenAI` client constructor. All standard OpenAI API methods work. vLLM also supports the Anthropic Messages API format.

---

**Q: How do I set up authentication?**

Set the `VLLM_API_KEY` environment variable before starting the server. Clients must then pass `Authorization: Bearer <key>` in all requests. See §3.4 for details.

---

**Q: How do I disable usage statistics collection?**

Set `VLLM_NO_USAGE_STATS=1` or `VLLM_DO_NOT_TRACK=1` in your environment before starting vLLM.

---

**Q: What Python versions are supported?**

Python 3.9 or later is required. Python 3.12 is the recommended version (used in the official Docker image).

---

**Q: What is the `--served-model-name` flag for?**

It sets the model name returned by `GET /v1/models` and accepted in request `"model"` fields. Use it to present a friendly alias (e.g., `gpt-4-turbo`) without exposing the actual HuggingFace model path, or to ensure compatibility with clients that expect a specific model name.

---

**Q: How do I report a bug?**

1. Run `vllm collect-env` and copy the output.
2. Reproduce the issue with the minimal configuration.
3. File an issue at `https://github.com/vllm-project/vllm/issues` with the env output, minimal reproduction steps, and error messages.

---

**Q: What is `--dtype auto` and when should I change it?**

`auto` selects the dtype based on the model's config (usually `bfloat16` for modern models). Override with `--dtype float16` for GPUs that don't support bfloat16, or `--dtype float32` for full precision (very high VRAM cost). Only change from `auto` if you know your GPU or model requires a specific dtype.

---

**Q: How does vLLM handle requests when the GPU is at capacity?**

vLLM queues incoming requests. The continuous batching scheduler adds new requests to in-flight batches as GPU capacity becomes available. No request is dropped unless you implement rate limiting at a proxy layer.

---

**Q: Can I use vLLM on Windows?**

vLLM is primarily developed for Linux. Windows users should use WSL2 (Windows Subsystem for Linux) with CUDA support, or Docker Desktop with GPU access. Native Windows installation is not officially supported.

---

**Q: What is `VLLM_TARGET_DEVICE=empty`?**

`empty` is a special device target for testing and development — it loads the vLLM framework without initializing any hardware backend. It is used for CI tests and development workflows that don't require actual GPU computation.

---

**Q: How do I maximize throughput?**

1. Enable continuous batching (enabled by default).
2. Enable prefix caching (`--enable-prefix-caching`) if requests share prefixes.
3. Enable chunked prefill (`--enable-chunked-prefill`).
4. Tune `--max-num-seqs` and `--max-num-batched-tokens` for your workload.
5. Use quantization to fit more KV cache in VRAM.
6. Use tensor parallelism to spread across multiple GPUs.
7. Benchmark with `vllm bench serve` to find the optimal configuration.

---

**Q: Does vLLM support the Anthropic API format?**

Yes. vLLM includes Anthropic Messages API support, allowing Anthropic SDK clients to connect directly to a vLLM server without modification.

---

**Q: What is the Admin Guide for?**

The vLLM Admin Guide covers architecture internals, deployment operations, Docker/container setup, Ray cluster configuration, monitoring, security hardening, and detailed configuration reference for operators and architects running vLLM in production.

---

**Q: Where can I find the full model support list?**

The complete list of 200+ supported architectures is maintained in the official vLLM documentation. The briefing data covers the major categories; check the official docs for your specific model.

---

**Q: How are memory blocks allocated for the KV cache?**

vLLM uses PagedAttention with fixed-size blocks. The `--block-size` flag controls block size. Smaller blocks reduce memory fragmentation for short sequences; larger blocks improve memory bandwidth for long sequences. The default is tuned automatically for most use cases.

---

**Q: Does vLLM support tool calling with streaming?**

Yes. Tool calling works with streaming responses. When the model outputs a tool call, the tool call arguments stream incrementally via SSE before the final response.

---

**Q: Can I run multiple vLLM instances on the same machine?**

Yes. Run each instance on a different port and use different `CUDA_VISIBLE_DEVICES` values to assign specific GPUs to each instance. This is useful for serving different models simultaneously or for A/B testing.

---

**Q: What is the difference between `/v1/embeddings` and `/v2/embed`?**

`/v1/embeddings` follows the OpenAI embeddings API format. `/v2/embed` follows the Cohere embeddings API format. Use whichever matches your client library. Both are served by the same underlying embedding model.

---

**Q: How does vLLM handle very long contexts?**

vLLM supports models with context lengths up to their maximum trained length (e.g., 128K or 1M tokens for some models). Use `--max-model-len` to cap the effective length. PagedAttention manages KV cache memory efficiently for long contexts. For very long inputs, also enable chunked prefill to avoid latency spikes.

---

_Generated May 2026 from vllm-project/vllm at 58da4ee. AI-generated documentation. The library is checked by sampling (three repositories so far); this guide was not individually reviewed. Verify against the project before relying on it. Unofficial; not affiliated with or reviewed by the project._