vLLM — Administrator & Developer Guide

This guide covers architecture, configuration, deployment, operations, performance tuning, security, and developer internals for vLLM — the high-throughput, memory-efficient inference and serving engine for large language models. For installation, everyday usage, and API client examples, see the companion User Guide.

1. Architecture Overview

vLLM is a production-grade LLM inference engine originally developed at UC Berkeley's Sky Computing Lab, now maintained by a community of 2,000+ contributors. It is designed around three core innovations: PagedAttention for memory management, continuous batching for throughput, and a pluggable backend system that abstracts over CUDA, ROCm, XPU, and CPU execution.

1.1 System Topology

                         ┌─────────────────────────────────────────────────┐
                         │                  Client Layer                    │
                         │  (OpenAI SDK / curl / any HTTP client)           │
                         └──────────────────────┬──────────────────────────┘
                                                │ HTTP / HTTPS
                         ┌──────────────────────▼──────────────────────────┐
                         │          OpenAI-Compatible API Server            │
                         │   vllm/entrypoints/openai/api_server.py          │
                         │   FastAPI + Uvicorn + uvloop                      │
                         │   Routes: /v1/chat/completions, /v1/completions,  │
                         │           /v1/embeddings, /v1/models, /health     │
                         └──────────────────────┬──────────────────────────┘
                                                │
                         ┌──────────────────────▼──────────────────────────┐
                         │               Engine Layer (V1)                  │
                         │   vllm/v1/  — AsyncLLMEngine / LLMEngine         │
                         │   EngineArgs / AsyncEngineArgs                   │
                         │   VllmConfig (aggregate config dataclass)        │
                         └────────┬──────────────────────────┬─────────────┘
                                  │                          │
              ┌───────────────────▼──────────┐  ┌───────────▼──────────────┐
              │       Scheduler (V1)          │  │   KV Cache Manager (V1)  │
              │  vllm/v1/core/scheduler.py    │  │   vllm/v1/core/          │
              │  Continuous batching          │  │   kv_cache_manager.py    │
              │  Chunked prefill              │  │   PagedAttention blocks  │
              │  Prefix caching              │  │   Block allocator        │
              └───────────────────┬──────────┘  └──────────────────────────┘
                                  │
              ┌───────────────────▼──────────────────────────────────────┐
              │                    Worker Layer                           │
              │  vllm/v1/worker/  — GPUModelRunner, CPUModelRunner       │
              │  Model execution, KV cache I/O, sampling                 │
              └────────────┬────────────────────────────┬────────────────┘
                           │                            │
         ┌─────────────────▼──────────┐  ┌─────────────▼─────────────────┐
         │     Model Executor          │  │    Distributed Backend        │
         │  vllm/model_executor/       │  │  vllm/distributed/            │
         │  200+ model implementations │  │  Tensor / Pipeline / DP       │
         │  Quantization layers        │  │  NCCL / Ray / gloo            │
         └─────────────────────────────┘  └───────────────────────────────┘

Note

vLLM maintains two engine generations simultaneously. The V1 engine (vllm/v1/) is the current production path with improved scheduler, KV cache manager, and multiprocessing architecture. The legacy engine path remains for compatibility but V1 is the active development target. VLLM_ENABLE_V1_MULTIPROCESSING=1 is enabled by default.

1.2 Request Lifecycle

HTTP Request
    │
    ▼
API Server (FastAPI)
    │  Validate, parse, authenticate (VLLM_API_KEY)
    ▼
AsyncLLMEngine.generate()
    │  Tokenize, build SamplingParams, enqueue
    ▼
Scheduler
    │  Continuous batching — running + waiting + swapped queues
    │  Prefix cache lookup (if enabled)
    │  Chunked prefill (if enabled)
    ▼
GPUModelRunner.execute_model()
    │  Forward pass through model layers
    │  Custom C++/CUDA kernels (attention, MoE, quantization)
    ▼
Sampler
    │  Top-p, top-k, temperature, penalties, logits processing
    ▼
Detokenizer
    │  Stream tokens or return full sequence
    ▼
HTTP Response (streaming SSE or JSON)

1.3 Component Map

1.4 V1 Engine Architecture

The V1 engine (vllm/v1/) represents the current generation design with a clean multiprocessing boundary:

 Main Process (API Server + Engine Orchestrator)
 │
 │  ZMQ / RPC (VLLM_RPC_BASE_PATH)
 │
 ▼
 Engine Core Process
 │   Scheduler — manages sequence groups, block allocation
 │   KV Cache Manager — PagedAttention block table
 │   Prefix Cache — hash-keyed block reuse
 │
 │  NCCL / shared memory
 │
 ▼
 Worker Processes (one per GPU in TP group)
     GPUModelRunner — forward passes
     CUDAGraph capture and replay
     Custom CUDA kernels

1.5 Memory Management: PagedAttention

PagedAttention divides the KV cache into fixed-size blocks (configured via --block-size, default 16 tokens). Blocks are allocated on demand from a global pool, allowing:

• Non-contiguous physical memory for logically contiguous sequences
• Cross-request KV block sharing when prefix caching is enabled
• Swap to CPU when GPU memory is exhausted
• Efficient beam search without memory duplication

Logical KV Cache (sequence view):
  [tok0][tok1][tok2][tok3] | [tok4][tok5][tok6][tok7] | [tok8]...
        Block 0 (full)            Block 1 (full)         Block 2 (partial)

Physical GPU Memory:
  Block pool: [B0][B3][B7][B1][B5]...  (any order, managed by allocator)
  Block table: maps logical → physical

1.6 Continuous Batching

Unlike static batching (one batch per forward pass), continuous batching allows new requests to join and completed requests to leave mid-batch. The scheduler maintains three queues:

• Running: sequences currently executing (KV cache allocated)
• Waiting: sequences queued, not yet admitted
• Swapped: sequences whose KV blocks were evicted to CPU

At each scheduler step, the scheduler determines how many waiting sequences can be admitted (limited by free GPU blocks), optionally preempts low-priority running sequences, and constructs the next model input batch.

1.7 Speculative Decoding

vLLM supports multiple speculative decoding methods to accelerate autoregressive generation:

Configured via --speculative-config / -sc. See Section 3 for parameter details.

1.8 Distributed Execution Strategies

The --distributed-executor-backend flag selects the communication backend: ray (multi-node), mp (multi-process on single node, default), or uni (single process).

2. System Requirements

2.1 Hardware Requirements

GPU Platform Support Matrix

2.2 Software Requirements

2.3 Docker Image Versions

The official Docker build pins (from docker/versions.json):

{
  "cuda": "13.0.0",
  "python": "3.12",
  "ubuntu": "22.04"
}

2.4 Network Ports

2.5 Quantization Hardware Compatibility

*Turing does not support Marlin MXFP4.

Note

FP8 quantization requires Ada Lovelace (compute 8.9) or Hopper (compute 9.0) architecture, or AMD MI300 series. Attempting to use FP8 on older architectures will raise an error at startup. Check the table above before selecting a quantization format for a given deployment target.

3. Configuration Reference

vLLM configuration is split into two layers: CLI flags passed to vllm serve (which map to EngineArgs fields) and environment variables that control runtime and build behavior. All CLI flags are also accessible programmatically via EngineArgs / AsyncEngineArgs.

3.1 VllmConfig Dataclass Hierarchy

The top-level VllmConfig (in vllm/config/) aggregates all sub-configs:

VllmConfig
├── ModelConfig          — model path, dtype, max_model_len, trust_remote_code
├── CacheConfig          — block_size, gpu_memory_utilization, kv_cache_dtype
├── ParallelConfig       — tp_size, pp_size, dp_size, distributed_executor_backend
├── SchedulerConfig      — max_num_batched_tokens, max_num_seqs, scheduling_policy
├── SpeculativeConfig    — draft model, num_speculative_tokens, method
├── LoRAConfig           — max_loras, max_lora_rank, lora_extra_vocab_size
├── CompilationConfig    — optimization_level, custom_ops, splitting_ops
├── AttentionConfig      — attention_backend
├── KernelConfig         — kernel selection
├── ObservabilityConfig  — tracing, metrics, logging
└── ... (additional sub-configs per feature area)

3.2 CLI Serve Flags Reference

Launch the server with:

vllm serve <model> [OPTIONS]

Important

All flags below are sourced from vllm/engine/arg_utils.py. Flag names use hyphens on the CLI (--max-model-len) but map to underscored Python attributes (max_model_len). vLLM validates arguments at startup and exits with an error for unrecognized options. Do not pass flags not documented here.

Model & Tokenizer

Memory & KV Cache

Parallelism

Scheduling

Quantization

Speculative Decoding

LoRA

Attention & Compilation

Reasoning & Tools

KV Transfer (P/D Disaggregation)

Server

Profiling & Misc

3.3 Environment Variables Reference

All environment variables are processed at import time in vllm/envs.py. Variables may be set in the shell, in a .env file, or in a container environment.

Important

Environment variables are read once at Python module import. Changes after the vLLM process has started have no effect. For production deployments, set all environment variables in the systemd unit file, Docker Compose environment section, or Kubernetes ConfigMap/Secret before the process launches.

Build & Compilation

Runtime Core

Logging & Observability

GPU & Memory

Media (Multimodal)

Distributed

Feature Toggles

ROCm-Specific

Additional VLLM_ROCM_* and VLLM_USE_FLASHINFER_MOE_* flags exist for fine-grained AMD kernel control; consult vllm/platforms/ source for the complete list.

3.4 Configuration via Shell Wrapper

vLLM itself does not have a standalone YAML config file — all configuration passes through CLI flags or environment variables. For production, use a shell wrapper script to group all settings:

#!/usr/bin/env bash
# /opt/vllm/start.sh

export VLLM_API_KEY="your-secret-key"
export CUDA_VISIBLE_DEVICES="0,1,2,3"
export VLLM_NO_USAGE_STATS=1
export VLLM_LOGGING_LEVEL=INFO
export VLLM_LOG_STATS_INTERVAL=30

exec vllm serve meta-llama/Meta-Llama-3-70B-Instruct \
  --tensor-parallel-size 4 \
  --gpu-memory-utilization 0.90 \
  --max-model-len 32768 \
  --enable-prefix-caching \
  --port 8000 \
  --host 0.0.0.0

The --hf-overrides flag patches specific HuggingFace config fields at runtime:

vllm serve meta-llama/Meta-Llama-3-70B-Instruct \
  --hf-overrides '{"max_position_embeddings": 131072}' \
  --config-format hf

4. Deployment & Scaling

4.1 Docker Deployment

The official Docker images are published to Docker Hub and GHCR. The multi-stage Dockerfile lives in docker/:

# Pull pre-built image
docker pull vllm/vllm-openai:latest

# Build from source
docker build \
  --build-arg CUDA_VERSION=12.6.0 \
  --build-arg PYTHON_VERSION=3.12 \
  -f docker/Dockerfile \
  -t vllm-custom:latest .

Docker Compose (Single Node)

services:
  vllm:
    image: vllm/vllm-openai:latest
    runtime: nvidia
    environment:
      - NVIDIA_VISIBLE_DEVICES=all
      - VLLM_API_KEY=${VLLM_API_KEY}
      - VLLM_NO_USAGE_STATS=1
      - VLLM_LOGGING_LEVEL=INFO
    volumes:
      - ~/.cache/huggingface:/root/.cache/huggingface
      - /tmp/vllm_media_cache:/tmp/vllm_media_cache
    ports:
      - "8000:8000"
    command: >
      vllm serve meta-llama/Meta-Llama-3.1-8B-Instruct
        --tensor-parallel-size 1
        --gpu-memory-utilization 0.90
        --enable-prefix-caching
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
      interval: 30s
      timeout: 10s
      retries: 5
      start_period: 300s

4.2 Multi-GPU Single-Node (Tensor Parallelism)

Tensor parallelism splits each transformer layer across multiple GPUs. This is the primary scaling method for models that do not fit on a single GPU.

# 4-GPU tensor parallel
vllm serve meta-llama/Meta-Llama-3-70B-Instruct \
  --tensor-parallel-size 4 \
  --gpu-memory-utilization 0.90 \
  --max-model-len 32768

# Check GPU allocation
nvidia-smi

Rules for tensor parallel sizing:

• tp must be a divisor of the model's number of attention heads
• Use the largest tp that fits on a single node before adding pipeline parallelism
• NVLink between GPUs significantly improves TP throughput

4.3 Multi-Node (Pipeline + Tensor Parallelism)

For models exceeding single-node capacity, combine tensor and pipeline parallelism:

# Requires Ray initialized across nodes first
# Node 0 (head):
ray start --head --port=6379

# Node 1 (worker):
ray start --address=<head_ip>:6379

# Then on head node:
vllm serve meta-llama/Meta-Llama-3-405B-Instruct \
  --tensor-parallel-size 8 \
  --pipeline-parallel-size 2 \
  --distributed-executor-backend ray

Important

Multi-node deployments require Ray to be initialized before launching vLLM. All nodes must have identical model weights at the same path, or share a filesystem mount (NFS, Lustre, etc.) at the same mount point. Inter-node communication performance depends on the network fabric — InfiniBand or NVLink Fabric is strongly recommended for pipeline parallelism.

4.4 Data Parallelism

Data parallelism runs multiple full model replicas and shards requests across them. Best for high-throughput scenarios with models that fit on a single GPU (or TP group):

# 2 replicas, each using 2 GPUs for TP
VLLM_DP_MASTER_IP=0.0.0.0 \
VLLM_DP_SIZE=2 \
vllm serve Qwen/Qwen2.5-7B-Instruct \
  --tensor-parallel-size 2 \
  --data-parallel-size 2

4.5 Kubernetes Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-server
spec:
  replicas: 1
  selector:
    matchLabels:
      app: vllm-server
  template:
    metadata:
      labels:
        app: vllm-server
    spec:
      containers:
      - name: vllm
        image: vllm/vllm-openai:latest
        args:
          - "vllm"
          - "serve"
          - "meta-llama/Meta-Llama-3.1-8B-Instruct"
          - "--tensor-parallel-size"
          - "1"
          - "--gpu-memory-utilization"
          - "0.90"
          - "--enable-prefix-caching"
        env:
        - name: VLLM_API_KEY
          valueFrom:
            secretKeyRef:
              name: vllm-secrets
              key: api-key
        - name: VLLM_NO_USAGE_STATS
          value: "1"
        ports:
        - containerPort: 8000
        resources:
          limits:
            nvidia.com/gpu: 1
        volumeMounts:
        - name: model-cache
          mountPath: /root/.cache/huggingface
        livenessProbe:
          httpGet:
            path: /health
            port: 8000
          initialDelaySeconds: 300
          periodSeconds: 30
        readinessProbe:
          httpGet:
            path: /health
            port: 8000
          initialDelaySeconds: 60
          periodSeconds: 10
      volumes:
      - name: model-cache
        persistentVolumeClaim:
          claimName: model-cache-pvc

4.6 ROCm (AMD GPU) Deployment

docker run \
  --device /dev/kfd \
  --device /dev/dri \
  --security-opt seccomp=unconfined \
  --group-add video \
  -e VLLM_TARGET_DEVICE=rocm \
  -e VLLM_ROCM_USE_AITER_LINEAR=1 \
  -e VLLM_ROCM_USE_AITER_MOE=1 \
  -e HIP_VISIBLE_DEVICES=0,1 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-rocm:latest \
  vllm serve meta-llama/Meta-Llama-3-8B-Instruct \
    --tensor-parallel-size 2

4.7 CPU-Only Deployment

# Build with CPU target
VLLM_TARGET_DEVICE=cpu pip install vllm

# Run
VLLM_CPU_KVCACHE_SPACE=40 \
VLLM_CPU_OMP_THREADS_BIND=auto \
vllm serve Qwen/Qwen2.5-7B-Instruct \
  --device cpu \
  --dtype bfloat16

CPU deployment is significantly slower than GPU. Use only for testing or low-throughput scenarios.

4.8 Systemd Unit File

[Unit]
Description=vLLM Inference Server
After=network.target
Wants=network.target

[Service]
Type=simple
User=vllm
Group=vllm
WorkingDirectory=/opt/vllm
EnvironmentFile=/opt/vllm/.env
ExecStart=/opt/vllm/venv/bin/vllm serve meta-llama/Meta-Llama-3.1-8B-Instruct \
    --tensor-parallel-size 1 \
    --gpu-memory-utilization 0.90 \
    --enable-prefix-caching \
    --port 8000
Restart=on-failure
RestartSec=30
TimeoutStartSec=600
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=multi-user.target

sudo systemctl enable vllm
sudo systemctl start vllm
sudo journalctl -u vllm -f

4.9 Prefill/Decode Disaggregation (P/D)

vLLM supports disaggregating prefill (prompt processing) and decode (token generation) workloads across separate server instances, configured via --kv-transfer-config. This allows independent scaling of each phase.

Caution

P/D disaggregation is experimental. Known issues include OffloadingConnector segfaults in disaggregated mode (issue #39500) and GPU→CPU KV offload crashes with cuMemcpyBatchAsync (issue #39491). Validate thoroughly before production use and monitor closely.

5. Operations

5.1 Health Checks

# Basic health check
curl http://localhost:8000/health

# Model list (confirms model loaded and API is authenticated)
curl http://localhost:8000/v1/models \
  -H "Authorization: Bearer $VLLM_API_KEY"

Warning

The /generate endpoint (vllm/entrypoints/api_server.py) is a legacy demo server — not the production OpenAI-compatible API. For production, always use the OpenAI-compatible server (vllm/entrypoints/openai/api_server.py) which exposes /v1/chat/completions, /v1/completions, /v1/embeddings, and /v1/models. The legacy server has no authentication.

5.2 Model Loading & Warm-Up

vLLM downloads models from HuggingFace Hub on first launch. Subsequent starts use the local cache:

# Pre-download model before serving (avoids startup timeout on first request)
huggingface-cli download meta-llama/Meta-Llama-3.1-8B-Instruct

# Specify an explicit download directory
vllm serve meta-llama/Meta-Llama-3.1-8B-Instruct \
  --download-dir /data/models

Engine initialization (VLLM_ENGINE_READY_TIMEOUT_S, default 600 seconds) covers:

1. Model weight loading from disk
2. KV cache allocation and GPU memory profiling
3. CUDA graph capture at multiple batch sizes
4. Tokenizer loading

Note

CUDA graph capture runs dummy forward passes at multiple batch sizes to pre-compile execution graphs. This is a one-time cost that significantly improves latency for small batches. The --enforce-eager flag disables CUDA graphs entirely and is useful for debugging but reduces throughput. For production, CUDA graphs should remain enabled.

5.3 Log Management

# Set log level
export VLLM_LOGGING_LEVEL=DEBUG

# Custom logging config (Python dictConfig format)
export VLLM_LOGGING_CONFIG_PATH=/opt/vllm/logging.yaml

# Disable color (for log files and aggregators)
export VLLM_LOGGING_COLOR=never

# Prefix all log lines (useful in aggregated log streams)
export VLLM_LOGGING_PREFIX="[vllm-prod-01] "

Stats are logged every VLLM_LOG_STATS_INTERVAL seconds (default 10) including:

• Running/waiting/swapped request counts
• GPU KV cache utilization percentage
• CPU KV cache utilization percentage
• Throughput (tokens/sec)

5.4 Usage Telemetry

By default vLLM reports anonymous usage statistics to https://stats.vllm.ai. Data includes hardware configuration and feature flags — no prompts or model outputs.

# Disable telemetry
export VLLM_NO_USAGE_STATS=1
# or
export DO_NOT_TRACK=1

Note

The telemetry system respects the standard DO_NOT_TRACK environment variable used by many developer tools. Any of VLLM_NO_USAGE_STATS=1, VLLM_DO_NOT_TRACK=1, or DO_NOT_TRACK=1 disables reporting. The stats endpoint can be overridden via VLLM_USAGE_STATS_SERVER for air-gapped environments running an internal stats collector.

5.5 Monitoring with OpenTelemetry

vLLM integrates with distributed tracing via OpenTelemetry (OTLP protocol) through vllm/tracing/:

export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=http://jaeger:4317

vllm serve your-model \
  --enable-opentelemetry \
  --otlp-traces-endpoint http://jaeger:4317

Spans are created for HTTP request handling, engine scheduling, model forward passes, and tokenization/detokenization.

5.6 Updating vLLM

Warning

vLLM updates frequently and sometimes introduces breaking changes to configuration flags, quantization support, or model compatibility. Always test a new version with --enforce-eager first to surface errors quickly before re-enabling CUDA graphs for production.

# Check current version
vllm --version

# Upgrade via pip
pip install --upgrade vllm

# Upgrade via Docker
docker pull vllm/vllm-openai:latest
docker compose up -d --no-deps vllm

5.7 Collecting Diagnostic Information

When filing a bug report or investigating issues:

# Collect complete environment info
vllm collect-env

# Outputs: Python version, CUDA version, PyTorch version,
# GPU model, driver version, vLLM version, OS info

5.8 Backup & Recovery

vLLM is stateless — all persistent state is in:

• Model weights (HuggingFace cache or local path)
• LoRA adapters (if using --enable-lora)
• Media cache (VLLM_MEDIA_CACHE)

Recovery procedure: restore model weights, restart the vLLM process. The KV cache is re-profiled automatically on startup; no database or queue state needs to be restored.

6. Performance

6.1 Throughput vs. Latency Trade-offs

6.2 GPU Memory Utilization

--gpu-memory-utilization (default 0.90) controls what fraction of GPU VRAM is reserved for the KV cache after model weights are loaded:

kv_cache_vram = total_vram × gpu_memory_utilization − model_weights_vram

Raising this value increases the number of concurrent sequences. Values above 0.95 risk OOM errors under bursty load. Values below 0.80 reduce concurrency significantly on smaller GPU models.

6.3 Prefix Caching

Enable with --enable-prefix-caching. The engine maintains a hash-keyed map of KV blocks. When a new request shares a prefix with a cached block chain, those blocks are reused — dramatically reducing prefill cost for chat applications.

vllm serve your-model \
  --enable-prefix-caching \
  --gpu-memory-utilization 0.90

Best practices:

• Stable system prompts benefit most (100% cache hit for system prompt tokens)
• Few-shot prompt templates are excellent candidates
• Randomizing the system prompt per-request defeats caching entirely
• Monitor cache hit rate in stats output: prefix_cache_hit_rate

6.4 Chunked Prefill

--enable-chunked-prefill splits long prefill sequences across multiple scheduler steps, interleaving them with decode steps:

vllm serve your-model \
  --enable-chunked-prefill \
  --max-num-batched-tokens 8192

Benefits:

• Reduces time-to-first-token variance for other requests during long prefills
• Prevents long prompts from starving running decode sequences
• Enables admitting more sequences without OOM risk during burst

6.5 Quantization

Quantization reduces memory footprint and can improve throughput on quantization-accelerated hardware:

# AWQ (recommended for Turing/Ampere/Ada/Hopper)
vllm serve TheBloke/Llama-2-70B-AWQ \
  --quantization awq \
  --dtype half

# GPTQ
vllm serve TheBloke/Llama-2-70B-GPTQ \
  --quantization gptq

# FP8 (Ada/Hopper only)
vllm serve meta-llama/Meta-Llama-3-70B-Instruct \
  --quantization fp8 \
  --kv-cache-dtype fp8_e5m2

# BitsAndBytes (4-bit NF4, broad compatibility)
vllm serve meta-llama/Meta-Llama-3-70B-Instruct \
  --quantization bitsandbytes \
  --load-format bitsandbytes

6.6 Speculative Decoding

Speculative decoding accelerates generation by predicting multiple tokens with a cheap draft model, then verifying with the full model. Net speedup depends on the draft acceptance rate.

# N-gram speculative decoding (no draft model needed, good for repetitive text)
vllm serve your-model \
  --speculative-config '{"method": "ngram", "num_speculative_tokens": 5}'

# Draft model
vllm serve meta-llama/Meta-Llama-3-70B-Instruct \
  --speculative-config '{
    "model": "meta-llama/Meta-Llama-3-8B-Instruct",
    "num_speculative_tokens": 5
  }'

# EAGLE
vllm serve meta-llama/Meta-Llama-3-70B-Instruct \
  --speculative-config '{
    "method": "eagle",
    "model": "your-eagle-draft-model",
    "num_speculative_tokens": 5
  }'

Caution

Known speculative decoding issues as of commit 58da4ee: N-gram produces corrupted output on hybrid GDN models (issue #39273). MTP does not enforce thinking token budgets (issue #39573). Mamba prefix caching combined with MTP crashes on NemotronH models (issue #39809). Test speculative decoding against your specific model before production deployment.

6.7 torch.compile Integration

vllm/compilation/ integrates PyTorch's torch.compile for kernel fusion and ahead-of-time optimization:

# Maximum optimization
vllm serve your-model \
  --compilation-config '{"optimization_level": 3}'

# Shorthand
vllm serve your-model --optimization-level 3

# All optimizations at once
vllm serve your-model --performance-mode

Compilation increases startup time on the first run (compiles and caches) but reduces per-token latency for subsequent runs. The compilation cache persists via PyTorch's inductor cache.

6.8 Benchmarking

# Throughput benchmark
python -m vllm.benchmarks.benchmark_throughput \
  --model meta-llama/Meta-Llama-3.1-8B-Instruct \
  --dataset sharegpt \
  --num-prompts 1000

# Latency benchmark (single-stream)
python -m vllm.benchmarks.benchmark_latency \
  --model meta-llama/Meta-Llama-3.1-8B-Instruct \
  --input-len 512 \
  --output-len 128 \
  --num-iters 50

# Via CLI
vllm bench --model your-model

6.9 Profiling

Use the layerwise profiler (vllm/profiler/) for per-layer timing:

vllm serve your-model \
  --profiler-config '{"enabled": true, "output_dir": "/tmp/vllm_profile"}'

Combined with NVIDIA Nsight Systems:

nsys profile --trace cuda,nvtx \
  vllm serve your-model --enforce-eager

6.10 LoRA Adapter Management

When --enable-lora is set, LoRA adapters can be loaded at runtime without restarting the server. The engine maintains an LRU cache of loaded adapters up to --max-loras.

vllm serve base-model \
  --enable-lora \
  --max-loras 4 \
  --max-lora-rank 64

Request a specific adapter in the API call:

{
  "model": "base-model",
  "lora_request": {
    "lora_name": "my-adapter",
    "lora_int_id": 1,
    "lora_local_path": "/path/to/adapter"
  },
  "messages": [{"role": "user", "content": "Hello"}]
}

7. Security

7.1 Authentication

vLLM's built-in authentication is a single static API key:

export VLLM_API_KEY="your-secret-key"

All requests must include:

Authorization: Bearer your-secret-key

Warning

The built-in API key mechanism is minimal — it is a single shared secret with no per-user rate limiting, no audit logging per key, and no key rotation support. For production deployments with multiple users, place vLLM behind an API gateway (Kong, AWS API Gateway, nginx) that handles authentication, rate limiting, and key management.

7.2 TLS / HTTPS

Enable TLS directly on the vLLM server:

vllm serve your-model \
  --ssl-keyfile /path/to/server.key \
  --ssl-certfile /path/to/server.crt \
  --host 0.0.0.0 \
  --port 443

Alternatively, terminate TLS at a reverse proxy (nginx, Traefik, Caddy) and proxy to vLLM on localhost. This is the recommended approach for production.

7.3 Network Isolation

Caution

Never expose the vLLM HTTP port directly to the public internet without authentication. If VLLM_API_KEY is not set, the API is open to anyone who can reach the port. The legacy demo server (api_server.py) has no authentication at all. Always set VLLM_API_KEY in production and restrict network access at the firewall or load balancer level.

Recommended network architecture:

Internet → Load Balancer / API Gateway (TLS termination, auth, rate limiting)
               │
               ▼
           vLLM Server (private subnet only, not internet-routable)
               │  localhost or private cluster network
               ▼
           GPU Workers (same host or private cluster network)

7.4 Insecure Serialization

Caution

VLLM_ALLOW_INSECURE_SERIALIZATION=1 enables pickle-based object serialization between engine processes. Pickle is inherently unsafe if any untrusted data can reach the serialization path. Never enable this in production systems.

7.5 Remote Code Execution Risk

--trust-remote-code allows arbitrary Python code embedded in a HuggingFace model's files to execute during model loading. This is required for some custom model architectures.

Warning

Only use --trust-remote-code with models from sources you explicitly trust. HuggingFace Hub models can contain arbitrary Python that executes with the full privileges of the vLLM process. Always run vLLM as a non-root user. Consider using a read-only filesystem for the model cache directory and running in a container with restricted capabilities (--cap-drop ALL, --security-opt no-new-privileges:true).

7.6 Media Fetch SSRF Risk

For multimodal models, vLLM fetches media (images, video, audio) from URLs in requests. This creates a Server-Side Request Forgery (SSRF) risk if the server can reach internal network resources.

Mitigations:

• Run vLLM in a network namespace that cannot reach internal services
• Configure VLLM_MEDIA_CONNECTOR to use a proxy that enforces URL allowlists
• Set fetch timeouts conservatively to bound resource exhaustion

7.7 Request/Response Logging

As of commit 58da4ee, complete request/response logging is not built into vLLM (see issue #40155: feature request for complete request/response logging). To log full request/response pairs for audit or compliance purposes, implement logging middleware at the reverse proxy layer.

7.8 Secrets Management

Use a secrets manager (HashiCorp Vault, AWS Secrets Manager, Kubernetes Secrets) to inject these at runtime rather than storing them in container images or shell scripts.

8. Developer Internals

8.1 Codebase Scale

8.2 Build System

vLLM uses a CMake-based build system wrapped by setuptools (setup.py). The build system:

1. Detects target device from VLLM_TARGET_DEVICE or installed hardware
2. Configures CMake for the appropriate kernel set (CUDA, ROCm, XPU, CPU)
3. Compiles C++/CUDA kernels in csrc/
4. Installs the compiled extensions alongside the Python package

# Standard install (auto-detects GPU)
pip install -e .

# Force CUDA target
VLLM_TARGET_DEVICE=cuda pip install -e .

# ROCm target
VLLM_TARGET_DEVICE=rocm pip install -e .

# CPU-only (no GPU kernels)
VLLM_TARGET_DEVICE=cpu pip install -e .

# Accelerate build with sccache
pip install sccache
VLLM_DISABLE_SCCACHE=0 pip install -e .

# Parallel compilation
MAX_JOBS=16 NVCC_THREADS=4 pip install -e .

CUDA Kernel Modules (csrc/)

8.3 Adding a New Model

To add a new model architecture:

1. Create vllm/model_executor/models/your_model.py
2. Implement the model class inheriting from nn.Module with forward() accepting vLLM's input format
3. Register the model in vllm/model_executor/models/__init__.py
4. If the model has a custom attention pattern, implement the attention class in vllm/attention/backends/
5. Add tests in tests/models/

Key interfaces to implement:

• forward(input_ids, positions, kv_caches, attn_metadata) — main forward pass
• load_weights(weights_iterator) — weight loading from HuggingFace format
• sampler property — returns the Sampler instance

8.4 Plugin System

vllm/plugins/ provides a registration system for extending vLLM without modifying core code:

# Register via Python entry_points in setup.py or pyproject.toml
[options.entry_points]
vllm.general_plugins =
    my_plugin = my_package.plugin:register

8.5 Custom Quantization Plugin

Register a custom quantization method using the decorator pattern:

from vllm.model_executor.layers.quantization import register_quantization_config
from vllm.model_executor.layers.quantization.base_config import QuantizationConfig

@register_quantization_config("my_quant")
class MyQuantConfig(QuantizationConfig):
    def get_name(self) -> str:
        return "my_quant"

    def get_supported_act_dtypes(self):
        return [torch.float16, torch.bfloat16]

    def get_min_capability(self) -> int:
        return 75  # Turing+

    def get_config_filenames(self):
        return ["quantize_config.json"]

    @classmethod
    def from_config(cls, config: dict) -> "MyQuantConfig":
        return cls(...)

    def get_quant_method(self, layer, prefix):
        return MyQuantLinearMethod(self)

8.6 Platform Abstraction

vllm/platforms/ provides a unified interface hiding GPU vendor differences:

Platform (abstract)
├── CUDAPlatform        — NVIDIA CUDA
├── ROCmPlatform        — AMD ROCm/HIP
├── XPUPlatform         — Intel XPU
├── CPUPlatform         — x86 CPU
└── EmptyPlatform       — No-op (testing)

Platform-specific code should always use the abstraction layer:

from vllm.platforms import current_platform

if current_platform.is_cuda():
    # CUDA-specific code
elif current_platform.is_rocm():
    # ROCm-specific code

8.7 Reasoning Parsers

vllm/reasoning/ (21 files, 135K chars) provides pluggable parsers for chain-of-thought models:

# DeepSeek-R1
vllm serve deepseek-ai/DeepSeek-R1 --reasoning-parser deepseek_r1

# Qwen3
vllm serve Qwen/Qwen3-72B --reasoning-parser qwen3

The parser extracts reasoning_content separately from the final content in the API response.

8.8 Tool Calling Parsers

vllm/tool_parsers/ (37 files, 611K chars) implements function/tool calling for models that generate tool call JSON in their output. Different model families use different formats:

# Llama 3 JSON tool calling format
vllm serve meta-llama/Meta-Llama-3.1-8B-Instruct \
  --tool-call-parser llama3_json

8.9 Pooling Models (Embeddings & Reranking)

vLLM supports embedding, classification, and reranking via the pooling interface:

Pooling strategies: CLS, LAST, MEAN, ALL, STEP

Score types for reranking: cross-encoder, late-interaction (ColBERT), bi-encoder

8.10 Intermediate Representation

vllm/ir/ (3 files, 17K chars) provides an IR layer used by the compilation pipeline. It enables analysis and transformation of computation graphs before they are compiled by torch.compile.

8.11 Testing

# Run all unit tests
pytest tests/

# Run without slow tests
pytest tests/ -m "not slow"

# Run model-specific tests
pytest tests/models/ -k "llama"

# Integration tests (requires GPU)
pytest tests/entrypoints/ -v

# Kernel unit tests
pytest tests/kernels/ -v

Test directory structure:

• tests/models/ — per-model correctness tests
• tests/entrypoints/ — API server integration tests
• tests/kernels/ — CUDA kernel unit tests
• tests/samplers/ — sampling correctness tests
• tests/lora/ — LoRA adapter tests
• tests/quantization/ — quantization tests

8.12 Code Conventions

• Type hints: All public APIs use Python type hints throughout
• Dataclasses: Configuration uses @dataclass (VllmConfig and all sub-configs)
• Async: The async engine uses asyncio throughout; sync wrappers exist for offline/batch use
• Error messages: 1,550 distinct error messages extracted — errors are specific and actionable
• Platform guards: Platform-specific code uses current_platform.is_*() checks, never bare torch.cuda.is_available()
• Env vars: All environment variables centrally defined in vllm/envs.py with type coercions and defaults

9. Integration Points

9.1 API Endpoints

The OpenAI-compatible server (vllm/entrypoints/openai/api_server.py) exposes:

Additional endpoints are registered dynamically based on enabled features (tokenization, reranking, score endpoints for pooling models).

The legacy demo server (vllm/entrypoints/api_server.py):

9.2 OpenAI Client Compatibility

Any OpenAI Python SDK client can target a vLLM server:

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="your-vllm-api-key",  # matches VLLM_API_KEY
)

response = client.chat.completions.create(
    model="meta-llama/Meta-Llama-3.1-8B-Instruct",
    messages=[{"role": "user", "content": "Hello"}],
    max_tokens=512,
    temperature=0.7,
)
print(response.choices[0].message.content)

9.3 HuggingFace Transformers Integration

vllm/transformers_utils/ (12 files, 120K chars) bridges vLLM with the HuggingFace ecosystem:

• Loads model configs via AutoConfig
• Loads tokenizers via AutoTokenizer
• Applies chat templates via tokenizer.apply_chat_template()
• Handles special tokens, padding, BOS/EOS configuration

Use --hf-overrides to patch config.json fields without modifying the files on disk.

9.4 Ray Integration

vllm/ray/ (2 files, 5K chars) integrates with Ray for multi-node distributed execution:

import ray
ray.init(address="auto")  # Connect to existing cluster

Ray workers are managed by RayWorkerWrapper which handles remote method calls across the Ray actor system. Select with --distributed-executor-backend ray.

9.5 Structured Output Integration

For JSON-schema-constrained generation, vLLM integrates with:

• xgrammar — grammar-based constrained decoding (VLLM_XGRAMMAR_CACHE_MB controls cache)
• guidance — model guidance library

vllm serve your-model \
  --structured-outputs-config '{"backend": "xgrammar"}'

9.6 Observability Integrations

9.7 Offline Batch Processing

For batch inference without running a server:

# CLI batch mode
vllm run-batch \
  --model your-model \
  --input-file prompts.jsonl \
  --output-file outputs.jsonl

# Programmatic Python API
from vllm import LLM, SamplingParams

llm = LLM(model="your-model")
outputs = llm.generate(
    ["Prompt 1", "Prompt 2"],
    SamplingParams(temperature=0.8, max_tokens=256)
)
for output in outputs:
    print(output.outputs[0].text)

10. Troubleshooting

10.1 Engine Fails to Start

Symptom: Process exits during initialization; timeout waiting for engine.

Diagnosis:

# Disable CUDA graphs to isolate the error
vllm serve your-model --enforce-eager 2>&1 | head -100

# Check GPU memory
nvidia-smi --query-gpu=memory.free,memory.total --format=csv

Common causes:

• Insufficient GPU VRAM — lower --gpu-memory-utilization or use quantization
• CUDA version mismatch — verify nvcc --version matches PyTorch CUDA build
• Model too large — use tensor parallelism or quantization
• Missing model files — verify ~/.cache/huggingface/hub/ contains the model

10.2 CUDA Out of Memory (OOM)

Symptom: torch.cuda.OutOfMemoryError during inference.

Solutions:

# Reduce memory reservation
vllm serve your-model --gpu-memory-utilization 0.80

# Reduce max sequence length
vllm serve your-model --max-model-len 4096

# Reduce max concurrent sequences
vllm serve your-model --max-num-seqs 128

# Enable chunked prefill to smooth memory peaks
vllm serve your-model --enable-chunked-prefill

# Use quantization to reduce model footprint
vllm serve your-model --quantization awq

10.3 Slow Throughput

Symptom: Requests per second or tokens per second below expectations.

# Baseline measurement
python -m vllm.benchmarks.benchmark_throughput \
  --model your-model --num-prompts 500

# Watch GPU utilization live
watch -n 1 nvidia-smi

Solutions: Enable prefix caching, increase --max-num-batched-tokens, enable chunked prefill, verify CUDA graphs are active (not --enforce-eager), consider speculative decoding for decode-heavy workloads.

10.4 High Time-to-First-Token (TTFT)

Symptom: First tokens take much longer than subsequent tokens.

Cause: Long prompt processing (prefill) blocks decode of other requests.

# Chunked prefill distributes prefill cost across steps
vllm serve your-model \
  --enable-chunked-prefill \
  --max-num-batched-tokens 2048

10.5 Stuck Engine / Timeout

Symptom: VLLM_ENGINE_ITERATION_TIMEOUT_S exceeded; engine appears hung.

# Increase timeout for slow hardware
export VLLM_ENGINE_ITERATION_TIMEOUT_S=120

# Debug NCCL in multi-GPU setup
export NCCL_DEBUG=INFO
export NCCL_DEBUG_SUBSYS=ALL

Warning

If VLLM_KEEP_ALIVE_ON_ENGINE_DEATH=0 (default), the HTTP server terminates when the engine process dies. Set to 1 to keep the server accepting requests (which will error) while you diagnose the engine failure. In load-balanced deployments, leaving this at 0 means the health check fails and traffic routes away automatically, which is usually the correct behavior.

10.6 Multi-GPU NCCL Errors

Symptom: NCCL error, Timeout waiting for NCCL, deadlock in multi-GPU setup.

# Run P2P connectivity check (disabled by default)
export VLLM_SKIP_P2P_CHECK=0

# Override NCCL library path
export VLLM_NCCL_SO_PATH=/usr/local/lib/libnccl.so.2

# Disable NVLink P2P if causing issues
export NCCL_P2P_DISABLE=1

10.7 Quantization Errors

Symptom: Error about unsupported quantization on current hardware.

Check Section 2.5 hardware compatibility table. Common cases:

• FP8 on Ampere or older: use INT8 or AWQ instead
• Marlin on Volta: use GPTQ or BitsAndBytes
• AWQ on AMD: use GPTQ or GGUF

10.8 Speculative Decoding Issues

Known bugs as of commit 58da4ee:

Caution

Speculative decoding bugs can produce silently incorrect output — tokens that appear syntactically valid but contain semantic errors. Validate speculative decoding output quality with your specific model before production deployment.

10.9 LoRA Not Loading

Symptom: LoRA adapter fails to load or has no effect.

Checklist:

• --enable-lora must be set at server start
• --max-lora-rank must be ≥ the adapter's actual rank
• --max-loras must be increased when loading multiple adapters
• Adapter must be compatible with the base model architecture
• Adapter must be in HuggingFace PEFT format

10.10 Reasoning Parser Not Working

Symptom: reasoning_content is empty or content includes raw thinking tokens.

# Explicitly set the parser
vllm serve your-model --reasoning-parser deepseek_r1
# or
vllm serve your-model --reasoning-parser qwen3

Verify the model actually generates expected thinking tokens before the final answer.

10.11 P/D Disaggregation Crashes

Known issues:

• Issue #39500: OffloadingConnector segfaults in P/D disaggregated mode
• Issue #39491: GPU→CPU KV offload crashes with cuMemcpyBatchAsync

Workaround: Avoid OffloadingConnector in production until these are resolved upstream.

10.12 MoE Kernel Config Fails on GH200

Symptom: Error during MoE kernel configuration on NVIDIA GH200 (issue #37191).

# Disable DeepGEMM MoE path as workaround
export VLLM_MOE_USE_DEEP_GEMM=0

10.13 Batch-Invariant Attention Warning

Context: Issue #40173 tracks automatic selection of the highest-priority batch-invariant attention backend.

# Force a specific attention backend
vllm serve your-model --attention-backend flashinfer

# Or enable batch invariant mode
export VLLM_BATCH_INVARIANT=1

10.14 torch.compile Cache Issues

Symptom: Recompilation on every startup; slow startup after vLLM updates.

# Clear inductor cache
rm -rf ~/.cache/torch/inductor

# Or set an explicit cache directory
export TORCHINDUCTOR_CACHE_DIR=/opt/vllm/torch_cache

10.15 Media Fetch Failures (Multimodal)

Symptom: Image/video/audio URLs fail or time out.

export VLLM_IMAGE_FETCH_TIMEOUT=30
export VLLM_VIDEO_FETCH_TIMEOUT=120
export VLLM_MEDIA_CACHE=/tmp/vllm_media   # Enable caching to avoid re-fetching
export VLLM_MEDIA_FETCH_MAX_RETRIES=5
export VLLM_MEDIA_LOADING_THREAD_COUNT=16

10.16 RPC Timeout Between Engine Processes

Symptom: RPC timeout error; engine appears to hang waiting for workers.

# Increase RPC timeout (default 10000ms)
export VLLM_RPC_TIMEOUT=30000

# Increase model execution timeout (default 300s)
export VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=600

10.17 High Memory Usage at Idle

Explanation: This is expected. vLLM pre-allocates the full KV cache at startup (controlled by --gpu-memory-utilization). Pre-allocation eliminates runtime allocation overhead and provides deterministic memory usage.

To reduce idle footprint at the cost of reduced concurrent capacity:

vllm serve your-model --gpu-memory-utilization 0.70

10.18 Environment Variable Has No Effect

Cause: vLLM reads environment variables at Python module import time.

Solution: Set variables in the shell before running vllm serve, in the systemd EnvironmentFile=, or in the Docker environment: section. Restart the process after any change.

10.19 OpenTelemetry Traces Not Appearing

Checklist:

# Verify collector is reachable
curl http://jaeger:4317

# Set endpoint
export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=http://jaeger:4317

# Enable tracing
vllm serve your-model --enable-opentelemetry

10.20 Complete Request/Response Logging

As of commit 58da4ee, full request/response logging is not built into vLLM (issue #40155 tracks this feature request). Implement logging at the reverse proxy layer for compliance or audit requirements.

Appendix A: Quick Reference

A.1 Common Serve Command Patterns

# Small model, single GPU
vllm serve Qwen/Qwen2.5-7B-Instruct

# Production: 4-GPU tensor parallel, prefix caching, auth
VLLM_API_KEY=secret VLLM_NO_USAGE_STATS=1 \
vllm serve meta-llama/Meta-Llama-3-70B-Instruct \
  --tensor-parallel-size 4 \
  --gpu-memory-utilization 0.90 \
  --enable-prefix-caching \
  --max-model-len 32768 \
  --port 8000

# Quantized 70B on 2×A100 40GB
vllm serve TheBloke/Llama-2-70B-AWQ \
  --quantization awq \
  --tensor-parallel-size 2

# Embedding model
vllm serve BAAI/bge-m3 \
  --task embed \
  --enforce-eager

# Debug mode
VLLM_LOGGING_LEVEL=DEBUG \
vllm serve your-model --enforce-eager

A.2 Key Environment Variables: Quick Reference

# Auth & Network
VLLM_API_KEY=your-key
VLLM_PORT=8000

# Logging
VLLM_LOGGING_LEVEL=INFO
VLLM_NO_USAGE_STATS=1
VLLM_LOG_STATS_INTERVAL=10

# GPU selection
CUDA_VISIBLE_DEVICES=0,1,2,3

# Performance (defaults are production-ready)
VLLM_USE_DEEP_GEMM=1
VLLM_MOE_USE_DEEP_GEMM=1
VLLM_ENABLE_V1_MULTIPROCESSING=1

# Timeouts
VLLM_ENGINE_READY_TIMEOUT_S=600
VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=300
VLLM_RPC_TIMEOUT=10000

A.3 Pre-Launch Checklist

• [ ] VLLM_API_KEY is set (production)
• [ ] VLLM_NO_USAGE_STATS=1 is set (production)
• [ ] CUDA_VISIBLE_DEVICES is correct
• [ ] GPU VRAM ≥ (model size ÷ TP size) + KV cache headroom
• [ ] Model weights are downloaded, or HF_TOKEN is set for gated models
• [ ] --tensor-parallel-size matches GPU count
• [ ] --max-model-len fits within model's max_position_embeddings
• [ ] TLS is configured (or terminated at reverse proxy)
• [ ] Health check responds: curl http://localhost:8000/health
• [ ] Log stream is captured (systemd journal or Docker logs)
• [ ] For multi-GPU: Ray initialized (if using --distributed-executor-backend ray)
• [ ] For speculative decoding: tested against target model (see known bugs above)