# vLLM — Administrator & Developer Guide

This guide covers architecture, configuration, deployment, operations, performance tuning, security, and developer internals for vLLM — the high-throughput, memory-efficient inference and serving engine for large language models. For installation, everyday usage, and API client examples, see the companion **User Guide**.

---

## 1. Architecture Overview

vLLM is a production-grade LLM inference engine originally developed at UC Berkeley's Sky Computing Lab, now maintained by a community of 2,000+ contributors. It is designed around three core innovations: PagedAttention for memory management, continuous batching for throughput, and a pluggable backend system that abstracts over CUDA, ROCm, XPU, and CPU execution.

### 1.1 System Topology

```
                         ┌─────────────────────────────────────────────────┐
                         │                  Client Layer                    │
                         │  (OpenAI SDK / curl / any HTTP client)           │
                         └──────────────────────┬──────────────────────────┘
                                                │ HTTP / HTTPS
                         ┌──────────────────────▼──────────────────────────┐
                         │          OpenAI-Compatible API Server            │
                         │   vllm/entrypoints/openai/api_server.py          │
                         │   FastAPI + Uvicorn + uvloop                      │
                         │   Routes: /v1/chat/completions, /v1/completions,  │
                         │           /v1/embeddings, /v1/models, /health     │
                         └──────────────────────┬──────────────────────────┘
                                                │
                         ┌──────────────────────▼──────────────────────────┐
                         │               Engine Layer (V1)                  │
                         │   vllm/v1/  — AsyncLLMEngine / LLMEngine         │
                         │   EngineArgs / AsyncEngineArgs                   │
                         │   VllmConfig (aggregate config dataclass)        │
                         └────────┬──────────────────────────┬─────────────┘
                                  │                          │
              ┌───────────────────▼──────────┐  ┌───────────▼──────────────┐
              │       Scheduler (V1)          │  │   KV Cache Manager (V1)  │
              │  vllm/v1/core/scheduler.py    │  │   vllm/v1/core/          │
              │  Continuous batching          │  │   kv_cache_manager.py    │
              │  Chunked prefill              │  │   PagedAttention blocks  │
              │  Prefix caching              │  │   Block allocator        │
              └───────────────────┬──────────┘  └──────────────────────────┘
                                  │
              ┌───────────────────▼──────────────────────────────────────┐
              │                    Worker Layer                           │
              │  vllm/v1/worker/  — GPUModelRunner, CPUModelRunner       │
              │  Model execution, KV cache I/O, sampling                 │
              └────────────┬────────────────────────────┬────────────────┘
                           │                            │
         ┌─────────────────▼──────────┐  ┌─────────────▼─────────────────┐
         │     Model Executor          │  │    Distributed Backend        │
         │  vllm/model_executor/       │  │  vllm/distributed/            │
         │  200+ model implementations │  │  Tensor / Pipeline / DP       │
         │  Quantization layers        │  │  NCCL / Ray / gloo            │
         └─────────────────────────────┘  └───────────────────────────────┘
```

> [!NOTE]
> vLLM maintains two engine generations simultaneously. The V1 engine (`vllm/v1/`) is the current production path with improved scheduler, KV cache manager, and multiprocessing architecture. The legacy engine path remains for compatibility but V1 is the active development target. `VLLM_ENABLE_V1_MULTIPROCESSING=1` is enabled by default.

### 1.2 Request Lifecycle

```
HTTP Request
    │
    ▼
API Server (FastAPI)
    │  Validate, parse, authenticate (VLLM_API_KEY)
    ▼
AsyncLLMEngine.generate()
    │  Tokenize, build SamplingParams, enqueue
    ▼
Scheduler
    │  Continuous batching — running + waiting + swapped queues
    │  Prefix cache lookup (if enabled)
    │  Chunked prefill (if enabled)
    ▼
GPUModelRunner.execute_model()
    │  Forward pass through model layers
    │  Custom C++/CUDA kernels (attention, MoE, quantization)
    ▼
Sampler
    │  Top-p, top-k, temperature, penalties, logits processing
    ▼
Detokenizer
    │  Stream tokens or return full sequence
    ▼
HTTP Response (streaming SSE or JSON)
```

### 1.3 Component Map

| Layer | Module Path | Role |
|---|---|---|
| Config | `vllm/config/` (29 files, 455K chars) | `VllmConfig` aggregate + all sub-configs |
| Engine Args | `vllm/engine/` (4 files, 112K chars) | CLI arg parsing, `EngineArgs`, `AsyncEngineArgs` |
| Entrypoints | `vllm/entrypoints/` (9 files, 175K chars) | Server startup, chat utils, launcher |
| CLI | `vllm/entrypoints/cli/` (8 files, 31K chars) | `serve`, `chat`, `complete`, `bench`, `collect-env`, `launch`, `run-batch` |
| V1 Engine Core | `vllm/v1/core/` (7 files, 200K chars) | Scheduler, KV cache manager |
| V1 Workers | `vllm/v1/worker/` (23 files, 49K chars) | GPU/CPU model runners |
| V1 Sampling | `vllm/v1/sample/` (3 files, 50K chars) | Sampler, logits processing |
| Model Executor | `vllm/model_executor/` (4 files, 41K chars) | Execution layer, custom ops |
| Models | `vllm/model_executor/models/` (273 files, 6.3M chars) | 200+ model implementations |
| Quantization | `vllm/model_executor/layers/quantization/` | AWQ, GPTQ, FP8, INT8, Marlin, GGUF, BnB |
| Distributed | `vllm/distributed/` (6 files, 133K chars) | NCCL, Ray, tensor/pipeline/DP parallelism |
| LoRA | `vllm/lora/` (8 files, 92K chars) | LoRA adapter hot-swap |
| Multimodal | `vllm/multimodal/` (12 files, 175K chars) | Image, video, audio processing |
| Compilation | `vllm/compilation/` (12 files, 194K chars) | `torch.compile` integration |
| Reasoning | `vllm/reasoning/` (21 files, 135K chars) | Chain-of-thought parsers |
| Tool Parsers | `vllm/tool_parsers/` (37 files, 611K chars) | Function calling parsers |
| Renderers | `vllm/renderers/` (10 files, 93K chars) | Chat template rendering |
| Tokenizers | `vllm/tokenizers/` (11 files, 99K chars) | Tokenizer implementations |
| Platforms | `vllm/platforms/` (8 files, 143K chars) | CUDA, ROCm, XPU, CPU abstraction |
| Transformers Utils | `vllm/transformers_utils/` (12 files, 120K chars) | HuggingFace integration |
| Utils | `vllm/utils/` (35 files, 255K chars) | Shared utilities |
| Profiler | `vllm/profiler/` (3 files, 31K chars) | Layerwise profiling |
| Tracing | `vllm/tracing/` (3 files, 16K chars) | OpenTelemetry integration |
| IR | `vllm/ir/` (3 files, 17K chars) | Intermediate representation |
| Kernels (C++) | `csrc/` | CUDA/C++ kernels: attention, MoE, quantization, Mamba |
| Benchmarks | `vllm/benchmarks/` (6 files, 153K chars) | Throughput and latency benchmarks |
| Ray | `vllm/ray/` (2 files, 5K chars) | Ray cluster integration |
| Plugins | `vllm/plugins/` (1 file, 3K chars) | Plugin registration system |
| Helion Kernels | `vllm/kernels/` (5 files, 7K chars) | Helion-based kernel generation |

### 1.4 V1 Engine Architecture

The V1 engine (`vllm/v1/`) represents the current generation design with a clean multiprocessing boundary:

```
 Main Process (API Server + Engine Orchestrator)
 │
 │  ZMQ / RPC (VLLM_RPC_BASE_PATH)
 │
 ▼
 Engine Core Process
 │   Scheduler — manages sequence groups, block allocation
 │   KV Cache Manager — PagedAttention block table
 │   Prefix Cache — hash-keyed block reuse
 │
 │  NCCL / shared memory
 │
 ▼
 Worker Processes (one per GPU in TP group)
     GPUModelRunner — forward passes
     CUDAGraph capture and replay
     Custom CUDA kernels
```

### 1.5 Memory Management: PagedAttention

PagedAttention divides the KV cache into fixed-size blocks (configured via `--block-size`, default 16 tokens). Blocks are allocated on demand from a global pool, allowing:

- Non-contiguous physical memory for logically contiguous sequences
- Cross-request KV block sharing when prefix caching is enabled
- Swap to CPU when GPU memory is exhausted
- Efficient beam search without memory duplication

```
Logical KV Cache (sequence view):
  [tok0][tok1][tok2][tok3] | [tok4][tok5][tok6][tok7] | [tok8]...
        Block 0 (full)            Block 1 (full)         Block 2 (partial)

Physical GPU Memory:
  Block pool: [B0][B3][B7][B1][B5]...  (any order, managed by allocator)
  Block table: maps logical → physical
```

### 1.6 Continuous Batching

Unlike static batching (one batch per forward pass), continuous batching allows new requests to join and completed requests to leave mid-batch. The scheduler maintains three queues:

- **Running:** sequences currently executing (KV cache allocated)
- **Waiting:** sequences queued, not yet admitted
- **Swapped:** sequences whose KV blocks were evicted to CPU

At each scheduler step, the scheduler determines how many waiting sequences can be admitted (limited by free GPU blocks), optionally preempts low-priority running sequences, and constructs the next model input batch.

### 1.7 Speculative Decoding

vLLM supports multiple speculative decoding methods to accelerate autoregressive generation:

| Method | Description |
|---|---|
| EAGLE | Tree-based speculative decoding with trained draft head |
| MTP (Multi-Token Prediction) | Predict multiple tokens per forward pass |
| Draft Model | Separate smaller model generates draft tokens |
| PARD | Parallel auto-regressive decoding |
| MLP Speculator | MLP-based draft head |
| N-Gram | Reuse recent n-grams as draft tokens |
| Suffix Decoding | Match suffixes from a corpus |

Configured via `--speculative-config` / `-sc`. See Section 3 for parameter details.

### 1.8 Distributed Execution Strategies

| Strategy | Flag | Use Case |
|---|---|---|
| Tensor Parallelism (TP) | `--tensor-parallel-size` / `-tp` | Split model layers across GPUs on one node |
| Pipeline Parallelism (PP) | `--pipeline-parallel-size` / `-pp` | Split model by layers across nodes |
| Data Parallelism (DP) | `--data-parallel-size` / `-dp` | Replicate full model, shard request batch |
| Expert Parallelism (EP) | `--enable-expert-parallel` / `-ep` | MoE expert sharding (combined with TP/DP) |

The `--distributed-executor-backend` flag selects the communication backend: `ray` (multi-node), `mp` (multi-process on single node, default), or `uni` (single process).

---

## 2. System Requirements

### 2.1 Hardware Requirements

| Component | Minimum | Recommended |
|---|---|---|
| GPU (CUDA) | Turing (compute 7.5+) | Ampere or newer (compute 8.0+) |
| GPU VRAM | Depends on model size | 24 GB+ per GPU for 7B models |
| System RAM | 32 GB | 128 GB+ for large models |
| CPU | x86-64, 8+ cores | 32+ cores for multi-GPU |
| Storage | SSD, model size × 1.2 | NVMe for fast model loading |
| Network (multi-node) | 10 GbE | InfiniBand / NVLink for TP |

#### GPU Platform Support Matrix

| Platform | Compute Capability | Status |
|---|---|---|
| NVIDIA Volta (V100) | 7.0 | Supported (limited quantization support) |
| NVIDIA Turing (T4, RTX 2080) | 7.5 | Supported |
| NVIDIA Ampere (A100, A10, RTX 3090) | 8.0 / 8.6 | Fully supported |
| NVIDIA Ada (L4, L40, RTX 4090) | 8.9 | Fully supported, FP8 supported |
| NVIDIA Hopper (H100, H200, GH200) | 9.0 | Fully supported, FP8 + FlashInfer |
| AMD MI200/MI300 (ROCm) | — | Supported via ROCm backend |
| Intel GPU (Arc, Flex, Max) | — | Supported via XPU backend |
| x86 CPU | — | Supported (reduced performance) |

### 2.2 Software Requirements

| Component | Version |
|---|---|
| Python | 3.12+ |
| CUDA Toolkit | 12.x (Docker default: 13.0.0) |
| PyTorch | Compatible with CUDA version |
| Ubuntu (Docker) | 22.04 |
| ROCm (AMD) | 6.x+ |

### 2.3 Docker Image Versions

The official Docker build pins (from `docker/versions.json`):

```json
{
  "cuda": "13.0.0",
  "python": "3.12",
  "ubuntu": "22.04"
}
```

### 2.4 Network Ports

| Port | Purpose | Configurable |
|---|---|---|
| 8000 | Default HTTP API server | `--port` / `VLLM_PORT` |
| Dynamic | RPC between engine processes | `VLLM_RPC_BASE_PATH` |
| Dynamic | Ray cluster ports | Ray configuration |

### 2.5 Quantization Hardware Compatibility

| Implementation | Volta | Turing | Ampere | Ada | Hopper | AMD GPU | Intel GPU | x86 CPU |
|---|---|---|---|---|---|---|---|---|
| AWQ | No | Yes | Yes | Yes | Yes | No | Yes | Yes |
| GPTQ | Yes | Yes | Yes | Yes | Yes | No | Yes | Yes |
| Marlin | No | Yes* | Yes | Yes | Yes | No | No | No |
| INT8 (W8A8) | No | Yes | Yes | Yes | Yes | No | No | Yes |
| FP8 (W8A8) | No | No | No | Yes | Yes | Yes | No | No |
| BitsAndBytes | Yes | Yes | Yes | Yes | Yes | No | No | No |
| GGUF | Yes | Yes | Yes | Yes | Yes | Yes | No | No |

*Turing does not support Marlin MXFP4.

> [!NOTE]
> FP8 quantization requires Ada Lovelace (compute 8.9) or Hopper (compute 9.0) architecture, or AMD MI300 series. Attempting to use FP8 on older architectures will raise an error at startup. Check the table above before selecting a quantization format for a given deployment target.

---

## 3. Configuration Reference

vLLM configuration is split into two layers: **CLI flags** passed to `vllm serve` (which map to `EngineArgs` fields) and **environment variables** that control runtime and build behavior. All CLI flags are also accessible programmatically via `EngineArgs` / `AsyncEngineArgs`.

### 3.1 VllmConfig Dataclass Hierarchy

The top-level `VllmConfig` (in `vllm/config/`) aggregates all sub-configs:

```
VllmConfig
├── ModelConfig          — model path, dtype, max_model_len, trust_remote_code
├── CacheConfig          — block_size, gpu_memory_utilization, kv_cache_dtype
├── ParallelConfig       — tp_size, pp_size, dp_size, distributed_executor_backend
├── SchedulerConfig      — max_num_batched_tokens, max_num_seqs, scheduling_policy
├── SpeculativeConfig    — draft model, num_speculative_tokens, method
├── LoRAConfig           — max_loras, max_lora_rank, lora_extra_vocab_size
├── CompilationConfig    — optimization_level, custom_ops, splitting_ops
├── AttentionConfig      — attention_backend
├── KernelConfig         — kernel selection
├── ObservabilityConfig  — tracing, metrics, logging
└── ... (additional sub-configs per feature area)
```

### 3.2 CLI Serve Flags Reference

Launch the server with:

```bash
vllm serve <model> [OPTIONS]
```

> [!IMPORTANT]
> All flags below are sourced from `vllm/engine/arg_utils.py`. Flag names use hyphens on the CLI (`--max-model-len`) but map to underscored Python attributes (`max_model_len`). vLLM validates arguments at startup and exits with an error for unrecognized options. Do not pass flags not documented here.

#### Model & Tokenizer

| Flag | Type | Default | Description |
|---|---|---|---|
| `--model` | str | (required) | HuggingFace model ID or local path |
| `--tokenizer` | str | None | Tokenizer path/ID (defaults to model) |
| `--tokenizer-mode` | str | `auto` | `auto`, `slow`, `mistral` |
| `--trust-remote-code` | flag | False | Allow execution of remote model code |
| `--dtype` | str | `auto` | `auto`, `float16`, `bfloat16`, `float32` |
| `--seed` | int | 0 | Random seed for reproducibility |
| `--max-model-len` | int | None | Max sequence length (tokens); overrides model config |
| `--config-format` | str | `auto` | `auto`, `hf`, `mistral` |
| `--hf-token` | str | None | HuggingFace Hub token for gated models |
| `--hf-overrides` | JSON | None | Override HuggingFace model config fields as JSON dict |
| `--load-format` | str | `auto` | `auto`, `pt`, `safetensors`, `npcache`, `dummy`, `tensorizer`, `bitsandbytes`, `gguf`, `runai_streamer`, `fastsafetensors` |
| `--download-dir` | str | None | Directory for downloading models |
| `--served-model-name` | str | None | Override model name reported by `/v1/models` |

#### Memory & KV Cache

| Flag | Type | Default | Description |
|---|---|---|---|
| `--gpu-memory-utilization` | float | 0.90 | Fraction of GPU VRAM to use for KV cache |
| `--block-size` | int | 16 | KV cache block size in tokens |
| `--kv-cache-dtype` | str | `auto` | `auto`, `fp8`, `fp8_e5m2`, `fp8_e4m3` |
| `--enable-prefix-caching` | flag | False | Reuse KV blocks for matching prefixes |
| `--max-num-batched-tokens` | int | None | Max tokens per scheduler step |
| `--max-num-seqs` | int | 256 | Max concurrent sequences |

#### Parallelism

| Flag | Type | Default | Description |
|---|---|---|---|
| `--tensor-parallel-size` / `-tp` | int | 1 | Number of GPUs for tensor parallelism |
| `--pipeline-parallel-size` / `-pp` | int | 1 | Number of pipeline stages |
| `--data-parallel-size` / `-dp` | int | 1 | Number of data parallel replicas |
| `--enable-expert-parallel` / `-ep` | flag | False | Enable MoE expert parallelism |
| `--distributed-executor-backend` | str | `mp` | `mp` (multiprocess), `ray`, `uni` |

#### Scheduling

| Flag | Type | Default | Description |
|---|---|---|---|
| `--enable-chunked-prefill` | flag | False | Chunk long prefills across scheduler steps |
| `--scheduling-policy` | str | `fcfs` | `fcfs` (first-come-first-served), `priority` |
| `--max-logprobs` | int | 20 | Max log probabilities to return |

#### Quantization

| Flag | Type | Default | Description |
|---|---|---|---|
| `--quantization` / `-q` | str | None | `awq`, `gptq`, `fp8`, `bitsandbytes`, `gguf`, `marlin`, `squeezellm`, `compressed-tensors`, and others |

#### Speculative Decoding

| Flag | Type | Default | Description |
|---|---|---|---|
| `--speculative-config` / `-sc` | JSON | None | Speculative decoding config dict (method, draft model, num tokens, etc.) |

#### LoRA

| Flag | Type | Default | Description |
|---|---|---|---|
| `--enable-lora` | flag | False | Enable LoRA adapter support |
| `--max-loras` | int | 1 | Max number of LoRA adapters loaded simultaneously |
| `--max-lora-rank` | int | 16 | Maximum LoRA rank |

#### Attention & Compilation

| Flag | Type | Default | Description |
|---|---|---|---|
| `--attention-backend` | str | `auto` | `flash_attn`, `flashinfer`, `xformers`, `torch_sdpa`, `triton_attn`, `rocm_flash_attn`, `ipex`, `no_attention` |
| `--enforce-eager` | flag | False | Disable CUDA graph capture (slower but easier to debug) |
| `--compilation-config` / `-cc` | JSON | None | `torch.compile` configuration dict |
| `--optimization-level` | int | None | Compilation optimization level (0–3) |
| `--performance-mode` | flag | False | Enable all performance optimizations |

#### Reasoning & Tools

| Flag | Type | Default | Description |
|---|---|---|---|
| `--reasoning-parser` | str | None | Reasoning parser name (e.g., `deepseek_r1`, `qwen3`) |
| `--structured-outputs-config` | JSON | None | xgrammar / guidance structured output config |

#### KV Transfer (P/D Disaggregation)

| Flag | Type | Default | Description |
|---|---|---|---|
| `--kv-transfer-config` | JSON | None | Prefill/Decode disaggregation config |

#### Server

| Flag | Type | Default | Description |
|---|---|---|---|
| `--host` | str | `0.0.0.0` | Server bind host |
| `--port` | int | 8000 | Server bind port |
| `--ssl-keyfile` | str | None | Path to SSL private key |
| `--ssl-certfile` | str | None | Path to SSL certificate |
| `--disable-log-stats` | flag | False | Suppress periodic stats logging |

#### Profiling & Misc

| Flag | Type | Default | Description |
|---|---|---|---|
| `--profiler-config` | JSON | None | Layerwise profiler configuration |
| `--enable-sleep-mode` | flag | False | Allow engine to sleep when idle |

### 3.3 Environment Variables Reference

All environment variables are processed at import time in `vllm/envs.py`. Variables may be set in the shell, in a `.env` file, or in a container environment.

> [!IMPORTANT]
> Environment variables are read once at Python module import. Changes after the vLLM process has started have no effect. For production deployments, set all environment variables in the systemd unit file, Docker Compose environment section, or Kubernetes ConfigMap/Secret before the process launches.

#### Build & Compilation

| Variable | Default | Description |
|---|---|---|
| `VLLM_TARGET_DEVICE` | `cuda` | Build target: `cuda`, `rocm`, `cpu`, `xpu`, `empty` |
| `MAX_JOBS` | CPU count | Parallel compilation jobs during `pip install` |
| `NVCC_THREADS` | 1 | NVCC threads per compilation unit |
| `CMAKE_BUILD_TYPE` | — | CMake build type override |
| `VLLM_DISABLE_SCCACHE` | `0` | Set to `1` to disable sccache caching |
| `VLLM_USE_PRECOMPILED` | — | Use precompiled binary wheels |
| `VLLM_PRECOMPILED_WHEEL_LOCATION` | — | Path or URL to precompiled wheel |
| `VLLM_MAIN_CUDA_VERSION` | — | Override CUDA version detection during build |

#### Runtime Core

| Variable | Default | Description |
|---|---|---|
| `VLLM_API_KEY` | None | Bearer token required for all API requests (if set) |
| `VLLM_HOST_IP` | — | Advertised host IP for RPC/distributed setup |
| `VLLM_PORT` | `0` | Default port (0 = use `--port` flag, default 8000) |
| `VLLM_RPC_BASE_PATH` | — | Base path for ZMQ RPC sockets |
| `VLLM_ENGINE_ITERATION_TIMEOUT_S` | `60` | Seconds before a single engine iteration is considered stuck |
| `VLLM_ENGINE_READY_TIMEOUT_S` | `600` | Seconds to wait for engine initialization before failing |
| `VLLM_HTTP_TIMEOUT_KEEP_ALIVE` | `5` | HTTP keep-alive timeout in seconds |
| `VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS` | `300` | Timeout for a single model forward pass |
| `VLLM_RPC_TIMEOUT` | `10000` | RPC call timeout in milliseconds |
| `VLLM_BATCH_INVARIANT` | `0` | Enable batch-invariant attention backend selection |
| `VLLM_MAX_N_SEQUENCES` | `16384` | Maximum number of sequences tracked globally |

#### Logging & Observability

| Variable | Default | Description |
|---|---|---|
| `VLLM_CONFIGURE_LOGGING` | `1` | Set to `0` to skip vLLM logging configuration |
| `VLLM_LOGGING_LEVEL` | `INFO` | Python log level: `DEBUG`, `INFO`, `WARNING`, `ERROR` |
| `VLLM_LOGGING_CONFIG_PATH` | — | Path to a custom Python logging config file |
| `VLLM_LOGGING_STREAM` | `ext://sys.stdout` | Log output stream |
| `VLLM_LOGGING_PREFIX` | — | String prefix prepended to all log messages |
| `VLLM_LOGGING_COLOR` | `auto` | `auto`, `always`, `never` — ANSI color in logs |
| `VLLM_NO_USAGE_STATS` | `0` | Set to `1` to disable anonymous usage telemetry |
| `VLLM_DO_NOT_TRACK` | — | Alias for disabling tracking (also respects `DO_NOT_TRACK`) |
| `DO_NOT_TRACK` | — | Standard do-not-track env var (respected by vLLM) |
| `VLLM_USAGE_STATS_SERVER` | `https://stats.vllm.ai` | Endpoint for anonymous usage stats |
| `VLLM_TRACE_FUNCTION` | `0` | Set to `1` to enable function-level tracing (very verbose) |
| `VLLM_LOG_BATCHSIZE_INTERVAL` | `-1` | Log batch size every N seconds (-1 = disabled) |
| `VLLM_LOG_STATS_INTERVAL` | `10` | Seconds between periodic stats log lines |
| `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` | — | OpenTelemetry OTLP collector endpoint URL |
| `VLLM_DISABLE_LOG_LOGO` | `0` | Set to `1` to suppress the startup ASCII logo |

#### GPU & Memory

| Variable | Default | Description |
|---|---|---|
| `CUDA_VISIBLE_DEVICES` | — | Restrict visible NVIDIA GPUs (e.g., `0,1,2,3`) |
| `HIP_VISIBLE_DEVICES` | — | Restrict visible AMD GPUs (ROCm) |
| `CUDA_HOME` | — | Path to CUDA toolkit installation |
| `VLLM_NCCL_SO_PATH` | — | Override path to `libnccl.so` |
| `VLLM_CUDART_SO_PATH` | — | Override path to `libcudart.so` |
| `LOCAL_RANK` | `0` | GPU rank within a node (set by distributed launcher) |
| `VLLM_SKIP_P2P_CHECK` | `1` | Skip GPU P2P bandwidth check at startup |
| `VLLM_CPU_KVCACHE_SPACE` | `0` | GiB of CPU RAM to allocate for KV swap (CPU backend) |
| `VLLM_CPU_OMP_THREADS_BIND` | `auto` | OMP thread-to-CPU binding (CPU backend) |
| `VLLM_CPU_NUM_OF_RESERVED_CPU` | `0` | CPUs to reserve (not used by inference) on CPU backend |

#### Media (Multimodal)

| Variable | Default | Description |
|---|---|---|
| `VLLM_MEDIA_CACHE` | — | Directory for caching downloaded media files |
| `VLLM_MEDIA_CACHE_MAX_SIZE_MB` | `5120` | Max media cache size in megabytes |
| `VLLM_MEDIA_CACHE_TTL_HOURS` | `24` | Hours before cached media files expire |
| `VLLM_IMAGE_FETCH_TIMEOUT` | `5` | Seconds to wait when fetching images |
| `VLLM_VIDEO_FETCH_TIMEOUT` | `30` | Seconds to wait when fetching video |
| `VLLM_AUDIO_FETCH_TIMEOUT` | `10` | Seconds to wait when fetching audio |
| `VLLM_MEDIA_FETCH_MAX_RETRIES` | `3` | Retries for failed media downloads |
| `VLLM_MEDIA_LOADING_THREAD_COUNT` | `8` | Parallel threads for media preprocessing |
| `VLLM_VIDEO_LOADER_BACKEND` | `opencv` | Video decoding library: `opencv`, `decord`, etc. |
| `VLLM_MEDIA_CONNECTOR` | `http` | Media fetching connector type |

#### Distributed

| Variable | Default | Description |
|---|---|---|
| `VLLM_DP_MASTER_IP` | `127.0.0.1` | Data parallel master node IP |
| `VLLM_DP_MASTER_PORT` | `0` | Data parallel rendezvous port (0 = auto) |
| `VLLM_DP_SIZE` | `1` | Total number of data parallel replicas |
| `VLLM_DP_RANK` | `0` | This node's data parallel rank |
| `VLLM_RAY_PER_WORKER_GPUS` | `1.0` | GPU fraction allocated per Ray worker |
| `VLLM_WORKER_MULTIPROC_METHOD` | — | Python multiprocessing start method (`fork`, `spawn`, `forkserver`) |
| `VLLM_ENABLE_V1_MULTIPROCESSING` | `1` | Enable V1 engine multiprocessing architecture |

#### Feature Toggles

| Variable | Default | Description |
|---|---|---|
| `VLLM_USE_DEEP_GEMM` | `1` | Use DeepGEMM for optimized GEMM operations |
| `VLLM_MOE_USE_DEEP_GEMM` | `1` | Use DeepGEMM for MoE expert computation |
| `VLLM_MLA_DISABLE` | `0` | Set to `1` to disable Multi-head Latent Attention optimization |
| `VLLM_USE_TRITON_AWQ` | `0` | Use Triton kernels for AWQ instead of CUDA |
| `VLLM_DISABLE_PYNCCL` | `False` | Disable Python NCCL wrapper (use libnccl directly) |
| `VLLM_KEEP_ALIVE_ON_ENGINE_DEATH` | `0` | Keep HTTP server running if engine process dies |
| `VLLM_ALLOW_INSECURE_SERIALIZATION` | `0` | Allow pickle-based serialization (security risk) |
| `VLLM_SERVER_DEV_MODE` | `0` | Enable developer mode (additional debug endpoints) |
| `VLLM_XGRAMMAR_CACHE_MB` | `512` | xgrammar grammar cache size in MB |
| `VLLM_FLASHINFER_WORKSPACE_BUFFER_SIZE` | — | FlashInfer GPU workspace buffer size |

#### ROCm-Specific

| Variable | Default | Description |
|---|---|---|
| `VLLM_ROCM_USE_AITER` | `False` | Use AITER kernels on AMD GPUs |
| `VLLM_ROCM_USE_AITER_PAGED_ATTN` | `False` | Use AITER for paged attention |
| `VLLM_ROCM_USE_AITER_LINEAR` | `True` | Use AITER for linear (GEMM) operations |
| `VLLM_ROCM_USE_AITER_MOE` | `True` | Use AITER for MoE expert computation |
| `VLLM_ROCM_FP8_PADDING` | `1` | Enable FP8 tensor padding for AMD MI300 |
| `VLLM_ROCM_MOE_PADDING` | `1` | Enable MoE tensor padding on ROCm |

Additional `VLLM_ROCM_*` and `VLLM_USE_FLASHINFER_MOE_*` flags exist for fine-grained AMD kernel control; consult `vllm/platforms/` source for the complete list.

### 3.4 Configuration via Shell Wrapper

vLLM itself does not have a standalone YAML config file — all configuration passes through CLI flags or environment variables. For production, use a shell wrapper script to group all settings:

```bash
#!/usr/bin/env bash
# /opt/vllm/start.sh

export VLLM_API_KEY="your-secret-key"
export CUDA_VISIBLE_DEVICES="0,1,2,3"
export VLLM_NO_USAGE_STATS=1
export VLLM_LOGGING_LEVEL=INFO
export VLLM_LOG_STATS_INTERVAL=30

exec vllm serve meta-llama/Meta-Llama-3-70B-Instruct \
  --tensor-parallel-size 4 \
  --gpu-memory-utilization 0.90 \
  --max-model-len 32768 \
  --enable-prefix-caching \
  --port 8000 \
  --host 0.0.0.0
```

The `--hf-overrides` flag patches specific HuggingFace config fields at runtime:

```bash
vllm serve meta-llama/Meta-Llama-3-70B-Instruct \
  --hf-overrides '{"max_position_embeddings": 131072}' \
  --config-format hf
```

---

## 4. Deployment & Scaling

### 4.1 Docker Deployment

The official Docker images are published to Docker Hub and GHCR. The multi-stage `Dockerfile` lives in `docker/`:

```bash
# Pull pre-built image
docker pull vllm/vllm-openai:latest

# Build from source
docker build \
  --build-arg CUDA_VERSION=12.6.0 \
  --build-arg PYTHON_VERSION=3.12 \
  -f docker/Dockerfile \
  -t vllm-custom:latest .
```

#### Docker Compose (Single Node)

```yaml
services:
  vllm:
    image: vllm/vllm-openai:latest
    runtime: nvidia
    environment:
      - NVIDIA_VISIBLE_DEVICES=all
      - VLLM_API_KEY=${VLLM_API_KEY}
      - VLLM_NO_USAGE_STATS=1
      - VLLM_LOGGING_LEVEL=INFO
    volumes:
      - ~/.cache/huggingface:/root/.cache/huggingface
      - /tmp/vllm_media_cache:/tmp/vllm_media_cache
    ports:
      - "8000:8000"
    command: >
      vllm serve meta-llama/Meta-Llama-3.1-8B-Instruct
        --tensor-parallel-size 1
        --gpu-memory-utilization 0.90
        --enable-prefix-caching
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
      interval: 30s
      timeout: 10s
      retries: 5
      start_period: 300s
```

### 4.2 Multi-GPU Single-Node (Tensor Parallelism)

Tensor parallelism splits each transformer layer across multiple GPUs. This is the primary scaling method for models that do not fit on a single GPU.

```bash
# 4-GPU tensor parallel
vllm serve meta-llama/Meta-Llama-3-70B-Instruct \
  --tensor-parallel-size 4 \
  --gpu-memory-utilization 0.90 \
  --max-model-len 32768

# Check GPU allocation
nvidia-smi
```

Rules for tensor parallel sizing:
- `tp` must be a divisor of the model's number of attention heads
- Use the largest `tp` that fits on a single node before adding pipeline parallelism
- NVLink between GPUs significantly improves TP throughput

### 4.3 Multi-Node (Pipeline + Tensor Parallelism)

For models exceeding single-node capacity, combine tensor and pipeline parallelism:

```bash
# Requires Ray initialized across nodes first
# Node 0 (head):
ray start --head --port=6379

# Node 1 (worker):
ray start --address=<head_ip>:6379

# Then on head node:
vllm serve meta-llama/Meta-Llama-3-405B-Instruct \
  --tensor-parallel-size 8 \
  --pipeline-parallel-size 2 \
  --distributed-executor-backend ray
```

> [!IMPORTANT]
> Multi-node deployments require Ray to be initialized before launching vLLM. All nodes must have identical model weights at the same path, or share a filesystem mount (NFS, Lustre, etc.) at the same mount point. Inter-node communication performance depends on the network fabric — InfiniBand or NVLink Fabric is strongly recommended for pipeline parallelism.

### 4.4 Data Parallelism

Data parallelism runs multiple full model replicas and shards requests across them. Best for high-throughput scenarios with models that fit on a single GPU (or TP group):

```bash
# 2 replicas, each using 2 GPUs for TP
VLLM_DP_MASTER_IP=0.0.0.0 \
VLLM_DP_SIZE=2 \
vllm serve Qwen/Qwen2.5-7B-Instruct \
  --tensor-parallel-size 2 \
  --data-parallel-size 2
```

### 4.5 Kubernetes Deployment

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-server
spec:
  replicas: 1
  selector:
    matchLabels:
      app: vllm-server
  template:
    metadata:
      labels:
        app: vllm-server
    spec:
      containers:
      - name: vllm
        image: vllm/vllm-openai:latest
        args:
          - "vllm"
          - "serve"
          - "meta-llama/Meta-Llama-3.1-8B-Instruct"
          - "--tensor-parallel-size"
          - "1"
          - "--gpu-memory-utilization"
          - "0.90"
          - "--enable-prefix-caching"
        env:
        - name: VLLM_API_KEY
          valueFrom:
            secretKeyRef:
              name: vllm-secrets
              key: api-key
        - name: VLLM_NO_USAGE_STATS
          value: "1"
        ports:
        - containerPort: 8000
        resources:
          limits:
            nvidia.com/gpu: 1
        volumeMounts:
        - name: model-cache
          mountPath: /root/.cache/huggingface
        livenessProbe:
          httpGet:
            path: /health
            port: 8000
          initialDelaySeconds: 300
          periodSeconds: 30
        readinessProbe:
          httpGet:
            path: /health
            port: 8000
          initialDelaySeconds: 60
          periodSeconds: 10
      volumes:
      - name: model-cache
        persistentVolumeClaim:
          claimName: model-cache-pvc
```

### 4.6 ROCm (AMD GPU) Deployment

```bash
docker run \
  --device /dev/kfd \
  --device /dev/dri \
  --security-opt seccomp=unconfined \
  --group-add video \
  -e VLLM_TARGET_DEVICE=rocm \
  -e VLLM_ROCM_USE_AITER_LINEAR=1 \
  -e VLLM_ROCM_USE_AITER_MOE=1 \
  -e HIP_VISIBLE_DEVICES=0,1 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-rocm:latest \
  vllm serve meta-llama/Meta-Llama-3-8B-Instruct \
    --tensor-parallel-size 2
```

### 4.7 CPU-Only Deployment

```bash
# Build with CPU target
VLLM_TARGET_DEVICE=cpu pip install vllm

# Run
VLLM_CPU_KVCACHE_SPACE=40 \
VLLM_CPU_OMP_THREADS_BIND=auto \
vllm serve Qwen/Qwen2.5-7B-Instruct \
  --device cpu \
  --dtype bfloat16
```

CPU deployment is significantly slower than GPU. Use only for testing or low-throughput scenarios.

### 4.8 Systemd Unit File

```ini
[Unit]
Description=vLLM Inference Server
After=network.target
Wants=network.target

[Service]
Type=simple
User=vllm
Group=vllm
WorkingDirectory=/opt/vllm
EnvironmentFile=/opt/vllm/.env
ExecStart=/opt/vllm/venv/bin/vllm serve meta-llama/Meta-Llama-3.1-8B-Instruct \
    --tensor-parallel-size 1 \
    --gpu-memory-utilization 0.90 \
    --enable-prefix-caching \
    --port 8000
Restart=on-failure
RestartSec=30
TimeoutStartSec=600
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=multi-user.target
```

```bash
sudo systemctl enable vllm
sudo systemctl start vllm
sudo journalctl -u vllm -f
```

### 4.9 Prefill/Decode Disaggregation (P/D)

vLLM supports disaggregating prefill (prompt processing) and decode (token generation) workloads across separate server instances, configured via `--kv-transfer-config`. This allows independent scaling of each phase.

> [!CAUTION]
> P/D disaggregation is experimental. Known issues include OffloadingConnector segfaults in disaggregated mode (issue #39500) and GPU→CPU KV offload crashes with `cuMemcpyBatchAsync` (issue #39491). Validate thoroughly before production use and monitor closely.

---

## 5. Operations

### 5.1 Health Checks

```bash
# Basic health check
curl http://localhost:8000/health

# Model list (confirms model loaded and API is authenticated)
curl http://localhost:8000/v1/models \
  -H "Authorization: Bearer $VLLM_API_KEY"
```

> [!WARNING]
> The `/generate` endpoint (`vllm/entrypoints/api_server.py`) is a legacy demo server — **not** the production OpenAI-compatible API. For production, always use the OpenAI-compatible server (`vllm/entrypoints/openai/api_server.py`) which exposes `/v1/chat/completions`, `/v1/completions`, `/v1/embeddings`, and `/v1/models`. The legacy server has no authentication.

### 5.2 Model Loading & Warm-Up

vLLM downloads models from HuggingFace Hub on first launch. Subsequent starts use the local cache:

```bash
# Pre-download model before serving (avoids startup timeout on first request)
huggingface-cli download meta-llama/Meta-Llama-3.1-8B-Instruct

# Specify an explicit download directory
vllm serve meta-llama/Meta-Llama-3.1-8B-Instruct \
  --download-dir /data/models
```

Engine initialization (`VLLM_ENGINE_READY_TIMEOUT_S`, default 600 seconds) covers:
1. Model weight loading from disk
2. KV cache allocation and GPU memory profiling
3. CUDA graph capture at multiple batch sizes
4. Tokenizer loading

> [!NOTE]
> CUDA graph capture runs dummy forward passes at multiple batch sizes to pre-compile execution graphs. This is a one-time cost that significantly improves latency for small batches. The `--enforce-eager` flag disables CUDA graphs entirely and is useful for debugging but reduces throughput. For production, CUDA graphs should remain enabled.

### 5.3 Log Management

```bash
# Set log level
export VLLM_LOGGING_LEVEL=DEBUG

# Custom logging config (Python dictConfig format)
export VLLM_LOGGING_CONFIG_PATH=/opt/vllm/logging.yaml

# Disable color (for log files and aggregators)
export VLLM_LOGGING_COLOR=never

# Prefix all log lines (useful in aggregated log streams)
export VLLM_LOGGING_PREFIX="[vllm-prod-01] "
```

Stats are logged every `VLLM_LOG_STATS_INTERVAL` seconds (default 10) including:
- Running/waiting/swapped request counts
- GPU KV cache utilization percentage
- CPU KV cache utilization percentage
- Throughput (tokens/sec)

### 5.4 Usage Telemetry

By default vLLM reports anonymous usage statistics to `https://stats.vllm.ai`. Data includes hardware configuration and feature flags — no prompts or model outputs.

```bash
# Disable telemetry
export VLLM_NO_USAGE_STATS=1
# or
export DO_NOT_TRACK=1
```

> [!NOTE]
> The telemetry system respects the standard `DO_NOT_TRACK` environment variable used by many developer tools. Any of `VLLM_NO_USAGE_STATS=1`, `VLLM_DO_NOT_TRACK=1`, or `DO_NOT_TRACK=1` disables reporting. The stats endpoint can be overridden via `VLLM_USAGE_STATS_SERVER` for air-gapped environments running an internal stats collector.

### 5.5 Monitoring with OpenTelemetry

vLLM integrates with distributed tracing via OpenTelemetry (OTLP protocol) through `vllm/tracing/`:

```bash
export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=http://jaeger:4317

vllm serve your-model \
  --enable-opentelemetry \
  --otlp-traces-endpoint http://jaeger:4317
```

Spans are created for HTTP request handling, engine scheduling, model forward passes, and tokenization/detokenization.

### 5.6 Updating vLLM

> [!WARNING]
> vLLM updates frequently and sometimes introduces breaking changes to configuration flags, quantization support, or model compatibility. Always test a new version with `--enforce-eager` first to surface errors quickly before re-enabling CUDA graphs for production.

```bash
# Check current version
vllm --version

# Upgrade via pip
pip install --upgrade vllm

# Upgrade via Docker
docker pull vllm/vllm-openai:latest
docker compose up -d --no-deps vllm
```

### 5.7 Collecting Diagnostic Information

When filing a bug report or investigating issues:

```bash
# Collect complete environment info
vllm collect-env

# Outputs: Python version, CUDA version, PyTorch version,
# GPU model, driver version, vLLM version, OS info
```

### 5.8 Backup & Recovery

vLLM is stateless — all persistent state is in:
- Model weights (HuggingFace cache or local path)
- LoRA adapters (if using `--enable-lora`)
- Media cache (`VLLM_MEDIA_CACHE`)

Recovery procedure: restore model weights, restart the vLLM process. The KV cache is re-profiled automatically on startup; no database or queue state needs to be restored.

---

## 6. Performance

### 6.1 Throughput vs. Latency Trade-offs

| Goal | Primary Levers |
|---|---|
| Maximum throughput | Large `--max-num-batched-tokens`, `--enable-prefix-caching`, `--enable-chunked-prefill` |
| Minimum latency | `--max-num-seqs 1`, dedicated GPU per request |
| Memory efficiency | Lower `--gpu-memory-utilization`, quantization |
| Long context | `--enable-chunked-prefill`, `--max-model-len` tuning |

### 6.2 GPU Memory Utilization

`--gpu-memory-utilization` (default 0.90) controls what fraction of GPU VRAM is reserved for the KV cache after model weights are loaded:

```
kv_cache_vram = total_vram × gpu_memory_utilization − model_weights_vram
```

Raising this value increases the number of concurrent sequences. Values above 0.95 risk OOM errors under bursty load. Values below 0.80 reduce concurrency significantly on smaller GPU models.

### 6.3 Prefix Caching

Enable with `--enable-prefix-caching`. The engine maintains a hash-keyed map of KV blocks. When a new request shares a prefix with a cached block chain, those blocks are reused — dramatically reducing prefill cost for chat applications.

```bash
vllm serve your-model \
  --enable-prefix-caching \
  --gpu-memory-utilization 0.90
```

Best practices:
- Stable system prompts benefit most (100% cache hit for system prompt tokens)
- Few-shot prompt templates are excellent candidates
- Randomizing the system prompt per-request defeats caching entirely
- Monitor cache hit rate in stats output: `prefix_cache_hit_rate`

### 6.4 Chunked Prefill

`--enable-chunked-prefill` splits long prefill sequences across multiple scheduler steps, interleaving them with decode steps:

```bash
vllm serve your-model \
  --enable-chunked-prefill \
  --max-num-batched-tokens 8192
```

Benefits:
- Reduces time-to-first-token variance for other requests during long prefills
- Prevents long prompts from starving running decode sequences
- Enables admitting more sequences without OOM risk during burst

### 6.5 Quantization

Quantization reduces memory footprint and can improve throughput on quantization-accelerated hardware:

```bash
# AWQ (recommended for Turing/Ampere/Ada/Hopper)
vllm serve TheBloke/Llama-2-70B-AWQ \
  --quantization awq \
  --dtype half

# GPTQ
vllm serve TheBloke/Llama-2-70B-GPTQ \
  --quantization gptq

# FP8 (Ada/Hopper only)
vllm serve meta-llama/Meta-Llama-3-70B-Instruct \
  --quantization fp8 \
  --kv-cache-dtype fp8_e5m2

# BitsAndBytes (4-bit NF4, broad compatibility)
vllm serve meta-llama/Meta-Llama-3-70B-Instruct \
  --quantization bitsandbytes \
  --load-format bitsandbytes
```

### 6.6 Speculative Decoding

Speculative decoding accelerates generation by predicting multiple tokens with a cheap draft model, then verifying with the full model. Net speedup depends on the draft acceptance rate.

```bash
# N-gram speculative decoding (no draft model needed, good for repetitive text)
vllm serve your-model \
  --speculative-config '{"method": "ngram", "num_speculative_tokens": 5}'

# Draft model
vllm serve meta-llama/Meta-Llama-3-70B-Instruct \
  --speculative-config '{
    "model": "meta-llama/Meta-Llama-3-8B-Instruct",
    "num_speculative_tokens": 5
  }'

# EAGLE
vllm serve meta-llama/Meta-Llama-3-70B-Instruct \
  --speculative-config '{
    "method": "eagle",
    "model": "your-eagle-draft-model",
    "num_speculative_tokens": 5
  }'
```

> [!CAUTION]
> Known speculative decoding issues as of commit `58da4ee`: N-gram produces corrupted output on hybrid GDN models (issue #39273). MTP does not enforce thinking token budgets (issue #39573). Mamba prefix caching combined with MTP crashes on NemotronH models (issue #39809). Test speculative decoding against your specific model before production deployment.

### 6.7 torch.compile Integration

`vllm/compilation/` integrates PyTorch's `torch.compile` for kernel fusion and ahead-of-time optimization:

```bash
# Maximum optimization
vllm serve your-model \
  --compilation-config '{"optimization_level": 3}'

# Shorthand
vllm serve your-model --optimization-level 3

# All optimizations at once
vllm serve your-model --performance-mode
```

Compilation increases startup time on the first run (compiles and caches) but reduces per-token latency for subsequent runs. The compilation cache persists via PyTorch's inductor cache.

### 6.8 Benchmarking

```bash
# Throughput benchmark
python -m vllm.benchmarks.benchmark_throughput \
  --model meta-llama/Meta-Llama-3.1-8B-Instruct \
  --dataset sharegpt \
  --num-prompts 1000

# Latency benchmark (single-stream)
python -m vllm.benchmarks.benchmark_latency \
  --model meta-llama/Meta-Llama-3.1-8B-Instruct \
  --input-len 512 \
  --output-len 128 \
  --num-iters 50

# Via CLI
vllm bench --model your-model
```

### 6.9 Profiling

Use the layerwise profiler (`vllm/profiler/`) for per-layer timing:

```bash
vllm serve your-model \
  --profiler-config '{"enabled": true, "output_dir": "/tmp/vllm_profile"}'
```

Combined with NVIDIA Nsight Systems:

```bash
nsys profile --trace cuda,nvtx \
  vllm serve your-model --enforce-eager
```

### 6.10 LoRA Adapter Management

When `--enable-lora` is set, LoRA adapters can be loaded at runtime without restarting the server. The engine maintains an LRU cache of loaded adapters up to `--max-loras`.

```bash
vllm serve base-model \
  --enable-lora \
  --max-loras 4 \
  --max-lora-rank 64
```

Request a specific adapter in the API call:

```json
{
  "model": "base-model",
  "lora_request": {
    "lora_name": "my-adapter",
    "lora_int_id": 1,
    "lora_local_path": "/path/to/adapter"
  },
  "messages": [{"role": "user", "content": "Hello"}]
}
```

---

## 7. Security

### 7.1 Authentication

vLLM's built-in authentication is a single static API key:

```bash
export VLLM_API_KEY="your-secret-key"
```

All requests must include:

```http
Authorization: Bearer your-secret-key
```

> [!WARNING]
> The built-in API key mechanism is minimal — it is a single shared secret with no per-user rate limiting, no audit logging per key, and no key rotation support. For production deployments with multiple users, place vLLM behind an API gateway (Kong, AWS API Gateway, nginx) that handles authentication, rate limiting, and key management.

### 7.2 TLS / HTTPS

Enable TLS directly on the vLLM server:

```bash
vllm serve your-model \
  --ssl-keyfile /path/to/server.key \
  --ssl-certfile /path/to/server.crt \
  --host 0.0.0.0 \
  --port 443
```

Alternatively, terminate TLS at a reverse proxy (nginx, Traefik, Caddy) and proxy to vLLM on localhost. This is the recommended approach for production.

### 7.3 Network Isolation

> [!CAUTION]
> Never expose the vLLM HTTP port directly to the public internet without authentication. If `VLLM_API_KEY` is not set, the API is open to anyone who can reach the port. The legacy demo server (`api_server.py`) has no authentication at all. Always set `VLLM_API_KEY` in production and restrict network access at the firewall or load balancer level.

Recommended network architecture:

```
Internet → Load Balancer / API Gateway (TLS termination, auth, rate limiting)
               │
               ▼
           vLLM Server (private subnet only, not internet-routable)
               │  localhost or private cluster network
               ▼
           GPU Workers (same host or private cluster network)
```

### 7.4 Insecure Serialization

> [!CAUTION]
> `VLLM_ALLOW_INSECURE_SERIALIZATION=1` enables pickle-based object serialization between engine processes. Pickle is inherently unsafe if any untrusted data can reach the serialization path. Never enable this in production systems.

### 7.5 Remote Code Execution Risk

`--trust-remote-code` allows arbitrary Python code embedded in a HuggingFace model's files to execute during model loading. This is required for some custom model architectures.

> [!WARNING]
> Only use `--trust-remote-code` with models from sources you explicitly trust. HuggingFace Hub models can contain arbitrary Python that executes with the full privileges of the vLLM process. Always run vLLM as a non-root user. Consider using a read-only filesystem for the model cache directory and running in a container with restricted capabilities (`--cap-drop ALL`, `--security-opt no-new-privileges:true`).

### 7.6 Media Fetch SSRF Risk

For multimodal models, vLLM fetches media (images, video, audio) from URLs in requests. This creates a Server-Side Request Forgery (SSRF) risk if the server can reach internal network resources.

Mitigations:
- Run vLLM in a network namespace that cannot reach internal services
- Configure `VLLM_MEDIA_CONNECTOR` to use a proxy that enforces URL allowlists
- Set fetch timeouts conservatively to bound resource exhaustion

### 7.7 Request/Response Logging

As of commit `58da4ee`, complete request/response logging is not built into vLLM (see issue #40155: feature request for complete request/response logging). To log full request/response pairs for audit or compliance purposes, implement logging middleware at the reverse proxy layer.

### 7.8 Secrets Management

| Variable | Secret Type |
|---|---|
| `VLLM_API_KEY` | Server authentication key |
| `HF_TOKEN` / `--hf-token` | HuggingFace Hub access token |

Use a secrets manager (HashiCorp Vault, AWS Secrets Manager, Kubernetes Secrets) to inject these at runtime rather than storing them in container images or shell scripts.

---

## 8. Developer Internals

### 8.1 Codebase Scale

| Metric | Value |
|---|---|
| Total files | 4,441 |
| Total source chars | 19.9 million |
| HIGH-priority modules | 147 |
| LOW-priority modules | 373 |
| Extracted classes | 2,854 |
| Extracted functions | 13,462 |
| Error messages | 1,550 |
| CLI flags | 911 |
| Config keys (env vars) | 346 |
| Model implementations | 200+ |
| Source commit | 58da4ee (2026-04-17) |
| License | Apache-2.0 |

### 8.2 Build System

vLLM uses a CMake-based build system wrapped by `setuptools` (`setup.py`). The build system:

1. Detects target device from `VLLM_TARGET_DEVICE` or installed hardware
2. Configures CMake for the appropriate kernel set (CUDA, ROCm, XPU, CPU)
3. Compiles C++/CUDA kernels in `csrc/`
4. Installs the compiled extensions alongside the Python package

```bash
# Standard install (auto-detects GPU)
pip install -e .

# Force CUDA target
VLLM_TARGET_DEVICE=cuda pip install -e .

# ROCm target
VLLM_TARGET_DEVICE=rocm pip install -e .

# CPU-only (no GPU kernels)
VLLM_TARGET_DEVICE=cpu pip install -e .

# Accelerate build with sccache
pip install sccache
VLLM_DISABLE_SCCACHE=0 pip install -e .

# Parallel compilation
MAX_JOBS=16 NVCC_THREADS=4 pip install -e .
```

#### CUDA Kernel Modules (`csrc/`)

| Directory | Contents |
|---|---|
| `csrc/attention/` | PagedAttention CUDA kernels |
| `csrc/cpu/` | CPU-optimized attention kernels |
| `csrc/moe/` | Mixture-of-Experts routing kernels |
| `csrc/quantization/` | AWQ, GPTQ, FP8, INT8 quantization kernels |
| `csrc/cutlass/` | CUTLASS-based GEMM kernels |
| `csrc/quickreduce/` | Fast reduction operations |
| `csrc/rocm/` | ROCm-specific kernels |
| `csrc/mamba/` | Mamba SSM kernels |
| `csrc/core/` | Core utility kernels |
| `csrc/libtorch_stable/` | Stable LibTorch bindings |

### 8.3 Adding a New Model

To add a new model architecture:

1. Create `vllm/model_executor/models/your_model.py`
2. Implement the model class inheriting from `nn.Module` with `forward()` accepting vLLM's input format
3. Register the model in `vllm/model_executor/models/__init__.py`
4. If the model has a custom attention pattern, implement the attention class in `vllm/attention/backends/`
5. Add tests in `tests/models/`

Key interfaces to implement:
- `forward(input_ids, positions, kv_caches, attn_metadata)` — main forward pass
- `load_weights(weights_iterator)` — weight loading from HuggingFace format
- `sampler` property — returns the `Sampler` instance

### 8.4 Plugin System

`vllm/plugins/` provides a registration system for extending vLLM without modifying core code:

```python
# Register via Python entry_points in setup.py or pyproject.toml
[options.entry_points]
vllm.general_plugins =
    my_plugin = my_package.plugin:register
```

### 8.5 Custom Quantization Plugin

Register a custom quantization method using the decorator pattern:

```python
from vllm.model_executor.layers.quantization import register_quantization_config
from vllm.model_executor.layers.quantization.base_config import QuantizationConfig

@register_quantization_config("my_quant")
class MyQuantConfig(QuantizationConfig):
    def get_name(self) -> str:
        return "my_quant"

    def get_supported_act_dtypes(self):
        return [torch.float16, torch.bfloat16]

    def get_min_capability(self) -> int:
        return 75  # Turing+

    def get_config_filenames(self):
        return ["quantize_config.json"]

    @classmethod
    def from_config(cls, config: dict) -> "MyQuantConfig":
        return cls(...)

    def get_quant_method(self, layer, prefix):
        return MyQuantLinearMethod(self)
```

### 8.6 Platform Abstraction

`vllm/platforms/` provides a unified interface hiding GPU vendor differences:

```
Platform (abstract)
├── CUDAPlatform        — NVIDIA CUDA
├── ROCmPlatform        — AMD ROCm/HIP
├── XPUPlatform         — Intel XPU
├── CPUPlatform         — x86 CPU
└── EmptyPlatform       — No-op (testing)
```

Platform-specific code should always use the abstraction layer:

```python
from vllm.platforms import current_platform

if current_platform.is_cuda():
    # CUDA-specific code
elif current_platform.is_rocm():
    # ROCm-specific code
```

### 8.7 Reasoning Parsers

`vllm/reasoning/` (21 files, 135K chars) provides pluggable parsers for chain-of-thought models:

```bash
# DeepSeek-R1
vllm serve deepseek-ai/DeepSeek-R1 --reasoning-parser deepseek_r1

# Qwen3
vllm serve Qwen/Qwen3-72B --reasoning-parser qwen3
```

The parser extracts `reasoning_content` separately from the final `content` in the API response.

### 8.8 Tool Calling Parsers

`vllm/tool_parsers/` (37 files, 611K chars) implements function/tool calling for models that generate tool call JSON in their output. Different model families use different formats:

```bash
# Llama 3 JSON tool calling format
vllm serve meta-llama/Meta-Llama-3.1-8B-Instruct \
  --tool-call-parser llama3_json
```

### 8.9 Pooling Models (Embeddings & Reranking)

vLLM supports embedding, classification, and reranking via the pooling interface:

| Task Type | Description |
|---|---|
| `embed` | Generate dense embeddings |
| `classify` | Sequence classification |
| `token_classify` | Token-level classification (NER, etc.) |
| `token_embed` | Per-token embeddings |
| `plugin` | Custom pooling via plugin |

Pooling strategies: `CLS`, `LAST`, `MEAN`, `ALL`, `STEP`

Score types for reranking: `cross-encoder`, `late-interaction` (ColBERT), `bi-encoder`

### 8.10 Intermediate Representation

`vllm/ir/` (3 files, 17K chars) provides an IR layer used by the compilation pipeline. It enables analysis and transformation of computation graphs before they are compiled by `torch.compile`.

### 8.11 Testing

```bash
# Run all unit tests
pytest tests/

# Run without slow tests
pytest tests/ -m "not slow"

# Run model-specific tests
pytest tests/models/ -k "llama"

# Integration tests (requires GPU)
pytest tests/entrypoints/ -v

# Kernel unit tests
pytest tests/kernels/ -v
```

Test directory structure:
- `tests/models/` — per-model correctness tests
- `tests/entrypoints/` — API server integration tests
- `tests/kernels/` — CUDA kernel unit tests
- `tests/samplers/` — sampling correctness tests
- `tests/lora/` — LoRA adapter tests
- `tests/quantization/` — quantization tests

### 8.12 Code Conventions

- **Type hints:** All public APIs use Python type hints throughout
- **Dataclasses:** Configuration uses `@dataclass` (`VllmConfig` and all sub-configs)
- **Async:** The async engine uses `asyncio` throughout; sync wrappers exist for offline/batch use
- **Error messages:** 1,550 distinct error messages extracted — errors are specific and actionable
- **Platform guards:** Platform-specific code uses `current_platform.is_*()` checks, never bare `torch.cuda.is_available()`
- **Env vars:** All environment variables centrally defined in `vllm/envs.py` with type coercions and defaults

---

## 9. Integration Points

### 9.1 API Endpoints

The OpenAI-compatible server (`vllm/entrypoints/openai/api_server.py`) exposes:

| Method | Path | Auth Required | Description |
|---|---|---|---|
| GET | `/health` | No | Health check (returns 200 OK) |
| GET | `/v1/models` | Yes (if key set) | List loaded models |
| POST | `/v1/chat/completions` | Yes (if key set) | Chat completion |
| POST | `/v1/completions` | Yes (if key set) | Text completion |
| POST | `/v1/embeddings` | Yes (if key set) | Embedding generation |

Additional endpoints are registered dynamically based on enabled features (tokenization, reranking, score endpoints for pooling models).

The legacy demo server (`vllm/entrypoints/api_server.py`):

| Method | Path | Auth | Description |
|---|---|---|---|
| GET | `/health` | None | Health check |
| POST | `/generate` | None | Single-turn generation (not OpenAI-compatible) |

### 9.2 OpenAI Client Compatibility

Any OpenAI Python SDK client can target a vLLM server:

```python
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="your-vllm-api-key",  # matches VLLM_API_KEY
)

response = client.chat.completions.create(
    model="meta-llama/Meta-Llama-3.1-8B-Instruct",
    messages=[{"role": "user", "content": "Hello"}],
    max_tokens=512,
    temperature=0.7,
)
print(response.choices[0].message.content)
```

### 9.3 HuggingFace Transformers Integration

`vllm/transformers_utils/` (12 files, 120K chars) bridges vLLM with the HuggingFace ecosystem:

- Loads model configs via `AutoConfig`
- Loads tokenizers via `AutoTokenizer`
- Applies chat templates via `tokenizer.apply_chat_template()`
- Handles special tokens, padding, BOS/EOS configuration

Use `--hf-overrides` to patch `config.json` fields without modifying the files on disk.

### 9.4 Ray Integration

`vllm/ray/` (2 files, 5K chars) integrates with Ray for multi-node distributed execution:

```python
import ray
ray.init(address="auto")  # Connect to existing cluster
```

Ray workers are managed by `RayWorkerWrapper` which handles remote method calls across the Ray actor system. Select with `--distributed-executor-backend ray`.

### 9.5 Structured Output Integration

For JSON-schema-constrained generation, vLLM integrates with:
- **xgrammar** — grammar-based constrained decoding (`VLLM_XGRAMMAR_CACHE_MB` controls cache)
- **guidance** — model guidance library

```bash
vllm serve your-model \
  --structured-outputs-config '{"backend": "xgrammar"}'
```

### 9.6 Observability Integrations

| System | Integration Method |
|---|---|
| OpenTelemetry | Built-in via `vllm/tracing/` + `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` |
| Prometheus | Metrics at `/metrics` (when enabled) |
| Grafana | Prometheus scraping |
| Jaeger | OTLP trace export |
| Datadog | OTLP-compatible agent |

### 9.7 Offline Batch Processing

For batch inference without running a server:

```bash
# CLI batch mode
vllm run-batch \
  --model your-model \
  --input-file prompts.jsonl \
  --output-file outputs.jsonl
```

```python
# Programmatic Python API
from vllm import LLM, SamplingParams

llm = LLM(model="your-model")
outputs = llm.generate(
    ["Prompt 1", "Prompt 2"],
    SamplingParams(temperature=0.8, max_tokens=256)
)
for output in outputs:
    print(output.outputs[0].text)
```

---

## 10. Troubleshooting

### 10.1 Engine Fails to Start

**Symptom:** Process exits during initialization; timeout waiting for engine.

**Diagnosis:**
```bash
# Disable CUDA graphs to isolate the error
vllm serve your-model --enforce-eager 2>&1 | head -100

# Check GPU memory
nvidia-smi --query-gpu=memory.free,memory.total --format=csv
```

**Common causes:**
- Insufficient GPU VRAM — lower `--gpu-memory-utilization` or use quantization
- CUDA version mismatch — verify `nvcc --version` matches PyTorch CUDA build
- Model too large — use tensor parallelism or quantization
- Missing model files — verify `~/.cache/huggingface/hub/` contains the model

### 10.2 CUDA Out of Memory (OOM)

**Symptom:** `torch.cuda.OutOfMemoryError` during inference.

**Solutions:**
```bash
# Reduce memory reservation
vllm serve your-model --gpu-memory-utilization 0.80

# Reduce max sequence length
vllm serve your-model --max-model-len 4096

# Reduce max concurrent sequences
vllm serve your-model --max-num-seqs 128

# Enable chunked prefill to smooth memory peaks
vllm serve your-model --enable-chunked-prefill

# Use quantization to reduce model footprint
vllm serve your-model --quantization awq
```

### 10.3 Slow Throughput

**Symptom:** Requests per second or tokens per second below expectations.

```bash
# Baseline measurement
python -m vllm.benchmarks.benchmark_throughput \
  --model your-model --num-prompts 500

# Watch GPU utilization live
watch -n 1 nvidia-smi
```

**Solutions:** Enable prefix caching, increase `--max-num-batched-tokens`, enable chunked prefill, verify CUDA graphs are active (not `--enforce-eager`), consider speculative decoding for decode-heavy workloads.

### 10.4 High Time-to-First-Token (TTFT)

**Symptom:** First tokens take much longer than subsequent tokens.

**Cause:** Long prompt processing (prefill) blocks decode of other requests.

```bash
# Chunked prefill distributes prefill cost across steps
vllm serve your-model \
  --enable-chunked-prefill \
  --max-num-batched-tokens 2048
```

### 10.5 Stuck Engine / Timeout

**Symptom:** `VLLM_ENGINE_ITERATION_TIMEOUT_S` exceeded; engine appears hung.

```bash
# Increase timeout for slow hardware
export VLLM_ENGINE_ITERATION_TIMEOUT_S=120

# Debug NCCL in multi-GPU setup
export NCCL_DEBUG=INFO
export NCCL_DEBUG_SUBSYS=ALL
```

> [!WARNING]
> If `VLLM_KEEP_ALIVE_ON_ENGINE_DEATH=0` (default), the HTTP server terminates when the engine process dies. Set to `1` to keep the server accepting requests (which will error) while you diagnose the engine failure. In load-balanced deployments, leaving this at `0` means the health check fails and traffic routes away automatically, which is usually the correct behavior.

### 10.6 Multi-GPU NCCL Errors

**Symptom:** `NCCL error`, `Timeout waiting for NCCL`, deadlock in multi-GPU setup.

```bash
# Run P2P connectivity check (disabled by default)
export VLLM_SKIP_P2P_CHECK=0

# Override NCCL library path
export VLLM_NCCL_SO_PATH=/usr/local/lib/libnccl.so.2

# Disable NVLink P2P if causing issues
export NCCL_P2P_DISABLE=1
```

### 10.7 Quantization Errors

**Symptom:** Error about unsupported quantization on current hardware.

Check Section 2.5 hardware compatibility table. Common cases:
- FP8 on Ampere or older: use INT8 or AWQ instead
- Marlin on Volta: use GPTQ or BitsAndBytes
- AWQ on AMD: use GPTQ or GGUF

### 10.8 Speculative Decoding Issues

**Known bugs as of commit `58da4ee`:**

| Issue | Symptom | Workaround |
|---|---|---|
| #39273 | N-gram: corrupted output on hybrid GDN models | Disable speculative decoding |
| #39573 | MTP: thinking token budget not enforced | Disable MTP or thinking budget |
| #39809 | Mamba prefix caching + MTP crashes on NemotronH | Disable prefix caching or MTP |

> [!CAUTION]
> Speculative decoding bugs can produce silently incorrect output — tokens that appear syntactically valid but contain semantic errors. Validate speculative decoding output quality with your specific model before production deployment.

### 10.9 LoRA Not Loading

**Symptom:** LoRA adapter fails to load or has no effect.

**Checklist:**
- `--enable-lora` must be set at server start
- `--max-lora-rank` must be ≥ the adapter's actual rank
- `--max-loras` must be increased when loading multiple adapters
- Adapter must be compatible with the base model architecture
- Adapter must be in HuggingFace PEFT format

### 10.10 Reasoning Parser Not Working

**Symptom:** `reasoning_content` is empty or `content` includes raw thinking tokens.

```bash
# Explicitly set the parser
vllm serve your-model --reasoning-parser deepseek_r1
# or
vllm serve your-model --reasoning-parser qwen3
```

Verify the model actually generates expected thinking tokens before the final answer.

### 10.11 P/D Disaggregation Crashes

**Known issues:**
- Issue #39500: OffloadingConnector segfaults in P/D disaggregated mode
- Issue #39491: GPU→CPU KV offload crashes with `cuMemcpyBatchAsync`

**Workaround:** Avoid `OffloadingConnector` in production until these are resolved upstream.

### 10.12 MoE Kernel Config Fails on GH200

**Symptom:** Error during MoE kernel configuration on NVIDIA GH200 (issue #37191).

```bash
# Disable DeepGEMM MoE path as workaround
export VLLM_MOE_USE_DEEP_GEMM=0
```

### 10.13 Batch-Invariant Attention Warning

**Context:** Issue #40173 tracks automatic selection of the highest-priority batch-invariant attention backend.

```bash
# Force a specific attention backend
vllm serve your-model --attention-backend flashinfer

# Or enable batch invariant mode
export VLLM_BATCH_INVARIANT=1
```

### 10.14 torch.compile Cache Issues

**Symptom:** Recompilation on every startup; slow startup after vLLM updates.

```bash
# Clear inductor cache
rm -rf ~/.cache/torch/inductor

# Or set an explicit cache directory
export TORCHINDUCTOR_CACHE_DIR=/opt/vllm/torch_cache
```

### 10.15 Media Fetch Failures (Multimodal)

**Symptom:** Image/video/audio URLs fail or time out.

```bash
export VLLM_IMAGE_FETCH_TIMEOUT=30
export VLLM_VIDEO_FETCH_TIMEOUT=120
export VLLM_MEDIA_CACHE=/tmp/vllm_media   # Enable caching to avoid re-fetching
export VLLM_MEDIA_FETCH_MAX_RETRIES=5
export VLLM_MEDIA_LOADING_THREAD_COUNT=16
```

### 10.16 RPC Timeout Between Engine Processes

**Symptom:** `RPC timeout` error; engine appears to hang waiting for workers.

```bash
# Increase RPC timeout (default 10000ms)
export VLLM_RPC_TIMEOUT=30000

# Increase model execution timeout (default 300s)
export VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=600
```

### 10.17 High Memory Usage at Idle

**Explanation:** This is expected. vLLM pre-allocates the full KV cache at startup (controlled by `--gpu-memory-utilization`). Pre-allocation eliminates runtime allocation overhead and provides deterministic memory usage.

To reduce idle footprint at the cost of reduced concurrent capacity:
```bash
vllm serve your-model --gpu-memory-utilization 0.70
```

### 10.18 Environment Variable Has No Effect

**Cause:** vLLM reads environment variables at Python module import time.

**Solution:** Set variables in the shell before running `vllm serve`, in the systemd `EnvironmentFile=`, or in the Docker `environment:` section. Restart the process after any change.

### 10.19 OpenTelemetry Traces Not Appearing

**Checklist:**
```bash
# Verify collector is reachable
curl http://jaeger:4317

# Set endpoint
export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=http://jaeger:4317

# Enable tracing
vllm serve your-model --enable-opentelemetry
```

### 10.20 Complete Request/Response Logging

As of commit `58da4ee`, full request/response logging is not built into vLLM (issue #40155 tracks this feature request). Implement logging at the reverse proxy layer for compliance or audit requirements.

---

## Appendix A: Quick Reference

### A.1 Common Serve Command Patterns

```bash
# Small model, single GPU
vllm serve Qwen/Qwen2.5-7B-Instruct

# Production: 4-GPU tensor parallel, prefix caching, auth
VLLM_API_KEY=secret VLLM_NO_USAGE_STATS=1 \
vllm serve meta-llama/Meta-Llama-3-70B-Instruct \
  --tensor-parallel-size 4 \
  --gpu-memory-utilization 0.90 \
  --enable-prefix-caching \
  --max-model-len 32768 \
  --port 8000

# Quantized 70B on 2×A100 40GB
vllm serve TheBloke/Llama-2-70B-AWQ \
  --quantization awq \
  --tensor-parallel-size 2

# Embedding model
vllm serve BAAI/bge-m3 \
  --task embed \
  --enforce-eager

# Debug mode
VLLM_LOGGING_LEVEL=DEBUG \
vllm serve your-model --enforce-eager
```

### A.2 Key Environment Variables: Quick Reference

```bash
# Auth & Network
VLLM_API_KEY=your-key
VLLM_PORT=8000

# Logging
VLLM_LOGGING_LEVEL=INFO
VLLM_NO_USAGE_STATS=1
VLLM_LOG_STATS_INTERVAL=10

# GPU selection
CUDA_VISIBLE_DEVICES=0,1,2,3

# Performance (defaults are production-ready)
VLLM_USE_DEEP_GEMM=1
VLLM_MOE_USE_DEEP_GEMM=1
VLLM_ENABLE_V1_MULTIPROCESSING=1

# Timeouts
VLLM_ENGINE_READY_TIMEOUT_S=600
VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=300
VLLM_RPC_TIMEOUT=10000
```

### A.3 Pre-Launch Checklist

- [ ] `VLLM_API_KEY` is set (production)
- [ ] `VLLM_NO_USAGE_STATS=1` is set (production)
- [ ] `CUDA_VISIBLE_DEVICES` is correct
- [ ] GPU VRAM ≥ (model size ÷ TP size) + KV cache headroom
- [ ] Model weights are downloaded, or `HF_TOKEN` is set for gated models
- [ ] `--tensor-parallel-size` matches GPU count
- [ ] `--max-model-len` fits within model's `max_position_embeddings`
- [ ] TLS is configured (or terminated at reverse proxy)
- [ ] Health check responds: `curl http://localhost:8000/health`
- [ ] Log stream is captured (systemd journal or Docker logs)
- [ ] For multi-GPU: Ray initialized (if using `--distributed-executor-backend ray`)
- [ ] For speculative decoding: tested against target model (see known bugs above)

---

_Generated May 2026 from vllm-project/vllm at 58da4ee. AI-generated documentation. The library is checked by sampling (three repositories so far); this guide was not individually reviewed. Verify against the project before relying on it. Unofficial; not affiliated with or reviewed by the project._