# FastChat User Guide

FastChat is an open platform for **training, serving, and evaluating** large language model (LLM) based chatbots. It powers Chatbot Arena ([lmarena.ai](https://lmarena.ai)) and gives you three things on your own hardware:

- A **command-line chat** interface for talking to local or downloaded models.
- A **distributed multi-model serving system** with a web UI and **OpenAI-compatible REST APIs**, so FastChat can be a drop-in local replacement for the OpenAI API.
- **Training and evaluation** code for state-of-the-art models such as Vicuna and the MT-Bench benchmark.

This guide shows you how to install FastChat, chat with a model from your terminal, stand up a web UI and an OpenAI-compatible API server, run quantized models on limited hardware, fine-tune your own model, and evaluate models with MT-Bench. Every command, flag, and environment variable below is taken directly from the source code.

> [!NOTE]
> The Python package is named **`fschat`** (version `0.2.36`), but you import and run it as **`fastchat`** (e.g. `python3 -m fastchat.serve.cli`). Both names refer to the same project.

---

## Table of Contents

1. [Quick Start](#1-quick-start)
2. [Installation](#2-installation)
3. [Chatting from the Command Line](#3-chatting-from-the-command-line)
4. [Running on Limited Hardware](#4-running-on-limited-hardware)
5. [Serving with a Web GUI](#5-serving-with-a-web-gui)
6. [The OpenAI-Compatible API Server](#6-the-openai-compatible-api-server)
7. [High-Throughput Serving (vLLM, SGLang, and others)](#7-high-throughput-serving-vllm-sglang-and-others)
8. [Launching Chatbot Arena](#8-launching-chatbot-arena)
9. [Evaluating Models with MT-Bench](#9-evaluating-models-with-mt-bench)
10. [Fine-Tuning Your Own Model](#10-fine-tuning-your-own-model)
11. [Preparing Training Data](#11-preparing-training-data)
12. [How FastChat Works](#12-how-fastchat-works)
13. [Troubleshooting](#13-troubleshooting)
14. [Frequently Asked Questions](#14-frequently-asked-questions)
15. [Command & Flag Reference](#15-command--flag-reference)

---

## 1. Quick Start

If you have a GPU and just want to chat with a model:

```bash
pip3 install "fschat[model_worker,webui]"
python3 -m fastchat.serve.cli --model-path lmsys/vicuna-7b-v1.5
```

The model weights download automatically from Hugging Face on first run and are cached in your home folder (`~/.cache/huggingface/hub/`). Once loaded, you get an interactive prompt:

```text
USER: Hello! Who are you?
ASSISTANT: I am Vicuna, a language model trained by researchers...
```

To stand up an **OpenAI-compatible API** instead, you launch three processes — a controller, a model worker, and the API server:

```bash
python3 -m fastchat.serve.controller
python3 -m fastchat.serve.model_worker --model-path lmsys/vicuna-7b-v1.5
python3 -m fastchat.serve.openai_api_server --host localhost --port 8000
```

You can then call `http://localhost:8000/v1/chat/completions` exactly like the OpenAI API.

> [!TIP]
> Run each of the three serving processes in its own terminal (or with a process manager like `tmux`). They are long-running and talk to each other over local HTTP.

---

## 2. Installation

FastChat requires **Python 3.8 or newer**.

### Method 1: Install from PyPI

=== "Serving + Web UI"

    ```bash
    pip3 install "fschat[model_worker,webui]"
    ```

=== "Base only"

    ```bash
    pip3 install fschat
    ```

### Method 2: Install from source

Install from source when you want to fine-tune, modify the code, or track the latest commit.

```bash
git clone https://github.com/lm-sys/FastChat.git
cd FastChat
pip3 install --upgrade pip   # enable PEP 660 support
pip3 install -e ".[model_worker,webui]"
```

On a Mac you also need Rust and CMake before installing (required to build the `tokenizers` and `qwen` dependencies):

```bash
brew install rust cmake
```

### Optional dependency groups

FastChat splits its heavier dependencies into extras. Install only what you need:

| Extra | Install command | What it adds |
|---|---|---|
| `model_worker` | `pip3 install "fschat[model_worker]"` | `torch`, `transformers>=4.31`, `accelerate`, `peft`, `sentencepiece`, `protobuf`, `openai`, `anthropic` — needed to load and serve models |
| `webui` | `pip3 install "fschat[webui]"` | `gradio>=4.10`, `plotly`, `scipy` — the web chat and Arena UI |
| `train` | `pip3 install "fschat[train]"` | `einops`, `flash-attn>=2.0`, `wandb` — fine-tuning |
| `llm_judge` | `pip3 install "fschat[llm_judge]"` | `openai<1`, `anthropic>=0.3`, `ray` — MT-Bench evaluation |
| `dev` | `pip3 install "fschat[dev]"` | `black==23.3.0`, `pylint==2.8.2` — formatting and linting |

You can combine extras in one bracketed list, for example `pip3 install -e ".[model_worker,webui,train]"`.

> [!IMPORTANT]
> `transformers>=4.31` is required to use the 16K-context Vicuna models. The `model_worker` extra already pins this, but verify it if you installed `transformers` separately.

### Verify the installation

```bash
python3 -c "import fastchat; print('FastChat ready')"
```

```text
FastChat ready
```

---

## 3. Chatting from the Command Line

The CLI is the fastest way to talk to a model. The entry point is `fastchat.serve.cli`.

```bash
python3 -m fastchat.serve.cli --model-path lmsys/vicuna-7b-v1.5
```

`--model-path` accepts either a **local folder** or a **Hugging Face repo ID**. The default is `lmsys/vicuna-7b-v1.5`.

### In-chat commands

While chatting, these special inputs control the session:

| Type this | Effect |
|---|---|
| `!!exit` or an empty line | Exit the chat |
| `!!reset` | Start a new conversation (clears history) |
| `!!remove` | Remove the last prompt |
| `!!regen` | Regenerate the last assistant message |
| `!!save <filename>` | Save the conversation history to a JSON file |
| `!!load <filename>` | Load a conversation history from a JSON file |

### Sampling and generation options

| Flag | Default | Description |
|---|---|---|
| `--temperature` | `0.7` | Sampling temperature; lower is more deterministic |
| `--repetition_penalty` | `1.0` | Penalty applied to repeated tokens |
| `--max-new-tokens` | `512` | Maximum number of tokens to generate per reply |
| `--conv-template` | *(auto)* | Conversation prompt template to use |
| `--conv-system-msg` | *(none)* | Override the system message for the conversation |
| `--no-history` | off | Disable conversation history |

### Choosing a display style

The `--style` flag controls how output is rendered:

```bash
python3 -m fastchat.serve.cli --model-path lmsys/vicuna-7b-v1.5 --style rich
```

| Style | Behavior |
|---|---|
| `simple` *(default)* | Plain streaming text |
| `rich` | Rich-text rendering with better streaming and Markdown; supports `--multiline` and `--mouse` |
| `programmatic` | Machine-readable I/O for scripting FastChat from another program |

> [!NOTE]
> `--style rich` is experimental. It renders Markdown and non-ASCII content more cleanly but may not behave correctly on every terminal.

Enable multi-line input (use **ESC then Enter** for a newline) and mouse positioning in rich mode:

```bash
python3 -m fastchat.serve.cli --model-path lmsys/vicuna-7b-v1.5 --style rich --multiline --mouse
```

### Other useful CLI flags

| Flag | Description |
|---|---|
| `--debug` | Print debug information such as the full prompt sent to the model |
| `--judge-sent-end` | Enable correction logic for sentences cut off by an early end-of-sequence token |

### Models you can chat with

FastChat supports a wide range of model families, including Llama 2, Vicuna, Alpaca, Baize, ChatGLM, Dolly, Falcon, FastChat-T5, GPT4ALL, Guanaco, MPT, OpenAssistant, OpenChat, RedPajama, StableLM, WizardLM, Yuan2, and more. The official models published by LMSYS:

| Model | Chat command | Approx. GPU memory |
|---|---|---|
| Vicuna-7B | `python3 -m fastchat.serve.cli --model-path lmsys/vicuna-7b-v1.5` | ~14 GB |
| Vicuna-7B-16k | `python3 -m fastchat.serve.cli --model-path lmsys/vicuna-7b-v1.5-16k` | ~14 GB |
| Vicuna-13B | `python3 -m fastchat.serve.cli --model-path lmsys/vicuna-13b-v1.5` | ~28 GB |
| Vicuna-13B-16k | `python3 -m fastchat.serve.cli --model-path lmsys/vicuna-13b-v1.5-16k` | ~28 GB |
| Vicuna-33B | `python3 -m fastchat.serve.cli --model-path lmsys/vicuna-33b-v1.3` | ~60 GB |
| LongChat-7B | `python3 -m fastchat.serve.cli --model-path lmsys/longchat-7b-32k-v1.5` | ~14 GB |
| FastChat-T5-3B | `python3 -m fastchat.serve.cli --model-path lmsys/fastchat-t5-3b-v1.0` | ~6 GB |

> [!TIP]
> Vicuna is based on Llama 2 and must be used under Llama's model license. Weights download automatically from the Hugging Face repos shown above.

---

## 4. Running on Limited Hardware

You do not need a data-center GPU to run FastChat. The CLI, web worker, and API worker all share the same hardware flags (defined in `fastchat.model.model_adapter`), so everything below works for serving too.

### Multiple GPUs (model parallelism)

Aggregate memory across GPUs on the same machine:

```bash
python3 -m fastchat.serve.cli --model-path lmsys/vicuna-7b-v1.5 --num-gpus 2
```

If the automatic device mapping does not balance memory well, cap the per-GPU weight memory so more room is left for activations and longer context:

```bash
python3 -m fastchat.serve.cli --model-path lmsys/vicuna-7b-v1.5 --num-gpus 2 --max-gpu-memory 8GiB
```

You can also pin specific GPUs with `--gpus`:

```bash
python3 -m fastchat.serve.cli --model-path lmsys/vicuna-7b-v1.5 --gpus 0,2 --num-gpus 2
```

> [!CAUTION]
> If `--num-gpus` is larger than the number of GPUs listed in `--gpus`, FastChat raises `ValueError: Larger --num-gpus (N) than --gpus ...`. Keep the two consistent.

### CPU only

Runs without a GPU. Vicuna-7B needs ~30 GB and Vicuna-13B ~60 GB of system RAM:

```bash
python3 -m fastchat.serve.cli --model-path lmsys/vicuna-7b-v1.5 --device cpu
```

Use an Intel AI accelerator (AVX512_BF16 / AMX) to speed up CPU inference:

```bash
CPU_ISA=amx python3 -m fastchat.serve.cli --model-path lmsys/vicuna-7b-v1.5 --device cpu
```

### Apple Silicon and other accelerators

The `--device` flag selects the backend. Valid values are `cuda` (default), `cpu`, `mps`, `xpu`, and `npu`.

=== "Apple Silicon / AMD (Metal)"

    ```bash
    python3 -m fastchat.serve.cli --model-path lmsys/vicuna-7b-v1.5 --device mps --load-8bit
    ```

    Requires `torch >= 2.0`. Vicuna-7B runs on a 32 GB M1 MacBook at roughly 1–2 words/second.

=== "Intel XPU (Arc / Data Center GPU)"

    ```bash
    source /opt/intel/oneapi/setvars.sh
    python3 -m fastchat.serve.cli --model-path lmsys/vicuna-7b-v1.5 --device xpu
    ```

    Requires the Intel Extension for PyTorch. Vicuna-7B runs on an Intel Arc A770 16 GB.

=== "Ascend NPU"

    ```bash
    source /usr/local/Ascend/ascend-toolkit/set_env.sh
    python3 -m fastchat.serve.cli --model-path lmsys/vicuna-7b-v1.5 --device npu
    ```

    Requires the Ascend PyTorch Adapter.

### 8-bit compression

Cut memory usage roughly in half (with slightly reduced quality) by adding `--load-8bit`. It works on CPU, GPU, and Metal backends:

```bash
python3 -m fastchat.serve.cli --model-path lmsys/vicuna-7b-v1.5 --load-8bit
```

With 8-bit, Vicuna-13B fits on a single 16 GB GPU (RTX 3090/4080, T4, V100-16GB, or AMD RX 6800 XT).

To squeeze even further, offload weights that don't fit in VRAM to CPU memory:

```bash
python3 -m fastchat.serve.cli --model-path lmsys/vicuna-7b-v1.5 --load-8bit --cpu-offloading
```

> [!WARNING]
> `--cpu-offloading` requires `--load-8bit` and the `bitsandbytes` package, which is only available on Linux.

### 4-bit and accelerated quantization

FastChat ships flags for several quantization frameworks. Each set of flags is described in the linked docs:

| Framework | Enable flag | Notes |
|---|---|---|
| GPTQ (4-bit) | `--gptq-ckpt`, `--gptq-wbits 4`, `--gptq-groupsize`, `--gptq-act-order` | See `docs/gptq.md` |
| AWQ (4-bit) | `--awq-ckpt`, `--awq-wbits 4`, `--awq-groupsize` | See `docs/awq.md` |
| ExLlama V2 | `--enable-exllama`, `--exllama-max-seq-len`, `--exllama-gpu-split`, `--exllama-cache-8bit` | See `docs/exllama_v2.md` |
| xFasterTransformer | `--enable-xft`, `--xft-max-seq-len`, `--xft-dtype` | CPU only; see `docs/xFasterTransformer.md` |

> [!NOTE]
> `--enable-xft` forces the device to CPU. If you pass another `--device`, FastChat prints `xFasterTransformer now is only support CPUs. Reset device to CPU` and continues on CPU.

### Using ModelScope (for users in China)

Download weights from ModelScope instead of Hugging Face:

```bash
export FASTCHAT_USE_MODELSCOPE=True
python3 -m fastchat.serve.cli --model-path <modelscope-model-id>
```

---

## 5. Serving with a Web GUI

The web UI is a browser chat interface. It needs **three components** working together:

1. **Controller** — coordinates everything and routes requests to workers.
2. **Model worker(s)** — host one or more models.
3. **Gradio web server** — the page users interact with.

Run each in its own terminal:

```bash
# 1. Controller
python3 -m fastchat.serve.controller
```

```bash
# 2. Model worker (loads the model)
python3 -m fastchat.serve.model_worker --model-path lmsys/vicuna-7b-v1.5
```

Wait until the worker prints `Uvicorn running on ...` — at that point it has registered with the controller. Confirm the connection:

```bash
python3 -m fastchat.serve.test_message --model-name vicuna-7b-v1.5
```

```text
USER: Tell me a story with more than 1000 words.
ASSISTANT: Once upon a time...
```

```bash
# 3. Gradio web server (the UI)
python3 -m fastchat.serve.gradio_web_server
```

Open the printed local URL in your browser and start chatting.

> [!TIP]
> If models don't appear in the dropdown, reboot the Gradio web server — it reads the model list from the controller at startup unless you set `--model-list-mode reload`.

### Serving multiple models or higher throughput

Register several workers to one controller. Give each its own GPU and port:

```bash
# Worker 0
CUDA_VISIBLE_DEVICES=0 python3 -m fastchat.serve.model_worker \
  --model-path lmsys/vicuna-7b-v1.5 \
  --controller http://localhost:21001 --port 31000 --worker http://localhost:31000

# Worker 1
CUDA_VISIBLE_DEVICES=1 python3 -m fastchat.serve.model_worker \
  --model-path lmsys/fastchat-t5-3b-v1.0 \
  --controller http://localhost:21001 --port 31001 --worker http://localhost:31001
```

Pointing multiple workers at the **same** model increases throughput; pointing them at **different** models lets one UI serve several models at once.

### Useful web server flags

| Flag | Default | Description |
|---|---|---|
| `--host` | `0.0.0.0` | Interface to bind |
| `--port` | *(auto)* | Port to listen on |
| `--share` | off | Generate a public, shareable Gradio link |
| `--controller-url` | `http://localhost:21001` | Address of the controller |
| `--concurrency-count` | `10` | Size of the Gradio request queue |
| `--model-list-mode` | `once` | `once` loads the model list at startup; `reload` refreshes it every request |
| `--moderate` | off | Enable content moderation to block unsafe inputs |
| `--show-terms-of-use` | off | Show a terms-of-use dialog before loading |
| `--gradio-auth-path` | *(none)* | Path to a file of `user:password` pairs for basic auth |
| `--gradio-root-path` | *(none)* | Root path prefix when running behind a reverse proxy |

---

## 6. The OpenAI-Compatible API Server

FastChat can stand in for the OpenAI API. The server implements Chat Completions, Completions, and Embeddings, so existing OpenAI client code works against it with only a changed base URL.

### Launch the API server

You need the controller and at least one model worker running (see [Section 5](#5-serving-with-a-web-gui)), then:

```bash
python3 -m fastchat.serve.openai_api_server --host localhost --port 8000
```

### Supported endpoints

| Method | Path | Purpose |
|---|---|---|
| `GET` | `/v1/models` | List available models |
| `POST` | `/v1/chat/completions` | Chat completion (messages in, message out) |
| `POST` | `/v1/completions` | Text completion (prompt in, text out) |
| `POST` | `/v1/embeddings` | Generate embeddings |
| `POST` | `/v1/engines/{model_name}/embeddings` | Embeddings (engine-style path) |
| `POST` | `/api/v1/token_check` | Check token counts for a request |

### Call it with the OpenAI Python SDK

Install the OpenAI package (≥ 1.0) and point it at your local server. Use any non-empty string as the key:

```python
import openai

openai.api_key = "EMPTY"
openai.base_url = "http://localhost:8000/v1/"

model = "vicuna-7b-v1.5"

# Text completion
completion = openai.completions.create(model=model, prompt="Once upon a time", max_tokens=64)
print("Once upon a time" + completion.choices[0].text)

# Chat completion
chat = openai.chat.completions.create(
    model=model,
    messages=[{"role": "user", "content": "Hello! What is your name?"}],
)
print(chat.choices[0].message.content)
```

### Call it with cURL

```bash
# List models
curl http://localhost:8000/v1/models

# Chat completion
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "vicuna-7b-v1.5",
    "messages": [{"role": "user", "content": "Hello! What is your name?"}]
  }'

# Text completion
curl http://localhost:8000/v1/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "vicuna-7b-v1.5",
    "prompt": "Once upon a time",
    "max_tokens": 41,
    "temperature": 0.5
  }'

# Embeddings
curl http://localhost:8000/v1/embeddings \
  -H "Content-Type: application/json" \
  -d '{
    "model": "vicuna-7b-v1.5",
    "input": "Hello world!"
  }'
```

### Serving multiple models in one process

Replace the single `model_worker` with `multi_model_worker` to load several models into one process and accelerator:

```bash
python3 -m fastchat.serve.multi_model_worker \
  --model-path lmsys/vicuna-7b-v1.5   --model-names vicuna-7b-v1.5 \
  --model-path lmsys/longchat-7b-16k  --model-names longchat-7b-16k
```

> [!NOTE]
> Multi-model serving in a single process works best with PEFT/LoRA models that share a base model (handled by the `PeftModelAdapter`).

### Tuning timeouts and batch size

These environment variables are read at startup (see [Command & Flag Reference](#15-command--flag-reference) for the full list):

```bash
# A worker is considered timed out after this many seconds (default 100)
export FASTCHAT_WORKER_API_TIMEOUT=600

# Reduce embedding batch size if you hit out-of-memory while embedding (default 4)
export FASTCHAT_WORKER_API_EMBEDDING_BATCH_SIZE=1
```

> [!TIP]
> If the API server sits behind Nginx, disable response buffering for streaming to work: set `proxy_buffering off;` in the proxy location block.

---

## 7. High-Throughput Serving (vLLM, SGLang, and others)

The default `model_worker` is based on Hugging Face Transformers — broad compatibility but lower throughput. For batched, high-throughput serving, swap in an alternative worker. **Each is a drop-in replacement for `model_worker`** — keep the same controller and API server.

| Worker module | Backend | Highlight flags |
|---|---|---|
| `fastchat.serve.vllm_worker` | vLLM | `--gpu_memory_utilization` (default `0.9`), `--num-gpus`, `--trust_remote_code` |
| `fastchat.serve.sglang_worker` | SGLang | `--mem-fraction-static` (default `0.9`), `--num-gpus`, `--multimodal` |
| `fastchat.serve.lightllm_worker` | LightLLM | `--max_total_token_num`, `--tp`, `--running_max_req_size` |
| `fastchat.serve.mlx_worker` | MLX (Apple Silicon) | `--model-path` (default `microsoft/phi-2`) |
| `fastchat.serve.dashinfer_worker` | DashInfer (CPU) | `config_file`, `--model-path` (default `qwen/Qwen-7B-Chat`) |
| `fastchat.serve.huggingface_api_worker` | Remote HF Inference API | `--model-info-file`, `--seed` |

Example — serve Vicuna with vLLM:

```bash
python3 -m fastchat.serve.controller
python3 -m fastchat.serve.vllm_worker --model-path lmsys/vicuna-7b-v1.5
python3 -m fastchat.serve.openai_api_server --host localhost --port 8000
```

> [!WARNING]
> `--gpu_memory_utilization` (vLLM) and `--mem-fraction-static` (SGLang) reserve a fraction of GPU memory for weights, activations, and KV cache. Higher values raise throughput but can trigger out-of-memory errors. Lower the value if a worker crashes on startup.

See `docs/vllm_integration.md`, `docs/lightllm_integration.md`, `docs/dashinfer_integration.md`, and `docs/mlx_integration.md` for backend-specific setup.

---

## 8. Launching Chatbot Arena

Chatbot Arena is the side-by-side "battle" UI where two models answer the same prompt and you vote. To run it locally you use the **multi** web server, `gradio_web_server_multi`.

FastChat can put API-based models (OpenAI, Anthropic, Gemini, Mistral, and more) into the arena. Describe them in a JSON file, for example `api_endpoint.json`:

```json
{
    "gpt-4o-2024-05-13": {
        "model_name": "gpt-4o-2024-05-13",
        "api_base": "https://api.openai.com/v1",
        "api_type": "openai",
        "api_key": "<your key>",
        "anony_only": false
    }
}
```

For Anthropic models set `"api_type": "anthropic_message"`; for Gemini set `"api_type": "gemini"`. Then launch:

```bash
python3 -m fastchat.serve.gradio_web_server_multi --register-api-endpoint-file api_endpoint.json
```

To mix in your own locally served models, launch a controller and workers as in [Section 5](#5-serving-with-a-web-gui) first.

### Arena-specific flags

| Flag | Description |
|---|---|
| `--vision-arena` | Show tabs for the vision (image) arena |
| `--random-questions` | Load random sample questions from a JSON file |
| `--elo-results-file` | Load Elo leaderboard results and plots |
| `--leaderboard-table-file` | Load the leaderboard table |
| `--show-visualizer` | Show the Data Visualizer tab |
| `--password` | Set a password for the web server |
| `--ga-id` | Google Analytics ID |

> [!NOTE]
> The hosted Chatbot Arena enforces input length and rate limits (for example, a 12,000-character input limit and a 50-turn conversation limit). These are tunable when you self-host — see the Admin Guide for the environment variables involved.

---

## 9. Evaluating Models with MT-Bench

MT-Bench is a set of challenging multi-turn questions used to benchmark chat models. FastChat automates grading by prompting a strong judge model (such as GPT-4). The tools live in `fastchat/llm_judge`. Install the evaluation extras first:

```bash
pip3 install "fschat[llm_judge]"
cd fastchat/llm_judge
```

### Step 1 — Generate model answers

For a locally hosted model:

```bash
python3 gen_model_answer.py --model-path lmsys/vicuna-7b-v1.5 --model-id vicuna-7b-v1.5
```

For an API-based model (e.g. GPT-3.5):

```bash
python3 gen_api_answer.py --model gpt-3.5-turbo --bench-name mt_bench --parallel 4
```

### Step 2 — Generate judgments

```bash
python3 gen_judgment.py --model-list vicuna-7b-v1.5 --judge-model gpt-4 --mode single
```

The `--mode` flag selects the grading strategy:

| Mode | Meaning |
|---|---|
| `single` *(default)* | Single-answer grading (each answer scored on its own) |
| `pairwise-baseline` | Compare each model against a baseline (default `gpt-3.5-turbo`) |
| `pairwise-all` | Compare all model pairs against each other |

### Step 3 — Show the results

```bash
python3 show_result.py --bench-name mt_bench --judge-model gpt-4 --mode single
```

### Key evaluation flags

| Flag | Default | Used by | Description |
|---|---|---|---|
| `--bench-name` | `mt_bench` | all | Benchmark question set |
| `--judge-model` | `gpt-4` | judgment, results | The model acting as judge |
| `--baseline-model` | `gpt-3.5-turbo` | judgment, results | Baseline for pairwise modes |
| `--parallel` | `1` | answer gen | Number of concurrent API calls |
| `--num-choices` | `1` | answer gen | Completions to generate per question |
| `--max-new-token` | `1024` | model answers | Max tokens to generate |
| `--first-n` | *(all)* | judgment | Debug: only run the first N judgments |

> [!TIP]
> MT-Bench is the recommended way to benchmark models. The older 80-question Vicuna set lives in the separate [vicuna-blog-eval](https://github.com/lm-sys/vicuna-blog-eval) repository.

---

## 10. Fine-Tuning Your Own Model

FastChat's training code (based on Stanford Alpaca, extended for multi-turn conversations) lives in `fastchat/train`. Install the training extras:

```bash
pip3 install -e ".[train]"
```

### Train Vicuna-7B on local GPUs

The example below trains on 4× A100 (40 GB). Point `--model_name_or_path` at your base Llama weights and `--data_path` at your data:

```bash
torchrun --nproc_per_node=4 --master_port=20001 fastchat/train/train_mem.py \
    --model_name_or_path meta-llama/Llama-2-7b-hf \
    --data_path data/dummy_conversation.json \
    --bf16 True \
    --output_dir output_vicuna \
    --num_train_epochs 3 \
    --per_device_train_batch_size 2 \
    --per_device_eval_batch_size 2 \
    --gradient_accumulation_steps 16 \
    --evaluation_strategy "no" \
    --save_strategy "steps" \
    --save_steps 1200 \
    --save_total_limit 10 \
    --learning_rate 2e-5 \
    --weight_decay 0. \
    --warmup_ratio 0.03 \
    --lr_scheduler_type "cosine" \
    --logging_steps 1 \
    --fsdp "full_shard auto_wrap" \
    --fsdp_transformer_layer_cls_to_wrap 'LlamaDecoderLayer' \
    --tf32 True \
    --model_max_length 2048 \
    --gradient_checkpointing True \
    --lazy_preprocess True
```

Recommended Vicuna-13B hyperparameters (same family applies to 7B with smaller batch):

| Hyperparameter | Value |
|---|---|
| Global batch size | 128 |
| Learning rate | 2e-5 |
| Epochs | 3 |
| Max length | 2048 |
| Weight decay | 0 |

### Training variants

| Script | Use case |
|---|---|
| `train/train_mem.py` | Standard training with FlashAttention memory optimization |
| `train/train_xformers.py` | Memory-efficient attention via xFormers (use on V100, which lacks FlashAttention) |
| `train/train_lora.py` | LoRA fine-tuning (parameter-efficient) |
| `train/train_lora_t5.py` | LoRA for T5 models |
| `train/train_flant5.py` | Fine-tune Flan-T5 |
| `train/train_baichuan.py` | Fine-tune Baichuan |
| `train/train_yuan2.py` | Fine-tune Yuan2 |

> [!TIP]
> Turn on experiment tracking by passing `--report_to wandb` (or `tensorboard`, `mlflow`). See `docs/training.md` for LoRA, FastChat-T5, and other model recipes, and the SkyPilot guide for training on cloud spot instances.

> [!IMPORTANT]
> LMSYS does not release the ShareGPT dataset used to train Vicuna. The repo ships `data/dummy_conversation.json` so you can test the training pipeline; plug your own data into the same format.

---

## 11. Preparing Training Data

The `fastchat/data` package contains a pipeline of small scripts for cleaning and shaping conversation data. They all use `--in-file` / `--out-file` and can be chained. Run the whole pipeline at once with `prepare_all.py`, or invoke steps individually:

| Script | What it does |
|---|---|
| `data/clean_sharegpt.py` | Convert ShareGPT HTML to Markdown and drop invalid samples |
| `data/optional_clean.py` | Filter by language (`--keep-lang`, `--skip-lang`, `--reduce-rep`) |
| `data/split_long_conversation.py` | Split long conversations to fit `--max-length` |
| `data/split_train_test.py` | Split into train/test by `--ratio` (default `0.9`) |
| `data/sample.py` | Take a sample (`--begin`, `--end`, `--max-length`) |
| `data/merge.py` | Merge multiple JSON files into one |
| `data/extract_gpt4_only.py` | Keep only GPT-4 conversations |
| `data/extract_single_round.py` | Keep only single-round conversations |
| `data/get_stats.py` | Compute dataset statistics |
| `data/inspect_data.py` | Inspect random samples |

Example — clean, then split a ShareGPT export:

```bash
python3 -m fastchat.data.clean_sharegpt --in-file sharegpt.json --out-file sharegpt_clean.json
python3 -m fastchat.data.split_long_conversation \
  --in-file sharegpt_clean.json --out-file sharegpt_split.json \
  --model-name-or-path meta-llama/Llama-2-7b-hf --max-length 2048
```

See `docs/commands/data_cleaning.md` for the full recommended sequence.

---

## 12. How FastChat Works

You don't need this section to use FastChat, but it explains why the serving setup has three moving parts. (For the full architecture, security model, and deployment internals, see the **Admin Guide**.)

```text
            ┌──────────────┐
  Browser → │ Gradio Web   │
  / SDK   → │ or OpenAI    │ ──┐
            │ API Server   │   │  "which worker serves model X?"
            └──────────────┘   ▼
                          ┌───────────┐   registration +
                          │ Controller│ ◀ heartbeats
                          └───────────┘
                            │   routes request to a worker
              ┌─────────────┼─────────────┐
              ▼             ▼             ▼
        ┌──────────┐  ┌──────────┐  ┌──────────┐
        │ Worker 0 │  │ Worker 1 │  │ Worker N │   each hosts one or
        │ (model)  │  │ (model)  │  │ (model)  │   more models on a GPU
        └──────────┘  └──────────┘  └──────────┘
```

- **Workers** load models and register with the controller, then send periodic heartbeats so the controller knows they're alive.
- The **controller** keeps a registry of which workers serve which models and hands out a worker address for each request. It can dispatch by **shortest queue** (default) or **lottery**.
- The **web server** and **OpenAI API server** are just two front ends that ask the controller where to send a request.

This separation is what lets you scale out (add workers), serve many models behind one URL, and swap the inference backend (vLLM, SGLang, …) without changing the front end.

---

## 13. Troubleshooting

### Error: `Larger --num-gpus (N) than --gpus ...`

**Cause:** You set `--num-gpus` higher than the count of GPU IDs in `--gpus`.
**Solution:** Make the two match, e.g. `--gpus 0,1 --num-gpus 2`.
**Prevention:** If you only set `--num-gpus`, omit `--gpus` and let FastChat use the first N visible GPUs.

### Error: `Invalid style for console: <style>`

**Cause:** `--style` was given a value other than `simple`, `rich`, or `programmatic`.
**Solution:** Use one of the three supported styles.

### Error: `Invalid device: <device>`

**Cause:** `--device` was set to something other than `cuda`, `cpu`, `mps`, `xpu`, or `npu`.
**Solution:** Pick a supported backend for your hardware.

### Error: `ValueError: Cannot find any model weight files. Please check your (cached) weight path: <path>`

**Cause:** `--model-path` points to a folder with no model weights, or a download was interrupted.
**Solution:** Verify the Hugging Face repo ID, or delete the partial download under `~/.cache/huggingface/hub/` and rerun so it re-downloads.

### Error: `No valid model adapter for <model_path>`

**Cause:** FastChat doesn't recognize the model architecture from its path/config.
**Solution:** Check the supported model list (`docs/model_support.md`) and use a recognized model, or add an adapter as described there.

### CUDA out of memory when loading a model

**Cause:** The model is larger than available VRAM.
**Solution:** Add `--load-8bit`, reduce `--max-gpu-memory`, spread across GPUs with `--num-gpus`, add `--cpu-offloading` (Linux + 8-bit), or use a smaller model.
**Prevention:** Check the memory estimates in [Section 3](#models-you-can-chat-with) before loading.

### Out of memory while creating embeddings

**Cause:** The embedding batch size is too large for your GPU.
**Solution:** `export FASTCHAT_WORKER_API_EMBEDDING_BATCH_SIZE=1` before launching the worker.

### Models don't appear in the web UI dropdown

**Cause:** The Gradio server cached an empty model list at startup, or no worker has registered yet.
**Solution:** Make sure a model worker printed `Uvicorn running on ...`, then reboot the Gradio web server, or launch it with `--model-list-mode reload`.

### Worker requests time out

**Cause:** The model/hardware is slower than the default 100-second worker timeout.
**Solution:** `export FASTCHAT_WORKER_API_TIMEOUT=600` (or higher) before launching.

### The `stop` parameter is ignored via the OpenAI API

**Cause:** A known open issue ([#1048](https://github.com/lm-sys/FastChat/issues/1048)) where the `stop` field is not always honored in chat/completions.
**Solution:** Trim stop sequences client-side as a workaround, and track the upstream issue.

### Streaming hangs behind a reverse proxy

**Cause:** The proxy buffers the streamed response.
**Solution:** Disable buffering — in Nginx set `proxy_buffering off;` in the API server's location block.

### Mac install fails building a dependency

**Cause:** Rust/CMake toolchain missing.
**Solution:** `brew install rust cmake`, then reinstall.

---

## 14. Frequently Asked Questions

**Q: What's the difference between `fschat` and `fastchat`?**
`fschat` is the PyPI package name you install; `fastchat` is the importable Python module you run (`python3 -m fastchat.serve.cli`). Same project.

**Q: Do I need a GPU?**
No. Use `--device cpu` (needs lots of RAM), `--device mps` on Apple Silicon, or `--load-8bit` to fit larger models on small GPUs. A GPU is strongly recommended for usable speed.

**Q: Where are downloaded model weights stored?**
In the Hugging Face cache in your home folder: `~/.cache/huggingface/hub/<model_name>`.

**Q: Can I use my own model?**
Yes, if its architecture has a FastChat adapter. Pass a local folder or HF repo ID to `--model-path`. See `docs/model_support.md` to add a new model.

**Q: How do I serve models faster?**
Replace `model_worker` with `vllm_worker` or `sglang_worker` for batched, high-throughput serving. The controller and API server stay the same.

**Q: How do I run several models at once?**
Either launch one `model_worker` per model (each on its own port/GPU) and register them all to one controller, or use `multi_model_worker` to load several into one process.

**Q: Is the API really OpenAI-compatible?**
Yes — Chat Completions, Completions, and Embeddings are implemented. Point the OpenAI SDK's `base_url` at `http://localhost:8000/v1/` and use any non-empty API key. Some parameters (`logprobs`, `logit_bias`, `presence_penalty`, `frequency_penalty`) are not yet supported.

**Q: How do I require API keys on the server?**
Launch the API server with `--api-keys key1,key2`. Requests must then send a matching key.

**Q: Can I get a public link to share my demo?**
Yes — add `--share` to the Gradio web server to create a temporary public URL.

**Q: What models does the official project release?**
Vicuna (7B/13B/33B, plus 16K variants), LongChat-7B-32K, and FastChat-T5-3B. See [Section 3](#models-you-can-chat-with).

**Q: How do I evaluate a model I trained?**
Generate answers with `gen_model_answer.py`, judge them with `gen_judgment.py` (judge defaults to GPT-4), then summarize with `show_result.py`. See [Section 9](#9-evaluating-models-with-mt-bench).

**Q: Which Python version do I need?**
Python 3.8 or newer.

**Q: Can I use models from ModelScope instead of Hugging Face?**
Yes — `export FASTCHAT_USE_MODELSCOPE=True` before launching.

**Q: How do I save and resume a CLI conversation?**
Type `!!save mychat.json` to save and `!!load mychat.json` to restore.

**Q: Where do I configure content moderation?**
Add `--moderate` to the Gradio web server to block unsafe inputs.

---

## 15. Command & Flag Reference

### Main commands

| Command | Purpose |
|---|---|
| `python3 -m fastchat.serve.cli` | Interactive command-line chat |
| `python3 -m fastchat.serve.controller` | Start the controller |
| `python3 -m fastchat.serve.model_worker` | Start a Transformers model worker |
| `python3 -m fastchat.serve.vllm_worker` | High-throughput vLLM worker |
| `python3 -m fastchat.serve.sglang_worker` | SGLang worker |
| `python3 -m fastchat.serve.multi_model_worker` | Serve multiple models in one process |
| `python3 -m fastchat.serve.openai_api_server` | OpenAI-compatible REST API server |
| `python3 -m fastchat.serve.gradio_web_server` | Single-model web UI |
| `python3 -m fastchat.serve.gradio_web_server_multi` | Multi-model / Chatbot Arena UI |
| `python3 -m fastchat.serve.test_message` | Send a test message to a worker |
| `python3 -m fastchat.serve.huggingface_api` | One-shot generation via local model |

### Shared model/hardware flags (CLI and workers)

| Flag | Default | Description |
|---|---|---|
| `--model-path` | `lmsys/vicuna-7b-v1.5` | Local folder or Hugging Face repo ID |
| `--revision` | `main` | Hugging Face model revision |
| `--device` | `cuda` | `cuda`, `cpu`, `mps`, `xpu`, or `npu` |
| `--gpus` | *(all)* | GPU IDs, e.g. `0,2` |
| `--num-gpus` | `1` | Number of GPUs for model parallelism |
| `--max-gpu-memory` | *(auto)* | Max weight memory per GPU, e.g. `13GiB` |
| `--dtype` | *(auto)* | Override dtype; float16 on GPU, float32 on CPU by default |
| `--load-8bit` | off | 8-bit quantization |
| `--cpu-offloading` | off | Offload excess weights to CPU (needs 8-bit, Linux) |

### Controller flags

| Flag | Default | Description |
|---|---|---|
| `--host` | `localhost` | Bind address |
| `--port` | `21001` | Port |
| `--dispatch-method` | `shortest_queue` | `shortest_queue` or `lottery` |
| `--ssl` | off | Enable SSL (needs `SSL_KEYFILE`, `SSL_CERTFILE` env vars) |

### Model worker flags

| Flag | Default | Description |
|---|---|---|
| `--host` | `localhost` | Bind address |
| `--port` | `21002` | Port |
| `--worker-address` | `http://localhost:21002` | Address other services use to reach this worker |
| `--controller-address` | `http://localhost:21001` | Controller to register with |
| `--model-names` | *(from path)* | Comma-separated display names |
| `--conv-template` | *(auto)* | Conversation prompt template |
| `--limit-worker-concurrency` | `5` | Max concurrent requests (prevents OOM) |
| `--stream-interval` | `2` | Token streaming interval |
| `--no-register` | off | Don't register with the controller |
| `--seed` | *(random)* | Fixed random seed for generation |
| `--ssl` | off | Enable SSL |

### OpenAI API server flags

| Flag | Default | Description |
|---|---|---|
| `--host` | `localhost` | Bind address |
| `--port` | `8000` | Port |
| `--controller-address` | `http://localhost:21001` | Controller to query for workers |
| `--api-keys` | *(none)* | Comma-separated list of accepted API keys |
| `--allow-credentials` | off | Allow CORS credentials |
| `--allowed-origins` | `["*"]` | Allowed CORS origins |
| `--allowed-methods` | `["*"]` | Allowed CORS methods |
| `--allowed-headers` | `["*"]` | Allowed CORS headers |
| `--ssl` | off | Enable SSL |

### Key environment variables

| Variable | Default | Effect |
|---|---|---|
| `FASTCHAT_WORKER_API_TIMEOUT` | `100` | Seconds before a worker request times out |
| `FASTCHAT_WORKER_API_EMBEDDING_BATCH_SIZE` | `4` | Embedding batch size (lower to avoid OOM) |
| `FASTCHAT_USE_MODELSCOPE` | `False` | Download weights from ModelScope instead of Hugging Face |
| `CPU_ISA` | *(none)* | CPU instruction set for accelerated CPU inference, e.g. `amx` |
| `LOGDIR` | `.` | Directory for log files |
| `PEFT_SHARE_BASE_WEIGHTS` | `false` | Share base weights across PEFT models in multi-model serving |

---

_Generated June 2026 from lm-sys/FastChat at 587d5cf. AI-generated documentation. The library is checked by sampling (three repositories so far); this guide was not individually reviewed. Verify against the project before relying on it. Unofficial; not affiliated with or reviewed by the project._