FastChat User Guide

FastChat is an open platform for training, serving, and evaluating large language model (LLM) based chatbots. It powers Chatbot Arena (lmarena.ai) and gives you three things on your own hardware:

• A command-line chat interface for talking to local or downloaded models.
• A distributed multi-model serving system with a web UI and OpenAI-compatible REST APIs, so FastChat can be a drop-in local replacement for the OpenAI API.
• Training and evaluation code for state-of-the-art models such as Vicuna and the MT-Bench benchmark.

This guide shows you how to install FastChat, chat with a model from your terminal, stand up a web UI and an OpenAI-compatible API server, run quantized models on limited hardware, fine-tune your own model, and evaluate models with MT-Bench. Every command, flag, and environment variable below is taken directly from the source code.

Note

The Python package is named fschat (version 0.2.36), but you import and run it as fastchat (e.g. python3 -m fastchat.serve.cli). Both names refer to the same project.

Table of Contents

1. Quick Start
2. Installation
3. Chatting from the Command Line
4. Running on Limited Hardware
5. Serving with a Web GUI
6. The OpenAI-Compatible API Server
7. High-Throughput Serving (vLLM, SGLang, and others)
8. Launching Chatbot Arena
9. Evaluating Models with MT-Bench
10. Fine-Tuning Your Own Model
11. Preparing Training Data
12. How FastChat Works
13. Troubleshooting
14. Frequently Asked Questions
15. Command & Flag Reference

1. Quick Start

If you have a GPU and just want to chat with a model:

pip3 install "fschat[model_worker,webui]"
python3 -m fastchat.serve.cli --model-path lmsys/vicuna-7b-v1.5

The model weights download automatically from Hugging Face on first run and are cached in your home folder (~/.cache/huggingface/hub/). Once loaded, you get an interactive prompt:

USER: Hello! Who are you?
ASSISTANT: I am Vicuna, a language model trained by researchers...

To stand up an OpenAI-compatible API instead, you launch three processes — a controller, a model worker, and the API server:

python3 -m fastchat.serve.controller
python3 -m fastchat.serve.model_worker --model-path lmsys/vicuna-7b-v1.5
python3 -m fastchat.serve.openai_api_server --host localhost --port 8000

You can then call http://localhost:8000/v1/chat/completions exactly like the OpenAI API.

Tip

Run each of the three serving processes in its own terminal (or with a process manager like tmux). They are long-running and talk to each other over local HTTP.

2. Installation

FastChat requires Python 3.8 or newer.

Method 1: Install from PyPI

Serving + Web UI

pip3 install "fschat[model_worker,webui]"

Base only

pip3 install fschat

Method 2: Install from source

Install from source when you want to fine-tune, modify the code, or track the latest commit.

git clone https://github.com/lm-sys/FastChat.git
cd FastChat
pip3 install --upgrade pip   # enable PEP 660 support
pip3 install -e ".[model_worker,webui]"

On a Mac you also need Rust and CMake before installing (required to build the tokenizers and qwen dependencies):

brew install rust cmake

Optional dependency groups

FastChat splits its heavier dependencies into extras. Install only what you need:

You can combine extras in one bracketed list, for example pip3 install -e ".[model_worker,webui,train]".

Important

transformers>=4.31 is required to use the 16K-context Vicuna models. The model_worker extra already pins this, but verify it if you installed transformers separately.

Verify the installation

python3 -c "import fastchat; print('FastChat ready')"

FastChat ready

3. Chatting from the Command Line

The CLI is the fastest way to talk to a model. The entry point is fastchat.serve.cli.

python3 -m fastchat.serve.cli --model-path lmsys/vicuna-7b-v1.5

--model-path accepts either a local folder or a Hugging Face repo ID. The default is lmsys/vicuna-7b-v1.5.

In-chat commands

While chatting, these special inputs control the session:

Sampling and generation options

Choosing a display style

The --style flag controls how output is rendered:

python3 -m fastchat.serve.cli --model-path lmsys/vicuna-7b-v1.5 --style rich

Note

--style rich is experimental. It renders Markdown and non-ASCII content more cleanly but may not behave correctly on every terminal.

Enable multi-line input (use ESC then Enter for a newline) and mouse positioning in rich mode:

python3 -m fastchat.serve.cli --model-path lmsys/vicuna-7b-v1.5 --style rich --multiline --mouse

Other useful CLI flags

Models you can chat with

FastChat supports a wide range of model families, including Llama 2, Vicuna, Alpaca, Baize, ChatGLM, Dolly, Falcon, FastChat-T5, GPT4ALL, Guanaco, MPT, OpenAssistant, OpenChat, RedPajama, StableLM, WizardLM, Yuan2, and more. The official models published by LMSYS:

Tip

Vicuna is based on Llama 2 and must be used under Llama's model license. Weights download automatically from the Hugging Face repos shown above.

4. Running on Limited Hardware

You do not need a data-center GPU to run FastChat. The CLI, web worker, and API worker all share the same hardware flags (defined in fastchat.model.model_adapter), so everything below works for serving too.

Multiple GPUs (model parallelism)

Aggregate memory across GPUs on the same machine:

python3 -m fastchat.serve.cli --model-path lmsys/vicuna-7b-v1.5 --num-gpus 2

If the automatic device mapping does not balance memory well, cap the per-GPU weight memory so more room is left for activations and longer context:

python3 -m fastchat.serve.cli --model-path lmsys/vicuna-7b-v1.5 --num-gpus 2 --max-gpu-memory 8GiB

You can also pin specific GPUs with --gpus:

python3 -m fastchat.serve.cli --model-path lmsys/vicuna-7b-v1.5 --gpus 0,2 --num-gpus 2

Caution

If --num-gpus is larger than the number of GPUs listed in --gpus, FastChat raises ValueError: Larger --num-gpus (N) than --gpus .... Keep the two consistent.

CPU only

Runs without a GPU. Vicuna-7B needs ~30 GB and Vicuna-13B ~60 GB of system RAM:

python3 -m fastchat.serve.cli --model-path lmsys/vicuna-7b-v1.5 --device cpu

Use an Intel AI accelerator (AVX512_BF16 / AMX) to speed up CPU inference:

CPU_ISA=amx python3 -m fastchat.serve.cli --model-path lmsys/vicuna-7b-v1.5 --device cpu

Apple Silicon and other accelerators

The --device flag selects the backend. Valid values are cuda (default), cpu, mps, xpu, and npu.

Apple Silicon / AMD (Metal)

python3 -m fastchat.serve.cli --model-path lmsys/vicuna-7b-v1.5 --device mps --load-8bit

Requires torch >= 2.0. Vicuna-7B runs on a 32 GB M1 MacBook at roughly 1–2 words/second.

Intel XPU (Arc / Data Center GPU)

source /opt/intel/oneapi/setvars.sh
python3 -m fastchat.serve.cli --model-path lmsys/vicuna-7b-v1.5 --device xpu

Requires the Intel Extension for PyTorch. Vicuna-7B runs on an Intel Arc A770 16 GB.

Ascend NPU

source /usr/local/Ascend/ascend-toolkit/set_env.sh
python3 -m fastchat.serve.cli --model-path lmsys/vicuna-7b-v1.5 --device npu

Requires the Ascend PyTorch Adapter.

8-bit compression

Cut memory usage roughly in half (with slightly reduced quality) by adding --load-8bit. It works on CPU, GPU, and Metal backends:

python3 -m fastchat.serve.cli --model-path lmsys/vicuna-7b-v1.5 --load-8bit

With 8-bit, Vicuna-13B fits on a single 16 GB GPU (RTX 3090/4080, T4, V100-16GB, or AMD RX 6800 XT).

To squeeze even further, offload weights that don't fit in VRAM to CPU memory:

python3 -m fastchat.serve.cli --model-path lmsys/vicuna-7b-v1.5 --load-8bit --cpu-offloading

Warning

--cpu-offloading requires --load-8bit and the bitsandbytes package, which is only available on Linux.

4-bit and accelerated quantization

FastChat ships flags for several quantization frameworks. Each set of flags is described in the linked docs:

Note

--enable-xft forces the device to CPU. If you pass another --device, FastChat prints xFasterTransformer now is only support CPUs. Reset device to CPU and continues on CPU.

Using ModelScope (for users in China)

Download weights from ModelScope instead of Hugging Face:

export FASTCHAT_USE_MODELSCOPE=True
python3 -m fastchat.serve.cli --model-path <modelscope-model-id>

5. Serving with a Web GUI

The web UI is a browser chat interface. It needs three components working together:

1. Controller — coordinates everything and routes requests to workers.
2. Model worker(s) — host one or more models.
3. Gradio web server — the page users interact with.

Run each in its own terminal:

# 1. Controller
python3 -m fastchat.serve.controller

# 2. Model worker (loads the model)
python3 -m fastchat.serve.model_worker --model-path lmsys/vicuna-7b-v1.5

Wait until the worker prints Uvicorn running on ... — at that point it has registered with the controller. Confirm the connection:

python3 -m fastchat.serve.test_message --model-name vicuna-7b-v1.5

USER: Tell me a story with more than 1000 words.
ASSISTANT: Once upon a time...

# 3. Gradio web server (the UI)
python3 -m fastchat.serve.gradio_web_server

Open the printed local URL in your browser and start chatting.

Tip

If models don't appear in the dropdown, reboot the Gradio web server — it reads the model list from the controller at startup unless you set --model-list-mode reload.

Serving multiple models or higher throughput

Register several workers to one controller. Give each its own GPU and port:

# Worker 0
CUDA_VISIBLE_DEVICES=0 python3 -m fastchat.serve.model_worker \
  --model-path lmsys/vicuna-7b-v1.5 \
  --controller http://localhost:21001 --port 31000 --worker http://localhost:31000

# Worker 1
CUDA_VISIBLE_DEVICES=1 python3 -m fastchat.serve.model_worker \
  --model-path lmsys/fastchat-t5-3b-v1.0 \
  --controller http://localhost:21001 --port 31001 --worker http://localhost:31001

Pointing multiple workers at the same model increases throughput; pointing them at different models lets one UI serve several models at once.

Useful web server flags

6. The OpenAI-Compatible API Server

FastChat can stand in for the OpenAI API. The server implements Chat Completions, Completions, and Embeddings, so existing OpenAI client code works against it with only a changed base URL.

Launch the API server

You need the controller and at least one model worker running (see Section 5), then:

python3 -m fastchat.serve.openai_api_server --host localhost --port 8000

Supported endpoints

Call it with the OpenAI Python SDK

Install the OpenAI package (≥ 1.0) and point it at your local server. Use any non-empty string as the key:

import openai

openai.api_key = "EMPTY"
openai.base_url = "http://localhost:8000/v1/"

model = "vicuna-7b-v1.5"

# Text completion
completion = openai.completions.create(model=model, prompt="Once upon a time", max_tokens=64)
print("Once upon a time" + completion.choices[0].text)

# Chat completion
chat = openai.chat.completions.create(
    model=model,
    messages=[{"role": "user", "content": "Hello! What is your name?"}],
)
print(chat.choices[0].message.content)

Call it with cURL

# List models
curl http://localhost:8000/v1/models

# Chat completion
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "vicuna-7b-v1.5",
    "messages": [{"role": "user", "content": "Hello! What is your name?"}]
  }'

# Text completion
curl http://localhost:8000/v1/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "vicuna-7b-v1.5",
    "prompt": "Once upon a time",
    "max_tokens": 41,
    "temperature": 0.5
  }'

# Embeddings
curl http://localhost:8000/v1/embeddings \
  -H "Content-Type: application/json" \
  -d '{
    "model": "vicuna-7b-v1.5",
    "input": "Hello world!"
  }'

Serving multiple models in one process

Replace the single model_worker with multi_model_worker to load several models into one process and accelerator:

python3 -m fastchat.serve.multi_model_worker \
  --model-path lmsys/vicuna-7b-v1.5   --model-names vicuna-7b-v1.5 \
  --model-path lmsys/longchat-7b-16k  --model-names longchat-7b-16k

Note

Multi-model serving in a single process works best with PEFT/LoRA models that share a base model (handled by the PeftModelAdapter).

Tuning timeouts and batch size

These environment variables are read at startup (see Command & Flag Reference for the full list):

# A worker is considered timed out after this many seconds (default 100)
export FASTCHAT_WORKER_API_TIMEOUT=600

# Reduce embedding batch size if you hit out-of-memory while embedding (default 4)
export FASTCHAT_WORKER_API_EMBEDDING_BATCH_SIZE=1

Tip

If the API server sits behind Nginx, disable response buffering for streaming to work: set proxy_buffering off; in the proxy location block.

7. High-Throughput Serving (vLLM, SGLang, and others)

The default model_worker is based on Hugging Face Transformers — broad compatibility but lower throughput. For batched, high-throughput serving, swap in an alternative worker. Each is a drop-in replacement for model_worker — keep the same controller and API server.

Example — serve Vicuna with vLLM:

python3 -m fastchat.serve.controller
python3 -m fastchat.serve.vllm_worker --model-path lmsys/vicuna-7b-v1.5
python3 -m fastchat.serve.openai_api_server --host localhost --port 8000

Warning

--gpu_memory_utilization (vLLM) and --mem-fraction-static (SGLang) reserve a fraction of GPU memory for weights, activations, and KV cache. Higher values raise throughput but can trigger out-of-memory errors. Lower the value if a worker crashes on startup.

See docs/vllm_integration.md, docs/lightllm_integration.md, docs/dashinfer_integration.md, and docs/mlx_integration.md for backend-specific setup.

8. Launching Chatbot Arena

Chatbot Arena is the side-by-side "battle" UI where two models answer the same prompt and you vote. To run it locally you use the multi web server, gradio_web_server_multi.

FastChat can put API-based models (OpenAI, Anthropic, Gemini, Mistral, and more) into the arena. Describe them in a JSON file, for example api_endpoint.json:

{
    "gpt-4o-2024-05-13": {
        "model_name": "gpt-4o-2024-05-13",
        "api_base": "https://api.openai.com/v1",
        "api_type": "openai",
        "api_key": "<your key>",
        "anony_only": false
    }
}

For Anthropic models set "api_type": "anthropic_message"; for Gemini set "api_type": "gemini". Then launch:

python3 -m fastchat.serve.gradio_web_server_multi --register-api-endpoint-file api_endpoint.json

To mix in your own locally served models, launch a controller and workers as in Section 5 first.

Arena-specific flags

Note

The hosted Chatbot Arena enforces input length and rate limits (for example, a 12,000-character input limit and a 50-turn conversation limit). These are tunable when you self-host — see the Admin Guide for the environment variables involved.

9. Evaluating Models with MT-Bench

MT-Bench is a set of challenging multi-turn questions used to benchmark chat models. FastChat automates grading by prompting a strong judge model (such as GPT-4). The tools live in fastchat/llm_judge. Install the evaluation extras first:

pip3 install "fschat[llm_judge]"
cd fastchat/llm_judge

Step 1 — Generate model answers

For a locally hosted model:

python3 gen_model_answer.py --model-path lmsys/vicuna-7b-v1.5 --model-id vicuna-7b-v1.5

For an API-based model (e.g. GPT-3.5):

python3 gen_api_answer.py --model gpt-3.5-turbo --bench-name mt_bench --parallel 4

Step 2 — Generate judgments

python3 gen_judgment.py --model-list vicuna-7b-v1.5 --judge-model gpt-4 --mode single

The --mode flag selects the grading strategy:

Step 3 — Show the results

python3 show_result.py --bench-name mt_bench --judge-model gpt-4 --mode single

Key evaluation flags

Tip

MT-Bench is the recommended way to benchmark models. The older 80-question Vicuna set lives in the separate vicuna-blog-eval repository.

10. Fine-Tuning Your Own Model

FastChat's training code (based on Stanford Alpaca, extended for multi-turn conversations) lives in fastchat/train. Install the training extras:

pip3 install -e ".[train]"

Train Vicuna-7B on local GPUs

The example below trains on 4× A100 (40 GB). Point --model_name_or_path at your base Llama weights and --data_path at your data:

torchrun --nproc_per_node=4 --master_port=20001 fastchat/train/train_mem.py \
    --model_name_or_path meta-llama/Llama-2-7b-hf \
    --data_path data/dummy_conversation.json \
    --bf16 True \
    --output_dir output_vicuna \
    --num_train_epochs 3 \
    --per_device_train_batch_size 2 \
    --per_device_eval_batch_size 2 \
    --gradient_accumulation_steps 16 \
    --evaluation_strategy "no" \
    --save_strategy "steps" \
    --save_steps 1200 \
    --save_total_limit 10 \
    --learning_rate 2e-5 \
    --weight_decay 0. \
    --warmup_ratio 0.03 \
    --lr_scheduler_type "cosine" \
    --logging_steps 1 \
    --fsdp "full_shard auto_wrap" \
    --fsdp_transformer_layer_cls_to_wrap 'LlamaDecoderLayer' \
    --tf32 True \
    --model_max_length 2048 \
    --gradient_checkpointing True \
    --lazy_preprocess True

Recommended Vicuna-13B hyperparameters (same family applies to 7B with smaller batch):

Training variants

Tip

Turn on experiment tracking by passing --report_to wandb (or tensorboard, mlflow). See docs/training.md for LoRA, FastChat-T5, and other model recipes, and the SkyPilot guide for training on cloud spot instances.

Important

LMSYS does not release the ShareGPT dataset used to train Vicuna. The repo ships data/dummy_conversation.json so you can test the training pipeline; plug your own data into the same format.

11. Preparing Training Data

The fastchat/data package contains a pipeline of small scripts for cleaning and shaping conversation data. They all use --in-file / --out-file and can be chained. Run the whole pipeline at once with prepare_all.py, or invoke steps individually:

Example — clean, then split a ShareGPT export:

python3 -m fastchat.data.clean_sharegpt --in-file sharegpt.json --out-file sharegpt_clean.json
python3 -m fastchat.data.split_long_conversation \
  --in-file sharegpt_clean.json --out-file sharegpt_split.json \
  --model-name-or-path meta-llama/Llama-2-7b-hf --max-length 2048

See docs/commands/data_cleaning.md for the full recommended sequence.

12. How FastChat Works

You don't need this section to use FastChat, but it explains why the serving setup has three moving parts. (For the full architecture, security model, and deployment internals, see the Admin Guide.)

            ┌──────────────┐
  Browser → │ Gradio Web   │
  / SDK   → │ or OpenAI    │ ──┐
            │ API Server   │   │  "which worker serves model X?"
            └──────────────┘   ▼
                          ┌───────────┐   registration +
                          │ Controller│ ◀ heartbeats
                          └───────────┘
                            │   routes request to a worker
              ┌─────────────┼─────────────┐
              ▼             ▼             ▼
        ┌──────────┐  ┌──────────┐  ┌──────────┐
        │ Worker 0 │  │ Worker 1 │  │ Worker N │   each hosts one or
        │ (model)  │  │ (model)  │  │ (model)  │   more models on a GPU
        └──────────┘  └──────────┘  └──────────┘

• Workers load models and register with the controller, then send periodic heartbeats so the controller knows they're alive.
• The controller keeps a registry of which workers serve which models and hands out a worker address for each request. It can dispatch by shortest queue (default) or lottery.
• The web server and OpenAI API server are just two front ends that ask the controller where to send a request.

This separation is what lets you scale out (add workers), serve many models behind one URL, and swap the inference backend (vLLM, SGLang, …) without changing the front end.

13. Troubleshooting

Error: Larger --num-gpus (N) than --gpus ...

Cause: You set --num-gpus higher than the count of GPU IDs in --gpus. Solution: Make the two match, e.g. --gpus 0,1 --num-gpus 2. Prevention: If you only set --num-gpus, omit --gpus and let FastChat use the first N visible GPUs.

Error: Invalid style for console: <style>

Cause: --style was given a value other than simple, rich, or programmatic. Solution: Use one of the three supported styles.

Error: Invalid device: <device>

Cause: --device was set to something other than cuda, cpu, mps, xpu, or npu. Solution: Pick a supported backend for your hardware.

Error: ValueError: Cannot find any model weight files. Please check your (cached) weight path: <path>

Cause: --model-path points to a folder with no model weights, or a download was interrupted. Solution: Verify the Hugging Face repo ID, or delete the partial download under ~/.cache/huggingface/hub/ and rerun so it re-downloads.

Error: No valid model adapter for <model_path>

Cause: FastChat doesn't recognize the model architecture from its path/config. Solution: Check the supported model list (docs/model_support.md) and use a recognized model, or add an adapter as described there.

CUDA out of memory when loading a model

Cause: The model is larger than available VRAM. Solution: Add --load-8bit, reduce --max-gpu-memory, spread across GPUs with --num-gpus, add --cpu-offloading (Linux + 8-bit), or use a smaller model. Prevention: Check the memory estimates in Section 3 before loading.

Out of memory while creating embeddings

Cause: The embedding batch size is too large for your GPU. Solution: export FASTCHAT_WORKER_API_EMBEDDING_BATCH_SIZE=1 before launching the worker.

Models don't appear in the web UI dropdown

Cause: The Gradio server cached an empty model list at startup, or no worker has registered yet. Solution: Make sure a model worker printed Uvicorn running on ..., then reboot the Gradio web server, or launch it with --model-list-mode reload.

Worker requests time out

Cause: The model/hardware is slower than the default 100-second worker timeout. Solution: export FASTCHAT_WORKER_API_TIMEOUT=600 (or higher) before launching.

The stop parameter is ignored via the OpenAI API

Cause: A known open issue (#1048) where the stop field is not always honored in chat/completions. Solution: Trim stop sequences client-side as a workaround, and track the upstream issue.

Streaming hangs behind a reverse proxy

Cause: The proxy buffers the streamed response. Solution: Disable buffering — in Nginx set proxy_buffering off; in the API server's location block.

Mac install fails building a dependency

Cause: Rust/CMake toolchain missing. Solution: brew install rust cmake, then reinstall.

14. Frequently Asked Questions

Q: What's the difference between fschat and fastchat? fschat is the PyPI package name you install; fastchat is the importable Python module you run (python3 -m fastchat.serve.cli). Same project.

Q: Do I need a GPU? No. Use --device cpu (needs lots of RAM), --device mps on Apple Silicon, or --load-8bit to fit larger models on small GPUs. A GPU is strongly recommended for usable speed.

Q: Where are downloaded model weights stored? In the Hugging Face cache in your home folder: ~/.cache/huggingface/hub/<model_name>.

Q: Can I use my own model? Yes, if its architecture has a FastChat adapter. Pass a local folder or HF repo ID to --model-path. See docs/model_support.md to add a new model.

Q: How do I serve models faster? Replace model_worker with vllm_worker or sglang_worker for batched, high-throughput serving. The controller and API server stay the same.

Q: How do I run several models at once? Either launch one model_worker per model (each on its own port/GPU) and register them all to one controller, or use multi_model_worker to load several into one process.

Q: Is the API really OpenAI-compatible? Yes — Chat Completions, Completions, and Embeddings are implemented. Point the OpenAI SDK's base_url at http://localhost:8000/v1/ and use any non-empty API key. Some parameters (logprobs, logit_bias, presence_penalty, frequency_penalty) are not yet supported.

Q: How do I require API keys on the server? Launch the API server with --api-keys key1,key2. Requests must then send a matching key.

Q: Can I get a public link to share my demo? Yes — add --share to the Gradio web server to create a temporary public URL.

Q: What models does the official project release? Vicuna (7B/13B/33B, plus 16K variants), LongChat-7B-32K, and FastChat-T5-3B. See Section 3.

Q: How do I evaluate a model I trained? Generate answers with gen_model_answer.py, judge them with gen_judgment.py (judge defaults to GPT-4), then summarize with show_result.py. See Section 9.

Q: Which Python version do I need? Python 3.8 or newer.

Q: Can I use models from ModelScope instead of Hugging Face? Yes — export FASTCHAT_USE_MODELSCOPE=True before launching.

Q: How do I save and resume a CLI conversation? Type !!save mychat.json to save and !!load mychat.json to restore.

Q: Where do I configure content moderation? Add --moderate to the Gradio web server to block unsafe inputs.

15. Command & Flag Reference

Main commands

Shared model/hardware flags (CLI and workers)

Controller flags

Model worker flags

OpenAI API server flags

Key environment variables