# LLaMA Factory User Guide LLaMA Factory is a unified framework for efficiently fine-tuning over 100 large language models. Whether you need supervised fine-tuning, preference optimization, reward modeling, or continual pretraining, LLaMA Factory provides a single CLI and web interface that handles it all — including multimodal models with image, video, and audio support. --- ## Table of Contents - [1. Quick Start](#1-quick-start) - [2. Installation](#2-installation) - [3. Configuration](#3-configuration) - [4. Core Usage](#4-core-usage) - [5. Common Tasks](#5-common-tasks) - [6. Troubleshooting](#6-troubleshooting) - [7. FAQ](#7-faq) --- ## 1. Quick Start Fine-tune a model in under five minutes. This example runs supervised fine-tuning (SFT) with LoRA on a single GPU. **Step 1 — Install:** ```bash pip install llamafactory ``` **Step 2 — Create a YAML config** (`my_sft.yaml`): ```yaml model_name_or_path: meta-llama/Llama-2-7b-hf template: llama2 dataset: alpaca_en finetuning_type: lora lora_rank: 8 lora_target: all output_dir: ./output/llama2_sft per_device_train_batch_size: 4 gradient_accumulation_steps: 4 learning_rate: 5.0e-5 num_train_epochs: 3 bf16: true logging_steps: 10 save_steps: 500 plot_loss: true report_to: none ``` **Step 3 — Train:** ```bash llamafactory-cli train my_sft.yaml ``` **Step 4 — Chat with your model:** ```bash llamafactory-cli chat my_sft.yaml ``` > [!TIP] > The shortcut `lmf` is a drop-in alias for `llamafactory-cli`. Every command shown in this guide works with either name: `lmf train my_sft.yaml` is identical to `llamafactory-cli train my_sft.yaml`. --- ## 2. Installation ### 2.1 Requirements | Requirement | Minimum | |---|---| | Python | 3.8+ | | PyTorch | 2.0+ (CUDA recommended) | | GPU VRAM | 8 GB (QLoRA), 16 GB+ (LoRA), 40 GB+ (full fine-tuning) | | Disk | Varies by model size; plan for model weights + checkpoints | ### 2.2 pip Install ```bash pip install llamafactory ``` For the latest development version: ```bash git clone https://github.com/hiyouga/LLaMA-Factory.git cd LLaMA-Factory pip install -e . ``` ### 2.3 Docker Pre-built Dockerfiles are provided in the `docker/` directory for three hardware platforms: | Platform | Dockerfile | |---|---| | NVIDIA CUDA | `docker/docker-cuda/Dockerfile` | | AMD ROCm | `docker/docker-rocm/Dockerfile` | | Ascend NPU | `docker/docker-npu/Dockerfile` | Build and run example (CUDA): ```bash cd docker/docker-cuda docker build -t llamafactory:cuda . docker run --gpus all -it llamafactory:cuda bash ``` ### 2.4 Verify Installation ```bash llamafactory-cli env ``` This prints your Python version, PyTorch version, CUDA availability, and installed optional dependencies. Use this output when filing bug reports. ```bash llamafactory-cli version ``` Prints the installed LLaMA Factory version. > [!NOTE] > Set `DISABLE_VERSION_CHECK=1` to skip the automatic version check on startup. This is useful in air-gapped environments or CI pipelines. --- ## 3. Configuration LLaMA Factory uses YAML configuration files (or JSON) for all training, inference, and export operations. Every parameter can also be overridden on the command line. ### 3.1 Configuration File Basics A config file is a flat YAML dictionary. There is no nesting — every parameter is a top-level key: ```yaml # Model model_name_or_path: meta-llama/Llama-2-7b-hf template: llama2 trust_remote_code: false # Data dataset: alpaca_en cutoff_len: 1024 max_samples: 10000 # Training stage: sft finetuning_type: lora output_dir: ./output/my_experiment per_device_train_batch_size: 4 gradient_accumulation_steps: 4 learning_rate: 5.0e-5 num_train_epochs: 3 lr_scheduler_type: cosine warmup_ratio: 0.1 bf16: true # Logging logging_steps: 10 save_steps: 500 plot_loss: true report_to: none ``` ### 3.2 CLI Overrides Any YAML parameter can be overridden directly on the command line: ```bash llamafactory-cli train my_config.yaml --learning_rate 1e-4 --num_train_epochs 5 ``` ### 3.3 Key Parameters Reference | Parameter | Values | Purpose | |---|---|---| | `model_name_or_path` | HuggingFace ID or local path | Which model to use | | `template` | `llama2`, `llama3`, `chatglm3`, `qwen`, etc. | Chat template format | | `stage` | `pt`, `sft`, `dpo`, `kto`, `rm` | Training stage | | `finetuning_type` | `full`, `lora` | Full fine-tuning or LoRA | | `dataset` | Comma-separated names | Datasets from `data/dataset_info.json` | | `cutoff_len` | Integer (e.g., `1024`, `2048`) | Max sequence length | | `output_dir` | Path | Where to save checkpoints | | `per_device_train_batch_size` | Integer | Batch size per GPU | | `gradient_accumulation_steps` | Integer | Effective batch multiplier | | `learning_rate` | Float | Learning rate | | `num_train_epochs` | Float | Number of training epochs | | `lr_scheduler_type` | `cosine`, `linear`, etc. | LR schedule | | `warmup_ratio` | Float (0.0–1.0) | Warmup proportion | | `bf16` | `true`/`false` | BFloat16 mixed precision | | `deepspeed` | Path to JSON | DeepSpeed config file | | `report_to` | `none`, `wandb`, `tensorboard`, `swanlab`, `mlflow` | Experiment tracking | | `plot_loss` | `true`/`false` | Generate loss plot after training | > For the complete parameter reference and architecture details, see the **LLaMA Factory Admin Guide**. ### 3.4 LoRA Configuration LoRA (Low-Rank Adaptation) is the most common fine-tuning method — it trains a small number of adapter parameters while freezing the base model. ```yaml finetuning_type: lora lora_rank: 8 lora_target: all ``` | Parameter | Purpose | |---|---| | `lora_rank` | Rank of the LoRA matrices (higher = more parameters, more capacity) | | `lora_target` | Which layers to apply LoRA to; `all` targets every linear layer | **QLoRA** adds 4-bit quantization to dramatically reduce VRAM: ```yaml finetuning_type: lora lora_rank: 8 lora_target: all quantization_bit: 4 quantization_method: bnb ``` ### 3.5 Training Stages | Stage | Config Value | What It Does | |---|---|---| | Continual Pretraining | `stage: pt` | Extend a model's knowledge with new text data | | Supervised Fine-Tuning | `stage: sft` | Train on instruction–response pairs | | Direct Preference Optimization | `stage: dpo` | Align model using preference pairs (chosen/rejected) | | Kahneman-Tversky Optimization | `stage: kto` | Binary feedback alignment (good/bad, no pairs needed) | | Reward Modeling | `stage: rm` | Train a reward model for RLHF | ### 3.6 V1 Training Engine LLaMA Factory includes a new v1 training engine with a plugin-based architecture (DataEngine, ModelEngine, BaseTrainer). Enable it with an environment variable: ```bash USE_V1=1 llamafactory-cli train my_config.yaml ``` The v1 engine uses YAML config files parsed with OmegaConf and offers a more modular, extensible design. It is opt-in and does not affect existing workflows unless explicitly enabled. ### 3.7 Alternative Model Hubs By default, models are downloaded from Hugging Face. Two alternative hubs are supported: ```bash # Use ModelScope (popular in China) USE_MODELSCOPE_HUB=1 llamafactory-cli train my_config.yaml # Use OpenMind Hub USE_OPENMIND_HUB=1 llamafactory-cli train my_config.yaml ``` --- ## 4. Core Usage ### 4.1 CLI Commands LLaMA Factory provides a single entry point with subcommands: | Command | Purpose | |---|---| | `llamafactory-cli train ` | Train or fine-tune a model | | `llamafactory-cli chat ` | Interactive CLI chat | | `llamafactory-cli api ` | Launch OpenAI-compatible API server | | `llamafactory-cli export ` | Merge LoRA adapters and/or export model | | `llamafactory-cli webchat ` | Browser-based chat demo | | `llamafactory-cli webui` | Launch LlamaBoard (Gradio web UI) | | `llamafactory-cli env` | Print environment info | | `llamafactory-cli version` | Print version | ### 4.2 Training #### Single-GPU Training ```bash llamafactory-cli train examples/train_lora/llama3_lora_sft.yaml ``` LLaMA Factory auto-detects available GPUs. With one GPU, it runs standard single-process training. #### Multi-GPU Training When multiple GPUs are detected, LLaMA Factory automatically launches `torchrun` for distributed data parallel (DDP) training. No manual configuration is needed for the common case. To force distributed mode even on a single GPU (useful for testing): ```bash FORCE_TORCHRUN=1 llamafactory-cli train my_config.yaml ``` Control GPU allocation and distributed settings with environment variables: | Env Var | Purpose | Default | |---|---|---| | `NPROC_PER_NODE` | Number of processes per node | Auto (all GPUs) | | `NNODES` | Total number of nodes | `1` | | `NODE_RANK` | Rank of this node | `0` | | `MASTER_ADDR` | Address of rank-0 node | `localhost` | | `MASTER_PORT` | Port for rank-0 communication | Auto | | `OPTIM_TORCH=1` | Enable DDP optimizations | Off | For elastic launch (fault-tolerant multi-node training): | Env Var | Purpose | |---|---| | `RDZV_ID` | Rendezvous ID (shared across nodes) | | `MIN_NNODES` | Minimum nodes to start training | | `MAX_NNODES` | Maximum nodes allowed | | `MAX_RESTARTS` | Maximum worker restart attempts | #### DeepSpeed Integration For large models that exceed single-GPU memory, use DeepSpeed ZeRO: ```yaml # Add to your training config deepspeed: examples/deepspeed/ds_z2_config.json ``` Pre-built DeepSpeed configs are included: | Config | Use Case | |---|---| | ZeRO Stage 0 | Baseline (no sharding) | | ZeRO Stage 2 | Optimizer + gradient sharding | | ZeRO Stage 3 | Full model sharding (largest models) | | ZeRO Stage 3 + Offload | Offload to CPU/NVMe for extreme memory savings | > [!IMPORTANT] > DeepSpeed ZeRO Stage 3 changes model loading behavior significantly. Some adapter configurations may not be compatible. Test with a small `max_samples` value before committing to a full training run. ### 4.3 Inference and Chat #### CLI Chat ```bash llamafactory-cli chat my_config.yaml ``` Opens an interactive terminal session. Type messages and receive model responses. The config must specify `model_name_or_path` and `template` at minimum. To chat with a LoRA-adapted model, include the adapter path in your config: ```yaml model_name_or_path: meta-llama/Llama-2-7b-hf adapter_name_or_path: ./output/llama2_sft template: llama2 finetuning_type: lora ``` #### Web Chat ```bash llamafactory-cli webchat my_config.yaml ``` Launches a browser-based chat interface. #### Inference Backends LLaMA Factory supports multiple inference backends for serving and chat: | Backend | Config Value | Best For | |---|---|---| | Hugging Face Transformers | `huggingface` | Default, maximum compatibility | | vLLM | `vllm` | High-throughput production serving | | SGLang | `sglang` | Structured generation, advanced batching | | KTransformers | `ktransformers` | Specific optimization scenarios | Specify the backend in your inference config: ```yaml model_name_or_path: meta-llama/Llama-2-7b-hf template: llama2 infer_backend: vllm ``` ### 4.4 API Server Launch an OpenAI-compatible API server: ```bash llamafactory-cli api my_inference_config.yaml ``` The server exposes these endpoints: | Endpoint | Method | Purpose | |---|---|---| | `/v1/models` | GET | List available models | | `/v1/chat/completions` | POST | Chat completion (streaming supported) | | `/v1/score/evaluation` | POST | Score evaluation (reward models) | Configure the server with environment variables: | Env Var | Default | Purpose | |---|---|---| | `API_PORT` | `8000` | Server port | | `API_HOST` | `0.0.0.0` | Bind address | | `API_KEY` | (none) | Authentication key | | `API_MODEL_NAME` | `gpt-3.5-turbo` | Model name reported to clients | | `API_VERBOSE` | (off) | Enable verbose request/response logging | | `FASTAPI_ROOT_PATH` | (none) | Root path for reverse proxy setups | | `ALLOW_LOCAL_FILES` | (off) | Allow file:// references in requests | | `SAFE_MEDIA_PATH` | (none) | Restrict local file access to this directory | > [!WARNING] > Setting `ALLOW_LOCAL_FILES=1` permits the API to read local files referenced in requests. In production, always set `SAFE_MEDIA_PATH` to restrict file access to a specific directory to prevent path traversal attacks. Example client call (using `curl`): ```bash curl http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -H "Authorization: Bearer your-api-key" \ -d '{ "model": "gpt-3.5-turbo", "messages": [{"role": "user", "content": "Hello!"}], "stream": false }' ``` ### 4.5 LlamaBoard (Web UI) LlamaBoard is a Gradio-based web UI that exposes the full training and inference pipeline in a browser: ```bash llamafactory-cli webui ``` This launches a local Gradio server. Configure it with environment variables: | Env Var | Purpose | |---|---| | `GRADIO_SHARE` | Set to `1` to create a public Gradio share link | | `GRADIO_IPV6` | Set to `1` to enable IPv6 binding | | `GRADIO_SERVER_NAME` | Override the default server bind address | LlamaBoard provides a point-and-click interface for: - Selecting models and datasets - Configuring training parameters (stage, LoRA rank, learning rate, etc.) - Starting, monitoring, and stopping training runs - Chatting with trained models - Exporting and merging adapters > [!TIP] > LlamaBoard is the fastest way to experiment with different training configurations. Once you find settings that work, export them as a YAML file for reproducible CLI-based training runs. --- ## 5. Common Tasks ### 5.1 Supervised Fine-Tuning with LoRA The most common use case: teach a model to follow instructions using your own data. **Config** (`sft_lora.yaml`): ```yaml model_name_or_path: meta-llama/Llama-2-7b-hf template: llama2 stage: sft finetuning_type: lora lora_rank: 8 lora_target: all dataset: alpaca_en cutoff_len: 1024 output_dir: ./output/llama2_lora_sft per_device_train_batch_size: 4 gradient_accumulation_steps: 4 learning_rate: 5.0e-5 num_train_epochs: 3 lr_scheduler_type: cosine warmup_ratio: 0.1 bf16: true logging_steps: 10 save_steps: 500 plot_loss: true report_to: none ``` ```bash llamafactory-cli train sft_lora.yaml ``` ### 5.2 QLoRA Fine-Tuning (Low VRAM) Run fine-tuning on consumer GPUs (8 GB+) by combining LoRA with 4-bit quantization: ```yaml model_name_or_path: meta-llama/Llama-2-7b-hf template: llama2 stage: sft finetuning_type: lora lora_rank: 8 lora_target: all quantization_bit: 4 quantization_method: bnb dataset: alpaca_en cutoff_len: 1024 output_dir: ./output/llama2_qlora_sft per_device_train_batch_size: 2 gradient_accumulation_steps: 8 learning_rate: 1.0e-4 num_train_epochs: 3 bf16: true logging_steps: 10 save_steps: 500 report_to: none ``` ### 5.3 Full Fine-Tuning For maximum quality when you have sufficient VRAM (40 GB+ per GPU, or use DeepSpeed): ```yaml model_name_or_path: meta-llama/Llama-2-7b-hf template: llama2 stage: sft finetuning_type: full dataset: alpaca_en cutoff_len: 1024 output_dir: ./output/llama2_full_sft per_device_train_batch_size: 2 gradient_accumulation_steps: 8 learning_rate: 1.0e-5 num_train_epochs: 3 bf16: true deepspeed: examples/deepspeed/ds_z2_config.json logging_steps: 10 save_steps: 500 report_to: none ``` ### 5.4 DPO (Direct Preference Optimization) Align a model using human preference data (chosen/rejected pairs). Start from an SFT checkpoint: ```yaml model_name_or_path: ./output/llama2_lora_sft template: llama2 stage: dpo finetuning_type: lora lora_rank: 8 lora_target: all pref_beta: 0.1 pref_loss: sigmoid dataset: dpo_en_demo cutoff_len: 1024 output_dir: ./output/llama2_dpo per_device_train_batch_size: 2 gradient_accumulation_steps: 4 learning_rate: 5.0e-6 num_train_epochs: 1 bf16: true logging_steps: 10 save_steps: 500 report_to: none ``` DPO loss variants: | `pref_loss` | Description | |---|---| | `sigmoid` | Standard DPO loss | | `orpo` | Odds Ratio Preference Optimization | | `simpo` | Simple Preference Optimization | ### 5.5 KTO (Kahneman-Tversky Optimization) KTO uses binary feedback (good/bad) instead of preference pairs, which is often easier to collect: ```yaml model_name_or_path: meta-llama/Llama-2-7b-hf template: llama2 stage: kto finetuning_type: lora lora_rank: 8 lora_target: all dataset: kto_en_demo cutoff_len: 1024 output_dir: ./output/llama2_kto per_device_train_batch_size: 2 gradient_accumulation_steps: 4 learning_rate: 5.0e-6 num_train_epochs: 1 bf16: true logging_steps: 10 save_steps: 500 report_to: none ``` ### 5.6 Reward Modeling Train a reward model for use in RLHF or best-of-n sampling: ```yaml model_name_or_path: meta-llama/Llama-2-7b-hf template: llama2 stage: rm finetuning_type: lora lora_rank: 8 lora_target: all dataset: dpo_en_demo cutoff_len: 1024 output_dir: ./output/llama2_reward per_device_train_batch_size: 2 gradient_accumulation_steps: 4 learning_rate: 1.0e-5 num_train_epochs: 1 bf16: true logging_steps: 10 save_steps: 500 report_to: none ``` ### 5.7 Continual Pretraining Extend a model's knowledge with domain-specific text: ```yaml model_name_or_path: meta-llama/Llama-2-7b-hf template: llama2 stage: pt finetuning_type: lora lora_rank: 8 lora_target: all dataset: c4_demo cutoff_len: 1024 output_dir: ./output/llama2_pretrain per_device_train_batch_size: 4 gradient_accumulation_steps: 4 learning_rate: 5.0e-5 num_train_epochs: 1 bf16: true logging_steps: 10 save_steps: 500 report_to: none ``` ### 5.8 Merging LoRA Adapters and Exporting After training with LoRA, merge the adapter weights back into the base model for deployment: ```yaml model_name_or_path: meta-llama/Llama-2-7b-hf adapter_name_or_path: ./output/llama2_lora_sft template: llama2 finetuning_type: lora export_dir: ./output/llama2_merged export_size: 5 export_device: auto ``` ```bash llamafactory-cli export merge_config.yaml ``` > [!CAUTION] > Do NOT use a quantized base model when merging LoRA adapters. The merge must operate on the full-precision base model. Using a quantized model will produce corrupted outputs. #### GPTQ Quantized Export Export a merged model with GPTQ quantization for efficient inference: ```yaml model_name_or_path: meta-llama/Llama-2-7b-hf adapter_name_or_path: ./output/llama2_lora_sft template: llama2 finetuning_type: lora export_dir: ./output/llama2_gptq export_quantization_bit: 4 export_quantization_dataset: alpaca_en ``` ### 5.9 Using Custom Datasets LLaMA Factory ships with 70+ pre-configured datasets in `data/dataset_info.json`. To add your own, create entries in that file or a custom dataset info JSON. #### Alpaca Format For instruction-following data (SFT, pretraining): ```json [ { "instruction": "Summarize the following text.", "input": "LLaMA Factory is a unified framework...", "output": "LLaMA Factory unifies LLM fine-tuning across 100+ models." } ] ``` #### ShareGPT Format For multi-turn conversation data: ```json [ { "conversations": [ {"from": "human", "value": "What is LLaMA Factory?"}, {"from": "gpt", "value": "LLaMA Factory is a unified fine-tuning framework."}, {"from": "human", "value": "What methods does it support?"}, {"from": "gpt", "value": "It supports LoRA, QLoRA, full fine-tuning, DPO, KTO, and more."} ] } ] ``` #### Identity Dataset For teaching a model its own identity, use the `{{name}}` and `{{author}}` template variables in your dataset. These are replaced at data loading time. #### Registering a Custom Dataset Add an entry to `data/dataset_info.json`: ```json { "my_custom_dataset": { "file_name": "my_data.json", "formatting": "alpaca" } } ``` Then reference it by name in your config: ```yaml dataset: my_custom_dataset ``` Multiple datasets can be combined with commas: ```yaml dataset: alpaca_en,my_custom_dataset ``` ### 5.10 Multimodal Fine-Tuning LLaMA Factory supports fine-tuning multimodal models that handle images, video, and audio. Multimodal data uses special placeholders in messages: | Placeholder | Media Type | |---|---| | `` | Image input | | `