# LLaMA Factory Administrator & Developer Guide

LLaMA Factory is a unified framework for efficiently fine-tuning 100+ large language models. This guide covers architecture, configuration, deployment, security, and developer internals for administrators, operators, and contributors working with the system at a production level.

For everyday usage, CLI commands, and getting-started workflows, see the **User Guide**.

---

## 1. Architecture Overview

### 1.1 Dual-Engine Design

LLaMA Factory ships two complete training engines selectable at runtime. Both share the same CLI entry point (`llamafactory-cli` / `lmf`) and configuration surface, but differ fundamentally in extensibility and internal design.

| Aspect | Legacy Engine (default) | V1 Engine (`USE_V1=1`) |
|---|---|---|
| Entry point | `cli.py` → `launcher.py` | `cli.py` → `v1/launcher.py` |
| Training loop | HuggingFace Trainer subclasses | Custom `BaseTrainer.fit()` with micro-batching |
| Extensibility | Monolithic, modify source | Plugin registry (`BasePlugin` pattern) |
| Distributed | Accelerate / DeepSpeed via HF | Native DeepSpeed, FSDP2, DDP |
| Supported stages | SFT, DPO, KTO, PPO, RM, PT | SFT, DPO, RM |
| Maturity | Production-proven | Newer, plugin-oriented |

> [!IMPORTANT]
> The V1 engine is activated exclusively via the `USE_V1=1` environment variable. There is no YAML flag or CLI switch. All V1-specific configuration (plugin configs, batching strategies) is ignored when V1 is not active.

### 1.2 Legacy Engine Module Map

```
src/llamafactory/
├── cli.py                  # Entry point, dispatches commands
├── launcher.py             # Command dispatch, torchrun auto-launch
├── train/
│   └── tuner.py            # Stage router → per-stage workflows
│       ├── sft/            # Supervised fine-tuning
│       ├── dpo/            # Direct preference optimization
│       ├── kto/            # KTO training
│       ├── ppo/            # Proximal policy optimization
│       ├── rm/             # Reward modeling
│       └── pt/             # Pretraining / continued pretraining
├── model/                  # Model loading, adapter mgmt, quantization
├── data/                   # Data loading, template system, multimodal, tool formatting
├── hparams/                # Argument dataclasses
├── chat/                   # ChatModel (HF, vLLM, SGLang, KTransformers engines)
├── webui/                  # Gradio-based LlamaBoard
├── api/                    # FastAPI OpenAI-compatible server
├── eval/                   # Evaluation (deprecated)
└── extras/                 # Constants, env detection, logging, misc, package checks
```

### 1.3 V1 Engine Module Map

```
src/llamafactory/v1/
├── launcher.py             # V1 command dispatch
├── core/
│   ├── data_engine.py      # torch Dataset from YAML/HF Hub/local
│   ├── model_engine.py     # Model init, PEFT, quantization, kernels
│   ├── base_trainer.py     # Training loop with micro-batch accumulation
│   ├── base_sampler.py     # Inference engine
│   └── utils/
│       ├── batching.py     # BatchGenerator with strategies
│       ├── checkpoint.py   # Distributed checkpoint save/resume
│       └── rendering.py    # ChatML message rendering
├── config/                 # OmegaConf + HfArgumentParser arg parsing
├── plugins/
│   ├── data_plugins/       # Data loading + conversion (alpaca, pair)
│   ├── model_plugins/      # Init, PEFT (LoRA), quant, kernels, templates, parallelization
│   ├── trainer_plugins/    # Optimizer, LR scheduler, distributed (DeepSpeed, FSDP2, Hub)
│   └── sampler_plugins/    # vLLM sampler
├── trainers/               # SFTTrainer, DPOTrainer, RMTrainer
├── samplers/               # CLISampler
├── accelerator/            # DistributedInterface, profiler
└── utils/                  # Types, logging, plugin system, helpers, constants
```

### 1.4 Command Dispatch Flow

```
llamafactory-cli <command>
        │
        ▼
    cli.py ─── USE_V1=1? ───► v1/launcher.py
        │ no                       │
        ▼                          ▼
    launcher.py              V1 dispatch
        │                    (train/chat)
        ▼
    Dispatch table:
      train   → train/tuner.py → stage-specific workflow
      api     → api/app.py (FastAPI + Uvicorn)
      chat    → chat/ChatModel (interactive CLI)
      export  → model export pipeline
      webchat → Gradio chat interface
      webui   → LlamaBoard (full Gradio UI)
      env     → print environment info
      version → print version
```

When the command is `train` and multiple GPUs are detected (or `FORCE_TORCHRUN=1`), `launcher.py` automatically re-launches the process under `torchrun` with appropriate distributed arguments.

### 1.5 Data Flow — Training Pipeline

```
YAML config ──► hparams parsing ──► DataEngine/DataLoader
                                         │
                Model loading ◄──────────┘
                     │
                     ▼
              Trainer.fit() / HF Trainer.train()
                     │
                     ├── Forward pass (micro-batch)
                     ├── Loss computation (stage-specific)
                     ├── Backward + gradient accumulation
                     ├── Optimizer step + LR schedule
                     └── Checkpoint save (every N steps)
                     │
                     ▼
              Output: adapter weights / merged model / checkpoints
```

---

## 2. System Requirements

### 2.1 Software Dependencies

| Dependency | Minimum Version | Purpose |
|---|---|---|
| Python | 3.9+ | Runtime |
| PyTorch | 2.x | Training framework |
| Transformers | 5.x+ | Model loading, tokenizers |
| PEFT | latest | LoRA and adapter methods |
| DeepSpeed | 0.14+ | ZeRO optimization, offloading |
| Accelerate | latest | Distributed training orchestration |
| Gradio | 4.x | WebUI (LlamaBoard) |
| FastAPI | latest | API server |
| Uvicorn | latest | ASGI server |
| OmegaConf | latest | V1 config parsing |
| Pydantic | 2.x | API protocol models |

### 2.2 Hardware Guidelines

| Scenario | GPU Memory | Recommended Setup |
|---|---|---|
| QLoRA 7B model | 8-16 GB | Single consumer GPU (RTX 3090/4090) |
| LoRA 7B model | 16-24 GB | Single A100/H100 or RTX 4090/5090 |
| Full fine-tune 7B | 80+ GB | Multi-GPU (2-4x A100 80GB) |
| Full fine-tune 70B | 320+ GB | 4-8x A100/H100, ZeRO-3 offload |
| LoRA 70B model | 40-80 GB | 1-2x A100 80GB with 4-bit quantization |

> [!NOTE]
> VRAM usage varies significantly with `cutoff_len`, `micro_batch_size`, and quantization settings. Use `RECORD_VRAM=1` to profile actual consumption during a short test run before committing to long training jobs.

### 2.3 Docker Images

Three official Docker images target different accelerator stacks:

| Image | Path | Base | Target Hardware |
|---|---|---|---|
| CUDA | `docker/docker-cuda/` | NVIDIA CUDA | NVIDIA GPUs |
| ROCm | `docker/docker-rocm/` | AMD ROCm | AMD GPUs |
| NPU | `docker/docker-npu/` | Ascend CANN | Huawei Ascend NPUs |

Each includes a Dockerfile with all dependencies pre-installed. Build with standard Docker commands from the respective directory.

---

## 3. Configuration Reference

### 3.1 Legacy Engine Arguments

The legacy engine uses six argument dataclasses parsed from YAML config files:

| Dataclass | Key Fields |
|---|---|
| `ModelArguments` | `model_name_or_path`, `adapter_name_or_path`, `quantization_bit`, `quantization_method` |
| `DataArguments` | `dataset`, `dataset_dir`, `template`, `cutoff_len`, `max_samples`, `streaming` |
| `TrainingArguments` | Extends HF `Seq2SeqTrainingArguments` — `output_dir`, `per_device_train_batch_size`, `learning_rate`, `num_train_epochs`, `bf16`, `deepspeed` |
| `FinetuningArgs` | `stage` (sft/dpo/kto/ppo/rm/pt), `finetuning_type` (lora/full/freeze), `lora_rank`, `lora_target` |
| `GeneratingArgs` | `temperature`, `top_p`, `top_k`, `max_new_tokens`, `repetition_penalty` |
| `EvaluationArgs` | `task`, `n_shot`, `batch_size` |

Configuration is YAML-based. Example skeleton:

```yaml
### model
model_name_or_path: meta-llama/Llama-2-7b-hf

### method
stage: sft
finetuning_type: lora
lora_rank: 8
lora_target: all

### dataset
dataset: alpaca_en
template: llama2
cutoff_len: 1024

### output
output_dir: saves/llama2-7b/lora/sft
logging_steps: 10
save_steps: 500

### train
per_device_train_batch_size: 2
gradient_accumulation_steps: 4
learning_rate: 1.0e-4
num_train_epochs: 3.0
bf16: true
```

### 3.2 V1 Engine Arguments

The V1 engine uses OmegaConf combined with HfArgumentParser, providing a cleaner configuration surface:

**ModelArguments:**

| Field | Default | Description |
|---|---|---|
| `model` | `Qwen/Qwen3-4B-Instruct-2507` | Model name or path |
| `template` | `qwen3_nothink` | Chat template name |
| `trust_remote_code` | (False) | Allow remote code execution |
| `model_class` | `llm` | Model class: `llm`, `cls`, or other |
| `init_config` | (PluginConfig) | Initialization plugin config |
| `peft_config` | (PluginConfig) | PEFT plugin config (LoRA, etc.) |
| `kernel_config` | (PluginConfig) | Kernel plugin config (NPU ops) |
| `quant_config` | (PluginConfig) | Quantization plugin config |

**TrainingArguments:**

| Field | Default | Description |
|---|---|---|
| `output_dir` | auto-generated UUID | Output directory |
| `micro_batch_size` | `1` | Per-device micro-batch size |
| `global_batch_size` | `None` (auto) | Total batch across all devices |
| `cutoff_len` | `2048` | Maximum sequence length |
| `learning_rate` | `1e-4` | Initial learning rate |
| `num_train_epochs` | `3` | Number of training epochs |
| `max_steps` | `None` | Override epoch-based with step count |
| `max_grad_norm` | `1.0` | Gradient clipping threshold |
| `bf16` | `True` | Use bfloat16 precision |
| `batching_strategy` | `normal` | One of: `normal`, `padding_free`, `dynamic_batching`, `dynamic_padding_free` |
| `batching_workers` | `16` | Data loading workers |
| `enable_activation_checkpointing` | `True` | Gradient checkpointing |
| `dist_config` | (PluginConfig) | Distributed training plugin |
| `optim_config` | (PluginConfig) | Optimizer plugin |
| `lr_scheduler_config` | (PluginConfig) | LR scheduler plugin |
| `seed` | `42` | Random seed |
| `resume_from_checkpoint` | `None` | Path or `"auto"` for latest |
| `save_steps` | (None) | Save every N steps |
| `save_epochs` | (None) | Save every N epochs |
| `save_ckpt_as_hf` | `False` | Convert checkpoints to HF format |
| `save_total_limit` | `None` | Max checkpoints to keep |
| `logging_steps` | `1` | Log every N steps |

**DataArguments:**

| Field | Default | Description |
|---|---|---|
| `train_dataset` | (required) | Training dataset specification |
| `eval_dataset` | `None` | Evaluation dataset specification |

**SampleArguments:**

| Field | Default | Description |
|---|---|---|
| `sample_backend` | `hf` | Inference backend: `hf` or `vllm` |
| `max_new_tokens` | `128` | Max generation length |

> [!TIP]
> V1 plugin configs use a nested structure with a mandatory `name` field. For example, to configure LoRA in V1:
> ```yaml
> peft_config:
>   name: lora
>   r: 16
>   lora_alpha: 32
>   target_modules: all
> ```

### 3.3 V1 Batching Strategies

| Strategy | Description | Use Case |
|---|---|---|
| `normal` | Standard padding to max length in batch | Default, simplest |
| `padding_free` | No padding, sequences concatenated | Memory-efficient for variable-length data |
| `dynamic_batching` | Batches adjusted by token budget | Best throughput for diverse sequence lengths |
| `dynamic_padding_free` | Dynamic batching + no padding | Maximum memory efficiency |

### 3.4 Environment Variable Reference

Complete list of environment variables that control runtime behavior:

**Engine Selection:**

| Variable | Default | Description |
|---|---|---|
| `USE_V1` | `0` | Enable V1 engine (`1` to activate) |

**API Server:**

| Variable | Default | Description |
|---|---|---|
| `API_KEY` | `None` | Bearer token for API authentication |
| `API_PORT` | `8000` | Server port |
| `API_HOST` | `0.0.0.0` | Server bind address |
| `API_MODEL_NAME` | `gpt-3.5-turbo` | Model name returned in API responses |
| `FASTAPI_ROOT_PATH` | `""` | Root path prefix (for reverse proxies) |
| `SAFE_MEDIA_PATH` | `api/safe_media` | Allowed local file access path |
| `ALLOW_LOCAL_FILES` | `1` | Enable local file access in API |
| `API_VERBOSE` | `1` | Verbose request logging |
| `MAX_CONCURRENT` | `1` | Maximum concurrent inference requests |

**Logging & Debugging:**

| Variable | Default | Description |
|---|---|---|
| `LLAMAFACTORY_VERBOSITY` | `INFO` | Log level |
| `DISABLE_VERSION_CHECK` | `0` | Skip dependency version checks |
| `RECORD_VRAM` | `0` | Record VRAM usage during training |
| `ALLOW_EXTRA_KEYS` | `0` | Allow unknown keys in YAML configs |

**Distributed Training:**

| Variable | Default | Description |
|---|---|---|
| `FORCE_TORCHRUN` | `0` | Force torchrun launch even for single GPU |
| `USE_MCA` | `0` | Enable MCA (forces torchrun) |
| `OPTIM_TORCH` | `1` | Enable DDP optimizations (expandable_segments, NCCL) |
| `NNODES` | `1` | Number of training nodes |
| `NODE_RANK` | `0` | Current node rank in multi-node setup |
| `NPROC_PER_NODE` | auto-detect | GPUs per node |
| `MASTER_ADDR` | `127.0.0.1` | Master node address |
| `MASTER_PORT` | auto | Master node port |

**Elastic Training:**

| Variable | Default | Description |
|---|---|---|
| `RDZV_ID` | `None` | Rendezvous ID for elastic launch |
| `MIN_NNODES` | `None` | Minimum nodes for elastic training |
| `MAX_NNODES` | `None` | Maximum nodes for elastic training |
| `MAX_RESTARTS` | `0` | Maximum worker restarts |

**Model Hubs:**

| Variable | Default | Description |
|---|---|---|
| `USE_MODELSCOPE_HUB` | `0` | Download models from ModelScope |
| `USE_OPENMIND_HUB` | `0` | Download models from OpenMind |

**WebUI:**

| Variable | Default | Description |
|---|---|---|
| `GRADIO_SERVER_NAME` | `0.0.0.0` | Gradio server bind address |

> [!WARNING]
> Setting `ALLOW_EXTRA_KEYS=1` suppresses errors for unrecognized YAML keys. This is dangerous in production because typos in configuration fields will be silently ignored rather than caught at startup. Use only during development or migration.

---

## 4. Deployment & Scaling

### 4.1 Single-GPU Deployment

The simplest deployment for inference serving:

```bash
# Start API server with a LoRA adapter
llamafactory-cli api \
  --model_name_or_path meta-llama/Llama-2-7b-hf \
  --adapter_name_or_path saves/llama2-7b/lora/sft \
  --template llama2 \
  --finetuning_type lora
```

Or equivalently with a YAML config:

```bash
llamafactory-cli api path/to/config.yaml
```

### 4.2 Multi-GPU Training

LLaMA Factory auto-detects available GPUs and launches `torchrun` automatically. Manual control is available via environment variables:

```bash
# 4 GPUs on a single node
NPROC_PER_NODE=4 llamafactory-cli train config.yaml

# Force torchrun even on single GPU (useful for debugging distributed code)
FORCE_TORCHRUN=1 llamafactory-cli train config.yaml
```

### 4.3 Multi-Node Training

```bash
# Node 0 (master)
NNODES=2 NODE_RANK=0 MASTER_ADDR=10.0.0.1 MASTER_PORT=29500 \
  NPROC_PER_NODE=8 llamafactory-cli train config.yaml

# Node 1
NNODES=2 NODE_RANK=1 MASTER_ADDR=10.0.0.1 MASTER_PORT=29500 \
  NPROC_PER_NODE=8 llamafactory-cli train config.yaml
```

### 4.4 Elastic Training

For fault-tolerant training that can scale nodes dynamically:

```bash
RDZV_ID=my_job MIN_NNODES=2 MAX_NNODES=4 MAX_RESTARTS=3 \
  NPROC_PER_NODE=8 llamafactory-cli train config.yaml
```

### 4.5 DeepSpeed Configurations

Pre-built DeepSpeed configs are provided in `examples/deepspeed/`:

| Config File | ZeRO Stage | Features |
|---|---|---|
| `ds_z0_config.json` | 0 | No sharding (baseline) |
| `ds_z2_config.json` | 2 | Optimizer + gradient sharding with auto tensor parallelism |
| `ds_z2_offload_config.json` | 2 | Stage 2 + CPU offloading |
| `ds_z3_config.json` | 3 | Full parameter sharding |
| `ds_z3_offload_config.json` | 3 | Stage 3 + CPU offloading |
| `ds_z3_fp8_config.json` | 3 | Stage 3 + FP8 precision |

Usage in legacy engine YAML:

```yaml
deepspeed: examples/deepspeed/ds_z3_config.json
```

Usage in V1 engine:

```yaml
dist_config:
  name: deepspeed
  config_file: examples/deepspeed/ds_z3_config.json
```

### 4.6 FSDP Configurations

Available FSDP configs in `examples/accelerate/`:

| Config | Description |
|---|---|
| `fsdp_config.yaml` | Standard FSDP |
| `fsdp2_config.yaml` | FSDP2 (newer PyTorch native) |
| `fsdp_config_offload.yaml` | FSDP with CPU offloading |
| `fsdp_config_multiple_nodes.yaml` | Multi-node FSDP |

### 4.7 Docker Deployment

```bash
# Build CUDA image
cd docker/docker-cuda
docker build -t llamafactory:cuda .

# Run training
docker run --gpus all -v /data/models:/models -v /data/output:/output \
  llamafactory:cuda llamafactory-cli train /config/train.yaml

# Run API server
docker run --gpus all -p 8000:8000 \
  -e API_KEY=your-secret-key \
  -e API_MODEL_NAME=my-fine-tuned-model \
  llamafactory:cuda llamafactory-cli api /config/api.yaml
```

> [!IMPORTANT]
> When running the API server in Docker, always set `API_KEY` to a strong secret. The default configuration has no authentication, and CORS allows all origins.

### 4.8 Reverse Proxy Setup

When deploying behind a reverse proxy (nginx, Traefik), set the `FASTAPI_ROOT_PATH` environment variable to match the proxy's path prefix:

```bash
# If proxied at /llm/v1/
FASTAPI_ROOT_PATH=/llm API_PORT=8000 llamafactory-cli api config.yaml
```

---

## 5. Operations

### 5.1 Checkpoint Management

**Legacy engine:** Uses HuggingFace Trainer checkpointing. Checkpoints are saved to `output_dir/checkpoint-{step}/`.

**V1 engine:** Uses `TrainingCheckpointCoordinator` for distributed-safe checkpointing:

- Saves model state, optimizer state, LR scheduler state, batch generator state, and RNG states
- `resume_from_checkpoint: "auto"` scans `output_dir` for the latest checkpoint
- `save_total_limit` rotates old checkpoints to control disk usage
- `save_ckpt_as_hf: true` converts distributed checkpoints to HuggingFace format (requires approximately 2x memory during save)

```yaml
# V1 checkpoint config example
output_dir: /output/my-training-run
save_steps: 500
save_total_limit: 3
resume_from_checkpoint: auto
save_ckpt_as_hf: false
```

> [!CAUTION]
> Setting `save_ckpt_as_hf: true` roughly doubles memory usage during checkpoint saves because the full model must be materialized in HuggingFace format. On memory-constrained setups, save in native format and convert post-training using the checkpoint conversion scripts.

### 5.2 Checkpoint Conversion Scripts

Located in `scripts/convert_ckpt/`:

| Script | Purpose |
|---|---|
| `llamafy_baichuan2.py` | Convert Baichuan2 to LLaMA format |
| `llamafy_qwen.py` | Convert Qwen to LLaMA format |
| `tiny_llama4.py` | Create tiny LLaMA4 for testing |
| `tiny_qwen3.py` | Create tiny Qwen3 for testing |

### 5.3 Dataset Registry

Datasets are registered in `data/dataset_info.json`. Each entry specifies:

```json
{
  "dataset_name": {
    "file_name": "path/to/data.json",
    "formatting": "alpaca",
    "columns": {
      "prompt": "instruction",
      "query": "input",
      "response": "output"
    }
  }
}
```

The V1 engine uses YAML-based dataset definitions with `DatasetInfo` objects supporting:

| Field | Description |
|---|---|
| `path` | Local path or HF Hub dataset ID |
| `source` | `hf_hub`, `ms_hub`, or `local` |
| `split` | Dataset split name |
| `converter` | Data converter plugin (`alpaca`, `pair`) |
| `size` | Dataset size hint |
| `weight` | Sampling weight for dataset mixing |
| `streaming` | Enable streaming mode |

> [!NOTE]
> In the V1 engine, all datasets in a training run must be either all streaming or all non-streaming. Mixing modes produces the error: "All datasets must be streaming or non-streaming."

### 5.4 Monitoring & Logging

**Log levels:** Controlled via `LLAMAFACTORY_VERBOSITY` (values: `DEBUG`, `INFO`, `WARNING`, `ERROR`, `CRITICAL`).

**VRAM profiling:** Set `RECORD_VRAM=1` to track GPU memory allocation throughout training.

**Training metrics:** Both engines log loss, learning rate, and throughput at intervals set by `logging_steps`.

**External loggers:** The legacy engine supports integration with MLflow, WandB, and TensorBoard via HuggingFace Trainer's reporting infrastructure.

### 5.5 Utility Scripts

Located in `scripts/stat_utils/`:

| Script | Purpose | Usage |
|---|---|---|
| `cal_flops.py` | Calculate model FLOPs | Pre-training capacity planning |
| `cal_mfu.py` | Calculate model FLU utilization | Measure hardware efficiency |
| `cal_ppl.py` | Calculate perplexity | Evaluate model quality |
| `cal_lr.py` | Calculate learning rate schedule | Visualize LR warmup/decay |
| `length_cdf.py` | Dataset length CDF | Choose appropriate `cutoff_len` |

### 5.6 Batch Size Constraints

The V1 engine enforces strict batch size relationships:

```
global_batch_size must be divisible by (data_parallel_size × micro_batch_size)
```

If `global_batch_size` is not set, it is computed automatically as `micro_batch_size × data_parallel_size`. Violations produce: "Global batch size must be divisible by DP size and micro batch size."

---

## 6. Performance

### 6.1 Memory Optimization Ladder

Listed from least to most aggressive, apply cumulatively as needed:

| Technique | Memory Savings | Config |
|---|---|---|
| BF16 training | ~50% vs FP32 | `bf16: true` |
| Gradient checkpointing | ~30-50% activation memory | V1: `enable_activation_checkpointing: true`; Legacy: `gradient_checkpointing: true` |
| LoRA | ~90% trainable params | `finetuning_type: lora` / V1: `peft_config.name: lora` |
| QLoRA (4-bit) | ~75% model memory | `quantization_bit: 4` + LoRA |
| DeepSpeed ZeRO-2 | Shard optimizer + gradients | `deepspeed: ds_z2_config.json` |
| DeepSpeed ZeRO-3 | Shard everything | `deepspeed: ds_z3_config.json` |
| ZeRO-3 + CPU offload | Offload to system RAM | `deepspeed: ds_z3_offload_config.json` |
| Padding-free batching | Eliminate padding waste | V1: `batching_strategy: padding_free` |

### 6.2 Throughput Optimization

**Batching strategy selection (V1):**
- `normal`: Safe default. Pads all sequences to the longest in the batch.
- `padding_free`: Concatenates sequences without padding. Best for datasets with high length variance. No wasted compute on pad tokens.
- `dynamic_batching`: Groups sequences by similar length into token-budget batches. Maximizes GPU utilization.
- `dynamic_padding_free`: Combines both strategies for maximum efficiency.

**Data loading:** `batching_workers: 16` (V1 default) controls parallel data preprocessing. Increase on CPU-rich nodes; decrease if CPU is the bottleneck.

**DDP optimization:** `OPTIM_TORCH=1` (default) enables PyTorch DDP optimizations including CUDA expandable memory segments and NCCL optimizations.

### 6.3 V1 Training Loop Internals

The `BaseTrainer.fit()` loop operates as follows:

1. **BatchGenerator initialization** — creates batches according to the configured `batching_strategy`
2. **Optimizer creation** — default AdamW, overridable via `optim_config` plugin
3. **LR scheduler creation** — configurable via `lr_scheduler_config` plugin
4. **Training loop** per step:
   - Accumulate `N` micro-batches (where `N = global_batch_size / (micro_batch_size × dp_size)`)
   - Each micro-batch: forward → loss → backward
   - Loss scaling uses valid-token weighting (not naive averaging)
   - If DeepSpeed: engine handles backward pass and optimizer step
   - If FSDP2/DDP: manual `clip_grad_norm_` → `optimizer.step()` → `scheduler.step()`
5. **Checkpoint save** at configured intervals
6. **Callbacks** fire at each logging step (LoggingCallback built-in, custom TrainerCallback supported)

### 6.4 Sequence Parallelism

The V1 engine supports Ulysses sequence parallelism for extremely long sequences via the model plugin system. Note the limitation: "Sequence parallel is not supported for qwen3.5 model."

### 6.5 NPU Kernel Optimizations

For Huawei Ascend NPUs, the V1 kernel plugin provides fused operations:

| Kernel | Operation |
|---|---|
| `fused_moe` | Fused Mixture-of-Experts |
| `swiglu` | Fused SwiGLU activation |
| `rms_norm` | Fused RMS normalization |
| `rope` | Fused rotary position embedding |

Configure via:

```yaml
kernel_config:
  name: npu
```

---

## 7. Security

### 7.1 API Authentication

The API server authenticates requests via Bearer token:

```
Authorization: Bearer <API_KEY>
```

Set via the `API_KEY` environment variable. When `API_KEY` is not set, the API operates without authentication.

> [!CAUTION]
> Running the API server without `API_KEY` set exposes the model to unauthenticated access. Always set a strong API key in production deployments.

### 7.2 CORS Policy

The API server allows all origins (`*`) by default. For production deployments behind a reverse proxy, configure CORS restrictions at the proxy level.

### 7.3 Local File Injection (LFI) Protection

The `check_lfi_path()` function in `api/common.py` restricts local file access to the directory specified by `SAFE_MEDIA_PATH`. Any request attempting to access files outside this boundary is rejected.

- Default `SAFE_MEDIA_PATH`: `api/safe_media` (relative to working directory)
- `ALLOW_LOCAL_FILES=0` disables all local file access through the API

### 7.4 Server-Side Request Forgery (SSRF) Protection

The `check_ssrf_url()` function in `api/common.py` prevents the API from making requests to internal network resources:

- Blocks private IP ranges (10.x, 172.16-31.x, 192.168.x, 127.x, etc.)
- Blocks reserved/link-local addresses
- Validates HTTP and HTTPS schemes only
- Rejects URLs targeting internal services

### 7.5 Remote Code Execution

The `trust_remote_code` parameter (both engines) allows execution of arbitrary Python code bundled with model repositories. This is required by some models but represents a significant security risk.

> [!WARNING]
> Only enable `trust_remote_code` for models from trusted sources. Malicious model repositories can execute arbitrary code during model loading. In production, prefer models that do not require remote code.

### 7.6 Security Hardening Checklist

| Item | Action |
|---|---|
| API authentication | Set `API_KEY` to a strong random secret |
| Local file access | Set `ALLOW_LOCAL_FILES=0` unless required |
| SAFE_MEDIA_PATH | Restrict to a dedicated media directory |
| Remote code | Set `trust_remote_code: false` unless model requires it |
| CORS | Restrict at reverse proxy level |
| Network exposure | Bind API to localhost (`API_HOST=127.0.0.1`) behind reverse proxy |
| Container isolation | Run containers with `--read-only` filesystem where possible |
| GPU isolation | Use `NVIDIA_VISIBLE_DEVICES` to restrict GPU access |

---

## 8. Developer Internals

### 8.1 V1 Plugin System

The V1 engine's extensibility is built on the `BasePlugin` registry pattern in `v1/utils/plugin.py`.

**Registration:**

```python
@InitPlugin("custom_init").register("load_model")
def custom_load_model(model_args, ...):
    # Custom model loading logic
    pass
```

**Invocation:**

```python
InitPlugin("custom_init")()  # Calls the registered method
```

**Plugin hierarchy:**

| Plugin Base Class | Purpose | Methods |
|---|---|---|
| `InitPlugin` | Model initialization | `load_model`, `load_tokenizer` |
| `PeftPlugin` | Parameter-efficient fine-tuning | `apply_peft`, `save_peft` |
| `QuantizationPlugin` | Model quantization | `quantize_model` |
| `KernelPlugin` | Custom compute kernels | `apply_kernels` |
| `DataLoaderPlugin` | Data loading customization | `load_data` |
| `DataConverterPlugin` | Data format conversion | `convert` (alpaca, pair formats) |
| `OptimizerPlugin` | Custom optimizers | `create_optimizer` |
| `LRSchedulerPlugin` | Learning rate schedules | `create_scheduler` |
| `DistributedPlugin` | Distributed training backends | `setup`, `teardown` |

**Plugin configs** use `PluginConfig` — a dict-like object requiring a `name` field that maps to the registered plugin. Missing the name field produces: "Plugin configuration must have a 'name' field."

### 8.2 V1 DataEngine Internals

`DataEngine` (`v1/core/data_engine.py`) implements a `torch.utils.data.Dataset`:

1. Parses dataset YAML config into `DatasetInfo` objects
2. Each dataset can come from HuggingFace Hub, ModelScope, OpenMind, or local files
3. Data converters transform raw records into the internal message format (ChatML)
4. Builds a flat `data_index` for efficient random access across multiple datasets
5. Supports dataset mixing via `weight` field
6. Streaming mode for datasets too large to fit in memory (disables random access — "Streaming dataset does not support index access.")

### 8.3 V1 ModelEngine Internals

`ModelEngine` (`v1/core/model_engine.py`) orchestrates model setup:

1. Load base model via `InitPlugin`
2. Apply quantization via `QuantizationPlugin` (if configured)
3. Apply PEFT via `PeftPlugin` (if configured)
4. Apply custom kernels via `KernelPlugin` (if configured)
5. Apply distributed wrapping (DDP, FSDP2, or DeepSpeed)

> [!NOTE]
> When using LoRA with the V1 engine, loading the model by meta device (lazy initialization) is not supported. Attempting this produces: "Currently lora stage does not support loading model by meta."

### 8.4 V1 Distributed Training

The `DistributedInterface` in `v1/accelerator/` abstracts distributed backends:

**DeepSpeed:** Uses the DeepSpeed engine to manage backward pass, optimizer step, and gradient accumulation. Configured via JSON config files (ZeRO stages 0/2/3).

**FSDP2:** Uses PyTorch's native FSDP2. Manages parameter sharding and gradient synchronization. Configured via YAML accelerate configs.

**DDP:** Standard PyTorch DistributedDataParallel. The default when no distributed config is specified with multiple GPUs.

The V1 engine requires `drop_last=True` for distributed training to ensure even batch distribution across ranks. Violating this produces: "Drop last must be True."

### 8.5 Legacy Engine ChatModel

The `ChatModel` class in `chat/` supports four inference backends:

| Backend | Engine | Use Case |
|---|---|---|
| HuggingFace | `transformers` generate | Default, broadest compatibility |
| vLLM | `vllm` | High-throughput serving |
| SGLang | `sglang` | Structured generation |
| KTransformers | `ktransformers` | Specialized inference |

### 8.6 API Server Internals

The FastAPI server (`api/app.py`) exposes three endpoints:

| Method | Path | Handler | Description |
|---|---|---|---|
| GET | `/v1/models` | `list_models` | Returns loaded model info |
| POST | `/v1/chat/completions` | `create_chat_completion` | OpenAI-compatible chat completion |
| POST | `/v1/score/evaluation` | `create_score_evaluation` | Score/evaluate responses |

Request/response models are defined in `api/protocol.py` using Pydantic v2. The API follows the OpenAI chat completions specification for maximum compatibility with existing client libraries.

Concurrency is controlled by `MAX_CONCURRENT` (default 1), which limits simultaneous inference requests to prevent GPU OOM.

### 8.7 Template System

Both engines use a template system to format conversations for different model families. Templates define:

- System message handling
- Turn delimiters and role tokens
- Tool/function call formatting
- Stop sequences

The V1 engine ships with built-in templates including `qwen3` and `qwen3_nothink` (the default, which suppresses chain-of-thought tokens).

### 8.8 Test Infrastructure

Two independent test suites:

| Suite | Path | Coverage |
|---|---|---|
| Legacy tests | `tests/` | Data processing, model loading, templates, training, evaluation |
| V1 tests | `tests_v1/` | V1 data engine, model engine, training, plugins |

Run with pytest:

```bash
# All legacy tests
pytest tests/

# All V1 tests
pytest tests_v1/

# Specific test
pytest tests_v1/test_data_engine.py -v
```

### 8.9 Project File Structure

```
hiyouga/LlamaFactory/
├── src/llamafactory/          # Main package
│   ├── cli.py                 # Entry point
│   ├── launcher.py            # Legacy command dispatch
│   ├── train/                 # Training stages (sft, dpo, kto, ppo, rm, pt)
│   ├── model/                 # Model loading & adapters
│   ├── data/                  # Data loading & templates
│   ├── hparams/               # Argument dataclasses
│   ├── chat/                  # Interactive chat (multi-backend)
│   ├── api/                   # FastAPI server
│   ├── webui/                 # LlamaBoard (Gradio)
│   ├── eval/                  # Evaluation (deprecated)
│   ├── extras/                # Utilities
│   └── v1/                    # V1 plugin-based engine
│       ├── core/              # DataEngine, ModelEngine, BaseTrainer
│       ├── config/            # OmegaConf argument parsing
│       ├── plugins/           # All plugin implementations
│       ├── trainers/          # SFT, DPO, RM trainers
│       ├── samplers/          # CLI sampler
│       ├── accelerator/       # Distributed interface
│       └── utils/             # Plugin system, helpers
├── data/                      # Demo datasets + dataset_info.json
├── examples/                  # YAML configs for all scenarios
│   ├── deepspeed/             # DeepSpeed JSON configs
│   ├── accelerate/            # FSDP YAML configs
│   └── ...                    # Per-stage example configs
├── docker/                    # Dockerfiles (CUDA, ROCm, NPU)
├── scripts/                   # Utilities (FLOPs, MFU, PPL, checkpoint conversion)
├── tests/                     # Legacy test suite
└── tests_v1/                  # V1 test suite
```

---

## 9. Integration Points

### 9.1 OpenAI API Compatibility

The API server implements the OpenAI chat completions protocol, making it a drop-in replacement for applications using the OpenAI client library:

```python
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="your-api-key"
)

response = client.chat.completions.create(
    model="my-model",
    messages=[{"role": "user", "content": "Hello!"}]
)
```

The `API_MODEL_NAME` environment variable controls what model name the server reports. Clients can use any model name in their requests; the server always uses the loaded model.

### 9.2 HuggingFace Hub Integration

- Models: Loaded directly from HuggingFace Hub via `model_name_or_path` / `model`
- Datasets: Referenced by Hub ID in dataset configs
- Adapters: Saved to and loaded from Hub-compatible format
- Checkpoints: `save_ckpt_as_hf` converts to Hub format

### 9.3 ModelScope and OpenMind Hubs

Alternative model/dataset sources activated via environment variables:

```bash
# Use ModelScope for model downloads
USE_MODELSCOPE_HUB=1 llamafactory-cli train config.yaml

# Use OpenMind for model downloads
USE_OPENMIND_HUB=1 llamafactory-cli train config.yaml
```

### 9.4 Gradio WebUI (LlamaBoard)

The WebUI provides a browser-based interface for:

- Training configuration and launch
- Model chat/evaluation
- Dataset preview

```bash
# Start WebUI
llamafactory-cli webui

# Or with custom binding
GRADIO_SERVER_NAME=0.0.0.0 llamafactory-cli webui
```

### 9.5 Tool Calling

The API supports tool/function calling in the OpenAI format. Example scripts are provided in `scripts/api_example/` demonstrating tool calling and image input.

---

## Appendix A: Troubleshooting

### A.1 "can't start new thread" (RuntimeError)

**Error:** `RuntimeError: can't start new thread`

**Cause:** System thread limit reached, commonly when `batching_workers` is too high or the system `ulimit` for threads is restrictive.

**Solution:** Reduce `batching_workers` in training config. Increase the system thread limit: `ulimit -u 65536`.

**Prevention:** Set `batching_workers` proportional to available CPU cores, not arbitrarily high.

### A.2 "backward through graph twice" with Unsloth + ZeRO-3

**Error:** RuntimeError about backward through the graph a second time, occurring with `unsloth_gc` enabled alongside Gemma4 and DeepSpeed ZeRO-3.

**Cause:** Incompatibility between Unsloth's gradient checkpointing and ZeRO-3's parameter partitioning for certain model architectures.

**Solution:** Disable `unsloth_gc` when using ZeRO-3 with affected models (Gemma4). Use standard gradient checkpointing instead.

**Prevention:** Test new model + optimizer + distributed combinations on small datasets before committing to full training runs.

### A.3 "ignore_mismatched_sizes" breaking model loading

**Error:** Model loading fails or produces incorrect results when `ignore_mismatched_sizes` is enabled in the loader.

**Cause:** Silently loading mismatched tensor shapes corrupts the model rather than failing loudly.

**Solution:** Remove `ignore_mismatched_sizes` and fix the root cause (wrong model path, incompatible adapter, architecture mismatch).

**Prevention:** Always verify model and adapter compatibility before loading.

### A.4 MLflow external logger failing

**Error:** MLflow integration produces errors or fails to log metrics.

**Cause:** MLflow configuration or connectivity issues with the tracking server.

**Solution:** Verify MLflow tracking URI is accessible. Check MLflow version compatibility with the HuggingFace Trainer version in use.

**Prevention:** Test MLflow connectivity independently before integrating with training.

### A.5 GPU 0 barrier warning in distributed training

**Error:** Warning about GPU 0 waiting at a barrier during distributed training.

**Cause:** Uneven workload distribution or a process on one rank taking significantly longer.

**Solution:** Ensure all ranks have equal data. Check for CPU bottlenecks on specific nodes. Verify network connectivity between nodes.

**Prevention:** Use `drop_last=True` (mandatory in V1) to ensure even batch distribution.

### A.6 Data packing error in SFT with multimodal models

**Error:** Data packing fails during SFT with models like Qwen3VL.

**Cause:** Packing (concatenating multiple examples into one sequence) conflicts with multimodal token handling.

**Solution:** Disable packing when training multimodal models.

**Prevention:** Review model-specific documentation for known training limitations.

### A.7 "Plugin configuration must have a 'name' field"

**Error:** Raised during V1 engine config parsing.

**Cause:** A plugin config block (e.g., `peft_config`, `dist_config`) is specified without the required `name` field.

**Solution:** Add the `name` field identifying which plugin implementation to use:

```yaml
peft_config:
  name: lora     # This field is mandatory
  r: 16
```

### A.8 "Global batch size must be divisible by DP size and micro batch size"

**Error:** Raised at training start in V1 engine.

**Cause:** The `global_batch_size` is not evenly divisible by the product of data parallel size and `micro_batch_size`.

**Solution:** Either remove `global_batch_size` (let it auto-compute) or set it to a valid multiple. For example, with 4 GPUs and `micro_batch_size: 2`, valid `global_batch_size` values are 8, 16, 24, etc.

> [!TIP]
> When troubleshooting distributed training issues, set `LLAMAFACTORY_VERBOSITY=DEBUG` and `NPROC_PER_NODE=1` to isolate whether the issue is distributed-specific or a general training problem.

---

_Generated May 2026 from hiyouga/LlamaFactory at e0bc3c1. AI-generated documentation. The library is checked by sampling (three repositories so far); this guide was not individually reviewed. Verify against the project before relying on it. Unofficial; not affiliated with or reviewed by the project._