LLaMA Factory Administrator & Developer Guide

LLaMA Factory is a unified framework for efficiently fine-tuning 100+ large language models. This guide covers architecture, configuration, deployment, security, and developer internals for administrators, operators, and contributors working with the system at a production level.

For everyday usage, CLI commands, and getting-started workflows, see the User Guide.

1. Architecture Overview

1.1 Dual-Engine Design

LLaMA Factory ships two complete training engines selectable at runtime. Both share the same CLI entry point (llamafactory-cli / lmf) and configuration surface, but differ fundamentally in extensibility and internal design.

Important

The V1 engine is activated exclusively via the USE_V1=1 environment variable. There is no YAML flag or CLI switch. All V1-specific configuration (plugin configs, batching strategies) is ignored when V1 is not active.

1.2 Legacy Engine Module Map

src/llamafactory/
├── cli.py                  # Entry point, dispatches commands
├── launcher.py             # Command dispatch, torchrun auto-launch
├── train/
│   └── tuner.py            # Stage router → per-stage workflows
│       ├── sft/            # Supervised fine-tuning
│       ├── dpo/            # Direct preference optimization
│       ├── kto/            # KTO training
│       ├── ppo/            # Proximal policy optimization
│       ├── rm/             # Reward modeling
│       └── pt/             # Pretraining / continued pretraining
├── model/                  # Model loading, adapter mgmt, quantization
├── data/                   # Data loading, template system, multimodal, tool formatting
├── hparams/                # Argument dataclasses
├── chat/                   # ChatModel (HF, vLLM, SGLang, KTransformers engines)
├── webui/                  # Gradio-based LlamaBoard
├── api/                    # FastAPI OpenAI-compatible server
├── eval/                   # Evaluation (deprecated)
└── extras/                 # Constants, env detection, logging, misc, package checks

1.3 V1 Engine Module Map

src/llamafactory/v1/
├── launcher.py             # V1 command dispatch
├── core/
│   ├── data_engine.py      # torch Dataset from YAML/HF Hub/local
│   ├── model_engine.py     # Model init, PEFT, quantization, kernels
│   ├── base_trainer.py     # Training loop with micro-batch accumulation
│   ├── base_sampler.py     # Inference engine
│   └── utils/
│       ├── batching.py     # BatchGenerator with strategies
│       ├── checkpoint.py   # Distributed checkpoint save/resume
│       └── rendering.py    # ChatML message rendering
├── config/                 # OmegaConf + HfArgumentParser arg parsing
├── plugins/
│   ├── data_plugins/       # Data loading + conversion (alpaca, pair)
│   ├── model_plugins/      # Init, PEFT (LoRA), quant, kernels, templates, parallelization
│   ├── trainer_plugins/    # Optimizer, LR scheduler, distributed (DeepSpeed, FSDP2, Hub)
│   └── sampler_plugins/    # vLLM sampler
├── trainers/               # SFTTrainer, DPOTrainer, RMTrainer
├── samplers/               # CLISampler
├── accelerator/            # DistributedInterface, profiler
└── utils/                  # Types, logging, plugin system, helpers, constants

1.4 Command Dispatch Flow

llamafactory-cli <command>
        │
        ▼
    cli.py ─── USE_V1=1? ───► v1/launcher.py
        │ no                       │
        ▼                          ▼
    launcher.py              V1 dispatch
        │                    (train/chat)
        ▼
    Dispatch table:
      train   → train/tuner.py → stage-specific workflow
      api     → api/app.py (FastAPI + Uvicorn)
      chat    → chat/ChatModel (interactive CLI)
      export  → model export pipeline
      webchat → Gradio chat interface
      webui   → LlamaBoard (full Gradio UI)
      env     → print environment info
      version → print version

When the command is train and multiple GPUs are detected (or FORCE_TORCHRUN=1), launcher.py automatically re-launches the process under torchrun with appropriate distributed arguments.

1.5 Data Flow — Training Pipeline

YAML config ──► hparams parsing ──► DataEngine/DataLoader
                                         │
                Model loading ◄──────────┘
                     │
                     ▼
              Trainer.fit() / HF Trainer.train()
                     │
                     ├── Forward pass (micro-batch)
                     ├── Loss computation (stage-specific)
                     ├── Backward + gradient accumulation
                     ├── Optimizer step + LR schedule
                     └── Checkpoint save (every N steps)
                     │
                     ▼
              Output: adapter weights / merged model / checkpoints

2. System Requirements

2.1 Software Dependencies

2.2 Hardware Guidelines

Note

VRAM usage varies significantly with cutoff_len, micro_batch_size, and quantization settings. Use RECORD_VRAM=1 to profile actual consumption during a short test run before committing to long training jobs.

2.3 Docker Images

Three official Docker images target different accelerator stacks:

Each includes a Dockerfile with all dependencies pre-installed. Build with standard Docker commands from the respective directory.

3. Configuration Reference

3.1 Legacy Engine Arguments

The legacy engine uses six argument dataclasses parsed from YAML config files:

Configuration is YAML-based. Example skeleton:

### model
model_name_or_path: meta-llama/Llama-2-7b-hf

### method
stage: sft
finetuning_type: lora
lora_rank: 8
lora_target: all

### dataset
dataset: alpaca_en
template: llama2
cutoff_len: 1024

### output
output_dir: saves/llama2-7b/lora/sft
logging_steps: 10
save_steps: 500

### train
per_device_train_batch_size: 2
gradient_accumulation_steps: 4
learning_rate: 1.0e-4
num_train_epochs: 3.0
bf16: true

3.2 V1 Engine Arguments

The V1 engine uses OmegaConf combined with HfArgumentParser, providing a cleaner configuration surface:

ModelArguments:

TrainingArguments:

DataArguments:

SampleArguments:

Tip

V1 plugin configs use a nested structure with a mandatory name field. For example, to configure LoRA in V1:

peft_config:
  name: lora
  r: 16
  lora_alpha: 32
  target_modules: all

3.3 V1 Batching Strategies

3.4 Environment Variable Reference

Complete list of environment variables that control runtime behavior:

Engine Selection:

API Server:

Logging & Debugging:

Distributed Training:

Elastic Training:

Model Hubs:

WebUI:

Warning

Setting ALLOW_EXTRA_KEYS=1 suppresses errors for unrecognized YAML keys. This is dangerous in production because typos in configuration fields will be silently ignored rather than caught at startup. Use only during development or migration.

4. Deployment & Scaling

4.1 Single-GPU Deployment

The simplest deployment for inference serving:

# Start API server with a LoRA adapter
llamafactory-cli api \
  --model_name_or_path meta-llama/Llama-2-7b-hf \
  --adapter_name_or_path saves/llama2-7b/lora/sft \
  --template llama2 \
  --finetuning_type lora

Or equivalently with a YAML config:

llamafactory-cli api path/to/config.yaml

4.2 Multi-GPU Training

LLaMA Factory auto-detects available GPUs and launches torchrun automatically. Manual control is available via environment variables:

# 4 GPUs on a single node
NPROC_PER_NODE=4 llamafactory-cli train config.yaml

# Force torchrun even on single GPU (useful for debugging distributed code)
FORCE_TORCHRUN=1 llamafactory-cli train config.yaml

4.3 Multi-Node Training

# Node 0 (master)
NNODES=2 NODE_RANK=0 MASTER_ADDR=10.0.0.1 MASTER_PORT=29500 \
  NPROC_PER_NODE=8 llamafactory-cli train config.yaml

# Node 1
NNODES=2 NODE_RANK=1 MASTER_ADDR=10.0.0.1 MASTER_PORT=29500 \
  NPROC_PER_NODE=8 llamafactory-cli train config.yaml

4.4 Elastic Training

For fault-tolerant training that can scale nodes dynamically:

RDZV_ID=my_job MIN_NNODES=2 MAX_NNODES=4 MAX_RESTARTS=3 \
  NPROC_PER_NODE=8 llamafactory-cli train config.yaml

4.5 DeepSpeed Configurations

Pre-built DeepSpeed configs are provided in examples/deepspeed/:

Usage in legacy engine YAML:

deepspeed: examples/deepspeed/ds_z3_config.json

Usage in V1 engine:

dist_config:
  name: deepspeed
  config_file: examples/deepspeed/ds_z3_config.json

4.6 FSDP Configurations

Available FSDP configs in examples/accelerate/:

4.7 Docker Deployment

# Build CUDA image
cd docker/docker-cuda
docker build -t llamafactory:cuda .

# Run training
docker run --gpus all -v /data/models:/models -v /data/output:/output \
  llamafactory:cuda llamafactory-cli train /config/train.yaml

# Run API server
docker run --gpus all -p 8000:8000 \
  -e API_KEY=your-secret-key \
  -e API_MODEL_NAME=my-fine-tuned-model \
  llamafactory:cuda llamafactory-cli api /config/api.yaml

Important

When running the API server in Docker, always set API_KEY to a strong secret. The default configuration has no authentication, and CORS allows all origins.

4.8 Reverse Proxy Setup

When deploying behind a reverse proxy (nginx, Traefik), set the FASTAPI_ROOT_PATH environment variable to match the proxy's path prefix:

# If proxied at /llm/v1/
FASTAPI_ROOT_PATH=/llm API_PORT=8000 llamafactory-cli api config.yaml

5. Operations

5.1 Checkpoint Management

Legacy engine: Uses HuggingFace Trainer checkpointing. Checkpoints are saved to output_dir/checkpoint-{step}/.

V1 engine: Uses TrainingCheckpointCoordinator for distributed-safe checkpointing:

• Saves model state, optimizer state, LR scheduler state, batch generator state, and RNG states
• resume_from_checkpoint: "auto" scans output_dir for the latest checkpoint
• save_total_limit rotates old checkpoints to control disk usage
• save_ckpt_as_hf: true converts distributed checkpoints to HuggingFace format (requires approximately 2x memory during save)

# V1 checkpoint config example
output_dir: /output/my-training-run
save_steps: 500
save_total_limit: 3
resume_from_checkpoint: auto
save_ckpt_as_hf: false

Caution

Setting save_ckpt_as_hf: true roughly doubles memory usage during checkpoint saves because the full model must be materialized in HuggingFace format. On memory-constrained setups, save in native format and convert post-training using the checkpoint conversion scripts.

5.2 Checkpoint Conversion Scripts

Located in scripts/convert_ckpt/:

5.3 Dataset Registry

Datasets are registered in data/dataset_info.json. Each entry specifies:

{
  "dataset_name": {
    "file_name": "path/to/data.json",
    "formatting": "alpaca",
    "columns": {
      "prompt": "instruction",
      "query": "input",
      "response": "output"
    }
  }
}

The V1 engine uses YAML-based dataset definitions with DatasetInfo objects supporting:

Note

In the V1 engine, all datasets in a training run must be either all streaming or all non-streaming. Mixing modes produces the error: "All datasets must be streaming or non-streaming."

5.4 Monitoring & Logging

Log levels: Controlled via LLAMAFACTORY_VERBOSITY (values: DEBUG, INFO, WARNING, ERROR, CRITICAL).

VRAM profiling: Set RECORD_VRAM=1 to track GPU memory allocation throughout training.

Training metrics: Both engines log loss, learning rate, and throughput at intervals set by logging_steps.

External loggers: The legacy engine supports integration with MLflow, WandB, and TensorBoard via HuggingFace Trainer's reporting infrastructure.

5.5 Utility Scripts

Located in scripts/stat_utils/:

5.6 Batch Size Constraints

The V1 engine enforces strict batch size relationships:

global_batch_size must be divisible by (data_parallel_size × micro_batch_size)

If global_batch_size is not set, it is computed automatically as micro_batch_size × data_parallel_size. Violations produce: "Global batch size must be divisible by DP size and micro batch size."

6. Performance

6.1 Memory Optimization Ladder

Listed from least to most aggressive, apply cumulatively as needed:

6.2 Throughput Optimization

Batching strategy selection (V1):

• normal: Safe default. Pads all sequences to the longest in the batch.
• padding_free: Concatenates sequences without padding. Best for datasets with high length variance. No wasted compute on pad tokens.
• dynamic_batching: Groups sequences by similar length into token-budget batches. Maximizes GPU utilization.
• dynamic_padding_free: Combines both strategies for maximum efficiency.

Data loading: batching_workers: 16 (V1 default) controls parallel data preprocessing. Increase on CPU-rich nodes; decrease if CPU is the bottleneck.

DDP optimization: OPTIM_TORCH=1 (default) enables PyTorch DDP optimizations including CUDA expandable memory segments and NCCL optimizations.

6.3 V1 Training Loop Internals

The BaseTrainer.fit() loop operates as follows:

1. BatchGenerator initialization — creates batches according to the configured batching_strategy
2. Optimizer creation — default AdamW, overridable via optim_config plugin
3. LR scheduler creation — configurable via lr_scheduler_config plugin
4. Training loop per step:
   • Accumulate N micro-batches (where N = global_batch_size / (micro_batch_size × dp_size))
   • Each micro-batch: forward → loss → backward
   • Loss scaling uses valid-token weighting (not naive averaging)
   • If DeepSpeed: engine handles backward pass and optimizer step
   • If FSDP2/DDP: manual clip_grad_norm_ → optimizer.step() → scheduler.step()
5. Checkpoint save at configured intervals
6. Callbacks fire at each logging step (LoggingCallback built-in, custom TrainerCallback supported)

6.4 Sequence Parallelism

The V1 engine supports Ulysses sequence parallelism for extremely long sequences via the model plugin system. Note the limitation: "Sequence parallel is not supported for qwen3.5 model."

6.5 NPU Kernel Optimizations

For Huawei Ascend NPUs, the V1 kernel plugin provides fused operations:

Configure via:

kernel_config:
  name: npu

7. Security

7.1 API Authentication

The API server authenticates requests via Bearer token:

Authorization: Bearer <API_KEY>

Set via the API_KEY environment variable. When API_KEY is not set, the API operates without authentication.

Caution

Running the API server without API_KEY set exposes the model to unauthenticated access. Always set a strong API key in production deployments.

7.2 CORS Policy

The API server allows all origins (*) by default. For production deployments behind a reverse proxy, configure CORS restrictions at the proxy level.

7.3 Local File Injection (LFI) Protection

The check_lfi_path() function in api/common.py restricts local file access to the directory specified by SAFE_MEDIA_PATH. Any request attempting to access files outside this boundary is rejected.

• Default SAFE_MEDIA_PATH: api/safe_media (relative to working directory)
• ALLOW_LOCAL_FILES=0 disables all local file access through the API

7.4 Server-Side Request Forgery (SSRF) Protection

The check_ssrf_url() function in api/common.py prevents the API from making requests to internal network resources:

• Blocks private IP ranges (10.x, 172.16-31.x, 192.168.x, 127.x, etc.)
• Blocks reserved/link-local addresses
• Validates HTTP and HTTPS schemes only
• Rejects URLs targeting internal services

7.5 Remote Code Execution

The trust_remote_code parameter (both engines) allows execution of arbitrary Python code bundled with model repositories. This is required by some models but represents a significant security risk.

Warning

Only enable trust_remote_code for models from trusted sources. Malicious model repositories can execute arbitrary code during model loading. In production, prefer models that do not require remote code.

7.6 Security Hardening Checklist

8. Developer Internals

8.1 V1 Plugin System

The V1 engine's extensibility is built on the BasePlugin registry pattern in v1/utils/plugin.py.

Registration:

@InitPlugin("custom_init").register("load_model")
def custom_load_model(model_args, ...):
    # Custom model loading logic
    pass

Invocation:

InitPlugin("custom_init")()  # Calls the registered method

Plugin hierarchy:

Plugin configs use PluginConfig — a dict-like object requiring a name field that maps to the registered plugin. Missing the name field produces: "Plugin configuration must have a 'name' field."

8.2 V1 DataEngine Internals

DataEngine (v1/core/data_engine.py) implements a torch.utils.data.Dataset:

1. Parses dataset YAML config into DatasetInfo objects
2. Each dataset can come from HuggingFace Hub, ModelScope, OpenMind, or local files
3. Data converters transform raw records into the internal message format (ChatML)
4. Builds a flat data_index for efficient random access across multiple datasets
5. Supports dataset mixing via weight field
6. Streaming mode for datasets too large to fit in memory (disables random access — "Streaming dataset does not support index access.")

8.3 V1 ModelEngine Internals

ModelEngine (v1/core/model_engine.py) orchestrates model setup:

1. Load base model via InitPlugin
2. Apply quantization via QuantizationPlugin (if configured)
3. Apply PEFT via PeftPlugin (if configured)
4. Apply custom kernels via KernelPlugin (if configured)
5. Apply distributed wrapping (DDP, FSDP2, or DeepSpeed)

Note

When using LoRA with the V1 engine, loading the model by meta device (lazy initialization) is not supported. Attempting this produces: "Currently lora stage does not support loading model by meta."

8.4 V1 Distributed Training

The DistributedInterface in v1/accelerator/ abstracts distributed backends:

DeepSpeed: Uses the DeepSpeed engine to manage backward pass, optimizer step, and gradient accumulation. Configured via JSON config files (ZeRO stages 0/2/3).

FSDP2: Uses PyTorch's native FSDP2. Manages parameter sharding and gradient synchronization. Configured via YAML accelerate configs.

DDP: Standard PyTorch DistributedDataParallel. The default when no distributed config is specified with multiple GPUs.

The V1 engine requires drop_last=True for distributed training to ensure even batch distribution across ranks. Violating this produces: "Drop last must be True."

8.5 Legacy Engine ChatModel

The ChatModel class in chat/ supports four inference backends:

8.6 API Server Internals

The FastAPI server (api/app.py) exposes three endpoints:

Request/response models are defined in api/protocol.py using Pydantic v2. The API follows the OpenAI chat completions specification for maximum compatibility with existing client libraries.

Concurrency is controlled by MAX_CONCURRENT (default 1), which limits simultaneous inference requests to prevent GPU OOM.

8.7 Template System

Both engines use a template system to format conversations for different model families. Templates define:

• System message handling
• Turn delimiters and role tokens
• Tool/function call formatting
• Stop sequences

The V1 engine ships with built-in templates including qwen3 and qwen3_nothink (the default, which suppresses chain-of-thought tokens).

8.8 Test Infrastructure

Two independent test suites:

Run with pytest:

# All legacy tests
pytest tests/

# All V1 tests
pytest tests_v1/

# Specific test
pytest tests_v1/test_data_engine.py -v

8.9 Project File Structure

hiyouga/LlamaFactory/
├── src/llamafactory/          # Main package
│   ├── cli.py                 # Entry point
│   ├── launcher.py            # Legacy command dispatch
│   ├── train/                 # Training stages (sft, dpo, kto, ppo, rm, pt)
│   ├── model/                 # Model loading & adapters
│   ├── data/                  # Data loading & templates
│   ├── hparams/               # Argument dataclasses
│   ├── chat/                  # Interactive chat (multi-backend)
│   ├── api/                   # FastAPI server
│   ├── webui/                 # LlamaBoard (Gradio)
│   ├── eval/                  # Evaluation (deprecated)
│   ├── extras/                # Utilities
│   └── v1/                    # V1 plugin-based engine
│       ├── core/              # DataEngine, ModelEngine, BaseTrainer
│       ├── config/            # OmegaConf argument parsing
│       ├── plugins/           # All plugin implementations
│       ├── trainers/          # SFT, DPO, RM trainers
│       ├── samplers/          # CLI sampler
│       ├── accelerator/       # Distributed interface
│       └── utils/             # Plugin system, helpers
├── data/                      # Demo datasets + dataset_info.json
├── examples/                  # YAML configs for all scenarios
│   ├── deepspeed/             # DeepSpeed JSON configs
│   ├── accelerate/            # FSDP YAML configs
│   └── ...                    # Per-stage example configs
├── docker/                    # Dockerfiles (CUDA, ROCm, NPU)
├── scripts/                   # Utilities (FLOPs, MFU, PPL, checkpoint conversion)
├── tests/                     # Legacy test suite
└── tests_v1/                  # V1 test suite

9. Integration Points

9.1 OpenAI API Compatibility

The API server implements the OpenAI chat completions protocol, making it a drop-in replacement for applications using the OpenAI client library:

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="your-api-key"
)

response = client.chat.completions.create(
    model="my-model",
    messages=[{"role": "user", "content": "Hello!"}]
)

The API_MODEL_NAME environment variable controls what model name the server reports. Clients can use any model name in their requests; the server always uses the loaded model.

9.2 HuggingFace Hub Integration

• Models: Loaded directly from HuggingFace Hub via model_name_or_path / model
• Datasets: Referenced by Hub ID in dataset configs
• Adapters: Saved to and loaded from Hub-compatible format
• Checkpoints: save_ckpt_as_hf converts to Hub format

9.3 ModelScope and OpenMind Hubs

Alternative model/dataset sources activated via environment variables:

# Use ModelScope for model downloads
USE_MODELSCOPE_HUB=1 llamafactory-cli train config.yaml

# Use OpenMind for model downloads
USE_OPENMIND_HUB=1 llamafactory-cli train config.yaml

9.4 Gradio WebUI (LlamaBoard)

The WebUI provides a browser-based interface for:

• Training configuration and launch
• Model chat/evaluation
• Dataset preview

# Start WebUI
llamafactory-cli webui

# Or with custom binding
GRADIO_SERVER_NAME=0.0.0.0 llamafactory-cli webui

9.5 Tool Calling

The API supports tool/function calling in the OpenAI format. Example scripts are provided in scripts/api_example/ demonstrating tool calling and image input.

Appendix A: Troubleshooting

A.1 "can't start new thread" (RuntimeError)

Error: RuntimeError: can't start new thread

Cause: System thread limit reached, commonly when batching_workers is too high or the system ulimit for threads is restrictive.

Solution: Reduce batching_workers in training config. Increase the system thread limit: ulimit -u 65536.

Prevention: Set batching_workers proportional to available CPU cores, not arbitrarily high.

A.2 "backward through graph twice" with Unsloth + ZeRO-3

Error: RuntimeError about backward through the graph a second time, occurring with unsloth_gc enabled alongside Gemma4 and DeepSpeed ZeRO-3.

Cause: Incompatibility between Unsloth's gradient checkpointing and ZeRO-3's parameter partitioning for certain model architectures.

Solution: Disable unsloth_gc when using ZeRO-3 with affected models (Gemma4). Use standard gradient checkpointing instead.

Prevention: Test new model + optimizer + distributed combinations on small datasets before committing to full training runs.

A.3 "ignore_mismatched_sizes" breaking model loading

Error: Model loading fails or produces incorrect results when ignore_mismatched_sizes is enabled in the loader.

Cause: Silently loading mismatched tensor shapes corrupts the model rather than failing loudly.

Solution: Remove ignore_mismatched_sizes and fix the root cause (wrong model path, incompatible adapter, architecture mismatch).

Prevention: Always verify model and adapter compatibility before loading.

A.4 MLflow external logger failing

Error: MLflow integration produces errors or fails to log metrics.

Cause: MLflow configuration or connectivity issues with the tracking server.

Solution: Verify MLflow tracking URI is accessible. Check MLflow version compatibility with the HuggingFace Trainer version in use.

Prevention: Test MLflow connectivity independently before integrating with training.

A.5 GPU 0 barrier warning in distributed training

Error: Warning about GPU 0 waiting at a barrier during distributed training.

Cause: Uneven workload distribution or a process on one rank taking significantly longer.

Solution: Ensure all ranks have equal data. Check for CPU bottlenecks on specific nodes. Verify network connectivity between nodes.

Prevention: Use drop_last=True (mandatory in V1) to ensure even batch distribution.

A.6 Data packing error in SFT with multimodal models

Error: Data packing fails during SFT with models like Qwen3VL.

Cause: Packing (concatenating multiple examples into one sequence) conflicts with multimodal token handling.

Solution: Disable packing when training multimodal models.

Prevention: Review model-specific documentation for known training limitations.

A.7 "Plugin configuration must have a 'name' field"

Error: Raised during V1 engine config parsing.

Cause: A plugin config block (e.g., peft_config, dist_config) is specified without the required name field.

Solution: Add the name field identifying which plugin implementation to use:

peft_config:
  name: lora     # This field is mandatory
  r: 16

A.8 "Global batch size must be divisible by DP size and micro batch size"

Error: Raised at training start in V1 engine.

Cause: The global_batch_size is not evenly divisible by the product of data parallel size and micro_batch_size.

Solution: Either remove global_batch_size (let it auto-compute) or set it to a valid multiple. For example, with 4 GPUs and micro_batch_size: 2, valid global_batch_size values are 8, 16, 24, etc.

Tip

When troubleshooting distributed training issues, set LLAMAFACTORY_VERBOSITY=DEBUG and NPROC_PER_NODE=1 to isolate whether the issue is distributed-specific or a general training problem.