Unsloth — User Guide

Unsloth is a Python library, CLI tool, and local web UI for fine-tuning large language models. It delivers ~2x faster training with ~60% less VRAM through custom Triton kernels, supports LoRA/QLoRA/full fine-tuning, GGUF export for Ollama and llama.cpp, vLLM-backed fast inference, and a visual data recipe builder called Unsloth Studio. Version 2026.4.6. Licensed under Apache-2.0 (core library) and AGPL-3.0 (Studio, MoE kernels).

1. Quick Start

Get from zero to a fine-tuned model in under five minutes.

Step 1: Install Unsloth

# Linux / macOS
curl -fsSL https://unsloth.ai/install | bash

# Windows PowerShell
iex ((New-Object System.Net.WebClient).DownloadString('https://unsloth.ai/install.ps1'))

Expected output (Linux example):

Detecting GPU... NVIDIA CUDA 12.4 found
Installing uv package manager...
Creating virtual environment at ~/.unsloth/studio/unsloth_studio/
Installing unsloth + dependencies...
Done! Run 'unsloth studio' to launch.

Tip

The one-line installer auto-detects your GPU (NVIDIA, AMD, or Intel), installs the correct PyTorch version, and sets up a virtual environment. It is the recommended installation method for all platforms.

Step 2: Fine-Tune a Model (CLI)

unsloth train \
  --model_name unsloth/llama-3-8b \
  --dataset yahma/alpaca-cleaned \
  --load_in_4bit \
  --max_steps 400 \
  --save_model \
  --save_method merged_16bit

Expected output:

Unsloth: Loading unsloth/llama-3-8b with 4bit quantization...
Unsloth: Applying LoRA adapters (r=16, alpha=16)...
Training: 100%|████████████████████████| 400/400 [12:34<00:00]
Training Loss: 0.9823
Saving model to outputs/merged_16bit...
Done!

Step 3: Fine-Tune a Model (Python)

from unsloth import FastLanguageModel
from trl import SFTTrainer
from transformers import TrainingArguments

# 1. Load model
model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="unsloth/llama-3-8b",
    max_seq_length=2048,
    load_in_4bit=True,
)

# 2. Apply LoRA
model = FastLanguageModel.get_peft_model(
    model,
    r=16,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
                     "gate_proj", "up_proj", "down_proj"],
    lora_alpha=16,
    lora_dropout=0,
)

# 3. Train
trainer = SFTTrainer(
    model=model,
    tokenizer=tokenizer,
    train_dataset=dataset,  # your prepared dataset
    args=TrainingArguments(
        per_device_train_batch_size=2,
        gradient_accumulation_steps=4,
        warmup_steps=5,
        max_steps=400,
        learning_rate=2e-4,
        optim="adamw_8bit",
        output_dir="outputs",
        seed=3407,
    ),
)
trainer.train()

# 4. Save
model.save_pretrained_merged("outputs/merged_16bit", tokenizer, save_method="merged_16bit")

Step 4: Export to GGUF for Ollama

model.save_pretrained_gguf("outputs/gguf", tokenizer, quantization_method="q8_0")

Expected output:

Unsloth: Converting to GGUF q8_0 format...
Unsloth: Saved to outputs/gguf/unsloth.Q8_0.gguf

Then load in Ollama:

ollama create my-model -f outputs/gguf/Modelfile
ollama run my-model

Step 5: Launch Unsloth Studio

unsloth studio

Expected output:

Unsloth Studio running at http://0.0.0.0:8888
Admin password saved to ~/.unsloth/studio/.bootstrap_password

Open http://localhost:8888 in your browser to access the visual training interface.

2. Installation

2.1 System Requirements

Important

Unsloth requires a supported GPU. CPU-only training is not supported. For GGUF-only workflows (no training), use the --no-torch installer flag to skip GPU detection.

2.2 One-Line Installer (Recommended)

The one-line installer creates an isolated virtual environment at ~/.unsloth/studio/unsloth_studio/, auto-detects your GPU, installs the correct PyTorch build, and uses uv as the package manager (auto-installed if missing).

Linux / macOS:

curl -fsSL https://unsloth.ai/install | bash

Windows PowerShell:

iex ((New-Object System.Net.WebClient).DownloadString('https://unsloth.ai/install.ps1'))

Installer flags:

Warning

On Windows, user names with spaces in the home directory path can cause the installer to fail (see GitHub issue #4904). Workaround: set the USERPROFILE environment variable to a path without spaces, or install from source into a custom directory.

2.3 pip Install

pip install unsloth

This installs the core library. You must have PyTorch already installed with the correct CUDA/ROCm/XPU support for your GPU.

2.4 Install from Source

git clone https://github.com/unslothai/unsloth.git
cd unsloth
pip install -e .

This gives you an editable install for development or contributing.

2.5 Post-Install Verification

from unsloth import FastLanguageModel
print("Unsloth installed successfully!")

Expected output:

Unsloth installed successfully!

If you see import errors, check that PyTorch is installed and your GPU is detected:

import torch
print(torch.cuda.is_available())   # NVIDIA
print(torch.version.cuda)          # CUDA version

Expected output (NVIDIA example):

True
12.4

2.6 Upgrading

# Via pip
pip install --upgrade unsloth

# Studio update
unsloth studio update

2.7 GPU-Specific Notes

NVIDIA (CUDA): The most mature path. CUDA 11.8 through 13.0 are supported. The installer picks the correct PyTorch wheel automatically.

AMD (ROCm / HIP): Supported via PyTorch ROCm builds. Studio's llama.cpp integration requires building from source with HIP support, which can fail on some configurations (GitHub issue #5093).

Intel (XPU): Supported via PyTorch XPU builds. Less commonly tested than NVIDIA/AMD paths.

3. Configuration

3.1 YAML Configuration Files

Unsloth accepts YAML configuration files for repeatable training runs. Save the following as config.yaml and pass it to the CLI or load it programmatically.

model:
  model_name: unsloth/Qwen3-4B
  max_seq_length: 2048
  dtype: null
  load_in_4bit: true

data:
  format_type: auto  # auto, alpaca, chatml, sharegpt

training:
  training_type: lora  # lora or full
  learning_rate: 0.0002
  per_device_train_batch_size: 2
  gradient_accumulation_steps: 4
  warmup_steps: 5
  num_train_epochs: 3
  max_steps: -1
  gradient_checkpointing: unsloth
  seed: 3407
  output_dir: outputs
  optimizer: adamw_8bit

lora:
  lora_r: 64
  lora_alpha: 16
  lora_dropout: 0.0
  target_modules: q_proj,k_proj,v_proj,o_proj,gate_proj,up_proj,down_proj

logging:
  use_wandb: false
  use_tensorboard: false

Configuration Fields Reference

model section:

data section:

training section:

lora section:

logging section:

3.2 Environment Variables

3.3 LoRA Configuration Guide

LoRA rank (r) controls the capacity of the adapter. Higher values learn more but use more VRAM and risk overfitting on small datasets.

The target_modules parameter controls which layers receive LoRA adapters. The default set covers all attention projections plus the MLP gate/up/down projections, which is the recommended configuration for most use cases.

3.4 Chat Templates

Unsloth includes 26+ built-in chat templates. Set the template to match your base model's expected format:

from unsloth.chat_templates import get_chat_template

tokenizer = get_chat_template(
    tokenizer,
    chat_template="llama-3.1",
    system_message="You are a helpful assistant.",
)

Available templates:

See the Admin Guide for the complete chat template specification and customization options.

4. Core Usage

4.1 Training with LoRA / QLoRA

LoRA (Low-Rank Adaptation) fine-tunes a small number of adapter parameters while freezing the base model. QLoRA adds 4-bit quantization to further reduce VRAM.

Basic QLoRA Training (Python)

from unsloth import FastLanguageModel
from trl import SFTTrainer
from transformers import TrainingArguments
from datasets import load_dataset

# Load model with 4-bit quantization (QLoRA)
model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="unsloth/llama-3-8b",
    max_seq_length=2048,
    dtype=None,          # auto-detect
    load_in_4bit=True,   # QLoRA
)

# Apply LoRA adapters
model = FastLanguageModel.get_peft_model(
    model,
    r=16,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
                     "gate_proj", "up_proj", "down_proj"],
    lora_alpha=16,
    lora_dropout=0,
)

# Load and format dataset
dataset = load_dataset("yahma/alpaca-cleaned", split="train")

# Configure trainer
trainer = SFTTrainer(
    model=model,
    tokenizer=tokenizer,
    train_dataset=dataset,
    args=TrainingArguments(
        per_device_train_batch_size=2,
        gradient_accumulation_steps=4,
        warmup_steps=5,
        max_steps=400,
        learning_rate=2e-4,
        optim="adamw_8bit",
        output_dir="outputs",
        seed=3407,
    ),
)

# Train
trainer_stats = trainer.train()
print(f"Training loss: {trainer_stats.training_loss:.4f}")

Expected output:

Unsloth: Loading unsloth/llama-3-8b with 4bit quantization...
Unsloth: Applying LoRA adapters...
Training: 100%|████████████████████████| 400/400 [12:34<00:00]
Training loss: 0.9823

Basic QLoRA Training (CLI)

unsloth train \
  --model_name unsloth/llama-3-8b \
  --dataset yahma/alpaca-cleaned \
  --load_in_4bit \
  --r 16 \
  --lora_alpha 16 \
  --per_device_train_batch_size 2 \
  --gradient_accumulation_steps 4 \
  --learning_rate 0.0002 \
  --max_steps 400 \
  --optim adamw_8bit \
  --save_model

Tip

Set gradient_checkpointing to unsloth (the default) for ~60% VRAM savings compared to standard gradient checkpointing. This is Unsloth's custom implementation that recomputes activations more efficiently.

Training with Padding-Free Packing

Packing combines multiple short samples into single sequences, eliminating wasted padding tokens and improving throughput:

unsloth train \
  --model_name unsloth/llama-3-8b \
  --dataset yahma/alpaca-cleaned \
  --load_in_4bit \
  --packing \
  --max_steps 400 \
  --save_model

Training on Responses Only

Train the model only on response tokens, masking instruction/system tokens from the loss:

from unsloth.chat_templates import train_on_responses_only

trainer = SFTTrainer(
    model=model,
    tokenizer=tokenizer,
    train_dataset=dataset,
    args=training_args,
)

# Mask everything except responses
trainer = train_on_responses_only(
    trainer,
    instruction_part="<|start_header_id|>user<|end_header_id|>\n\n",
    response_part="<|start_header_id|>assistant<|end_header_id|>\n\n",
)

trainer.train()

4.2 Full Fine-Tuning

Full fine-tuning updates all model parameters. It requires significantly more VRAM than LoRA but can achieve better results on large datasets.

from unsloth import FastLanguageModel

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="unsloth/llama-3-8b",
    max_seq_length=2048,
    dtype=None,
    load_in_4bit=False,  # Full precision for full fine-tuning
)

# No get_peft_model call — train all parameters
trainer = SFTTrainer(
    model=model,
    tokenizer=tokenizer,
    train_dataset=dataset,
    args=TrainingArguments(
        per_device_train_batch_size=1,
        gradient_accumulation_steps=8,
        learning_rate=5e-5,
        max_steps=200,
        output_dir="outputs-full",
    ),
)
trainer.train()

Via YAML config:

training:
  training_type: full
  learning_rate: 0.00005

4.3 Vision Model Fine-Tuning

Fine-tune vision-language models (Llama 3.2 Vision, Qwen2-VL, Gemma 3):

from unsloth import FastVisionModel

model, tokenizer = FastVisionModel.from_pretrained(
    model_name="unsloth/Llama-3.2-11B-Vision-Instruct",
    max_seq_length=2048,
    load_in_4bit=True,
)

model = FastVisionModel.get_peft_model(
    model,
    r=16,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
                     "gate_proj", "up_proj", "down_proj"],
    lora_alpha=16,
)

Warning

GGUF export is not supported for vision models. You will see: "Unsloth: Sorry GGUF is currently not supported for vision models!" Use merged_16bit or lora save methods instead.

4.4 Sentence Transformer Fine-Tuning

Fine-tune BERT, ModernBERT, and other sentence transformers for embedding tasks:

from unsloth import FastSentenceTransformer

model, tokenizer = FastSentenceTransformer.from_pretrained(
    model_name="BAAI/bge-base-en-v1.5",
    max_seq_length=512,
)

model = FastSentenceTransformer.get_peft_model(
    model,
    r=16,
)

Note

Sentence transformer fine-tuning requires the sentence-transformers package: pip install sentence-transformers.

4.5 Inference

Switch to Inference Mode (Python)

After training, switch the model to inference mode for generation:

FastLanguageModel.for_inference(model)

inputs = tokenizer("What is machine learning?", return_tensors="pt").to("cuda")
outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Expected output:

What is machine learning?

Machine learning is a subset of artificial intelligence that enables systems
to learn and improve from experience without being explicitly programmed...

To switch back to training mode:

FastLanguageModel.for_training(model)

Inference via CLI

unsloth inference \
  --model_name unsloth/llama-3-8b \
  --load_in_4bit

Default inference parameters: temperature=0.7, top_p=0.9, top_k=40, max_new_tokens=256, repetition_penalty=1.1.

Inference via Studio Chat

Launch Studio and use the Chat tab to load a GGUF model for interactive inference with an OpenAI-compatible API endpoint. See section 4.8 for Studio details.

4.6 Exporting Models

Save as Merged 16-bit

Merges LoRA adapters into the base model at full precision:

model.save_pretrained_merged(
    "outputs/merged_16bit",
    tokenizer,
    save_method="merged_16bit",
)

Save as Merged 4-bit

Merges and quantizes to 4-bit:

model.save_pretrained_merged(
    "outputs/merged_4bit",
    tokenizer,
    save_method="merged_4bit",
)

Caution

Merging into 4-bit causes accuracy loss. The model warns: "Unsloth: Merging into 4bit will cause your model to lose accuracy..." Use merged_16bit when quality matters, or export to GGUF with a specific quantization level for a better quality/size tradeoff.

Save LoRA Adapters Only

Saves just the adapter weights (smallest file size):

model.save_pretrained_merged(
    "outputs/lora_adapters",
    tokenizer,
    save_method="lora",
)

Export to GGUF

Convert to GGUF format for use with Ollama, llama.cpp, and other local inference tools:

model.save_pretrained_gguf(
    "outputs/gguf",
    tokenizer,
    quantization_method="q8_0",
)

Expected output:

Unsloth: Converting to GGUF q8_0 format...
Unsloth: Saved to outputs/gguf/unsloth.Q8_0.gguf

Available GGUF quantization methods:

Important

The iq2 family of quantization types is NOT supported. Stick to the q and f types listed above.

Export via CLI

unsloth export \
  --save_method merged_16bit

unsloth export \
  --save_method gguf \
  --quantization q8_0

Push to HuggingFace Hub

# Push merged model
model.push_to_hub_merged(
    "your-username/my-model",
    tokenizer,
    token="hf_...",
    save_method="merged_16bit",
)

# Push GGUF
model.push_to_hub_gguf(
    "your-username/my-model-gguf",
    tokenizer,
    quantization_method="q8_0",
    token="hf_...",
)

Via CLI:

unsloth train \
  --model_name unsloth/llama-3-8b \
  --dataset yahma/alpaca-cleaned \
  --load_in_4bit \
  --max_steps 400 \
  --push_model \
  --hub_path your-username/my-model \
  --hub_token hf_...

4.7 Data Preparation

Alpaca Format

The simplest format — instruction/input/output triples:

from datasets import load_dataset

dataset = load_dataset("yahma/alpaca-cleaned", split="train")
# Columns: instruction, input, output

ShareGPT Format

Multi-turn conversations:

from unsloth.chat_templates import to_sharegpt, standardize_data_formats, apply_chat_template

# Convert dataset to ShareGPT format
dataset = to_sharegpt(dataset, merged_prompt="{instruction}\n{input}")

# Standardize
dataset = standardize_data_formats(dataset)

# Apply chat template
dataset = apply_chat_template(dataset, tokenizer)

Applying Chat Templates to Datasets

from unsloth.chat_templates import get_chat_template

tokenizer = get_chat_template(
    tokenizer,
    chat_template="llama-3.1",
    mapping={"role": "from", "content": "value", "user": "human", "assistant": "gpt"},
    system_message="You are a helpful assistant.",
)

Raw Text Training (Continual Pretraining)

Train on unstructured text files:

from unsloth import RawTextDataLoader

dataloader = RawTextDataLoader(
    file_path="my_corpus.txt",
    tokenizer=tokenizer,
    max_seq_length=2048,
)

Via CLI:

unsloth train \
  --model_name unsloth/llama-3-8b \
  --raw_text_file my_corpus.txt \
  --load_in_4bit \
  --max_steps 400

Text Preprocessing

Clean and validate text data before training:

from unsloth import TextPreprocessor

preprocessor = TextPreprocessor()
cleaned_text = preprocessor.clean(raw_text)

Constructing Custom Chat Templates

Build both an Ollama-compatible modelfile and a HuggingFace Jinja template:

from unsloth.chat_templates import construct_chat_template

chat_template_str = construct_chat_template(
    tokenizer,
    chat_template="llama-3.1",
    default_system_message="You are a helpful assistant.",
    extra_eos_tokens=None,
)

4.8 Unsloth Studio

Unsloth Studio is a local web UI for training, inference, export, and data recipe building.

Launching Studio

unsloth studio

Expected output:

Unsloth Studio running at http://0.0.0.0:8888

Studio flags:

One-liner to start and immediately load a model:

unsloth studio run

First-Time Setup

On first launch, Studio creates a default admin account. The bootstrap admin password is saved to:

~/.unsloth/studio/.bootstrap_password

An onboarding wizard guides you through model selection, dataset choice, and hyperparameter configuration.

Training Tab

The Training tab provides a visual interface for fine-tuning:

• Select a base model from HuggingFace or a local path
• Choose a dataset
• Configure hyperparameters via form fields
• View live charts during training: training loss, eval loss, gradient norm, learning rate
• Browse training history

Chat Tab

The Chat tab loads GGUF models via llama.cpp for local inference:

• Load any GGUF model (HuggingFace or local file)
• OpenAI-compatible and Anthropic-compatible API endpoint
• Streaming responses
• Context usage bar showing token utilization
• Model selector dropdown

Export Tab

Export trained checkpoints in multiple formats:

• Merged 16-bit
• Merged 4-bit
• GGUF (select quantization method)
• LoRA adapters only
• Push directly to HuggingFace Hub

Data Recipes (Recipe Studio)

The visual node-based pipeline builder for synthetic data generation:

• Samplers: Category, uniform, gaussian, Bernoulli, datetime, UUID, person
• LLM Columns: Text generation, structured output, code generation, judge scoring
• Validators: Code validator, OXC validator
• Expressions: Custom transform expressions
• Seed Datasets: Load from HuggingFace, local files, or unstructured sources
• MCP Tool Integration: Connect external tools
• Execution Tracking: Monitor pipeline runs
• Publish: Push generated datasets to HuggingFace Hub

Studio API Access

Studio exposes an API for programmatic access. Generate API keys in Settings for automation.

Settings

• API key management
• Appearance: dark/light theme
• Chat settings
• Studio update instructions

Stopping Studio

unsloth studio stop

Resetting Studio Password

unsloth studio reset-password

4.9 Using the Generic FastModel Loader

For architectures not covered by FastLanguageModel or FastVisionModel, use the generic FastModel loader:

from unsloth import FastModel

model, tokenizer = FastModel.from_pretrained(
    model_name="some-org/some-model",
    max_seq_length=2048,
    load_in_4bit=True,
)

model = FastModel.get_peft_model(
    model,
    r=16,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
                     "gate_proj", "up_proj", "down_proj"],
)

FastModel falls back to transformers.AutoModel for loading, so it works with any transformers-compatible architecture.

4.10 UnslothTrainer and UnslothTrainingArguments

Unsloth provides enhanced versions of the standard HuggingFace trainer:

from unsloth import UnslothTrainer, UnslothTrainingArguments

args = UnslothTrainingArguments(
    per_device_train_batch_size=2,
    gradient_accumulation_steps=4,
    max_steps=400,
    learning_rate=2e-4,
    output_dir="outputs",
    embedding_learning_rate=1e-5,  # Separate LR for embeddings
)

trainer = UnslothTrainer(
    model=model,
    tokenizer=tokenizer,
    train_dataset=dataset,
    args=args,
)
trainer.train()

Additional parameters beyond standard TrainingArguments:

5. Common Tasks

5.1 Fine-Tune Llama 3 for Chat

from unsloth import FastLanguageModel
from unsloth.chat_templates import get_chat_template, apply_chat_template
from trl import SFTTrainer
from transformers import TrainingArguments
from datasets import load_dataset

# Load
model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="unsloth/llama-3-8b",
    max_seq_length=2048,
    load_in_4bit=True,
)

# LoRA
model = FastLanguageModel.get_peft_model(
    model, r=16,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
                     "gate_proj", "up_proj", "down_proj"],
)

# Chat template
tokenizer = get_chat_template(tokenizer, chat_template="llama-3")

# Dataset
dataset = load_dataset("yahma/alpaca-cleaned", split="train")
dataset = apply_chat_template(dataset, tokenizer)

# Train
trainer = SFTTrainer(
    model=model, tokenizer=tokenizer, train_dataset=dataset,
    args=TrainingArguments(
        per_device_train_batch_size=2,
        gradient_accumulation_steps=4,
        max_steps=400,
        learning_rate=2e-4,
        optim="adamw_8bit",
        output_dir="outputs",
    ),
)
trainer.train()

# Save
model.save_pretrained_merged("outputs/llama3-chat", tokenizer, save_method="merged_16bit")

5.2 Fine-Tune Qwen 3 for Chat

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="unsloth/Qwen3-4B",
    max_seq_length=2048,
    load_in_4bit=True,
)

model = FastLanguageModel.get_peft_model(model, r=16)
tokenizer = get_chat_template(tokenizer, chat_template="qwen-3")

5.3 Fine-Tune Gemma 4 for Chat

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="google/gemma-4-12b-it",
    max_seq_length=2048,
    load_in_4bit=True,
)

model = FastLanguageModel.get_peft_model(model, r=16)
tokenizer = get_chat_template(tokenizer, chat_template="gemma-4")

5.4 Fine-Tune Phi-4 for Chat

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="microsoft/phi-4",
    max_seq_length=2048,
    load_in_4bit=True,
)

model = FastLanguageModel.get_peft_model(model, r=16)
tokenizer = get_chat_template(tokenizer, chat_template="phi-4")

5.5 Fine-Tune Mistral for Chat

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="mistralai/Mistral-7B-v0.3",
    max_seq_length=2048,
    load_in_4bit=True,
)

model = FastLanguageModel.get_peft_model(model, r=16)
tokenizer = get_chat_template(tokenizer, chat_template="mistral")

5.6 Fine-Tune DeepSeek R1 for Reasoning

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="deepseek-ai/DeepSeek-R1-Distill-Llama-8B",
    max_seq_length=4096,
    load_in_4bit=True,
)

model = FastLanguageModel.get_peft_model(model, r=32)

5.7 Export to Ollama

Complete workflow from training to Ollama:

# After training...

# Export to GGUF
model.save_pretrained_gguf("outputs/gguf", tokenizer, quantization_method="q8_0")

The export generates a Modelfile in the output directory. Use it with Ollama:

cd outputs/gguf
ollama create my-finetuned-model -f Modelfile
ollama run my-finetuned-model

Expected output:

>>> Hello!
Hi there! I'm your fine-tuned assistant. How can I help you today?

5.8 Export Multiple GGUF Quantizations

Export several quantization levels to compare quality vs. size:

for quant in ["q4_k_m", "q5_k_m", "q8_0"]:
    model.save_pretrained_gguf(
        f"outputs/gguf_{quant}",
        tokenizer,
        quantization_method=quant,
    )

5.9 List Saved Checkpoints

unsloth list-checkpoints

Expected output:

Found checkpoints in outputs/:
  outputs/checkpoint-100/
  outputs/checkpoint-200/
  outputs/checkpoint-400/
  outputs/merged_16bit/

5.10 Use WandB for Training Monitoring

logging:
  use_wandb: true

Or set the environment variable:

export WANDB_API_KEY=your_key_here

Then training metrics will be logged to your Weights & Biases dashboard.

5.11 Use TensorBoard for Training Monitoring

logging:
  use_tensorboard: true

Then view logs:

tensorboard --logdir outputs/runs

5.12 Load a Gated HuggingFace Model

Some models require accepting license terms on HuggingFace. Set your token:

export HF_TOKEN=hf_your_token_here

Or pass it directly:

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="meta-llama/Llama-3-8B",
    max_seq_length=2048,
    load_in_4bit=True,
    token="hf_your_token_here",
)

5.13 Use ModelScope Instead of HuggingFace

For users in regions where HuggingFace is slow or blocked:

export UNSLOTH_USE_MODELSCOPE=True

Then use ModelScope model IDs in model_name.

5.14 Resume Training from a Checkpoint

trainer.train(resume_from_checkpoint="outputs/checkpoint-200")

Note

Checkpoint resumption with UnslothTrainer on some architectures (notably Gemma-4) has known issues. See GitHub issue #5082 if you encounter problems.

5.15 Train with Synthetic Data (Studio Data Recipes)

Use Studio's Recipe Studio to build synthetic datasets visually:

1. Launch Studio: unsloth studio
2. Navigate to the Data Recipes tab
3. Add a seed dataset node (HuggingFace, local, or unstructured)
4. Add sampler nodes to generate variety (category, gaussian, datetime, etc.)
5. Add LLM column nodes for text/code/structured generation
6. Add validator nodes to ensure quality
7. Connect nodes to build your pipeline
8. Execute the recipe and preview results
9. Publish the dataset to HuggingFace or save locally

6. Troubleshooting

Error: "Unsloth: Pytorch is not installed. Go to https://pytorch.org/."

Cause: PyTorch is not installed in the current Python environment, or it is installed but not importable.

Solution:

1. Activate your Unsloth virtual environment
2. Install PyTorch for your GPU:

   # NVIDIA CUDA 12.4
   pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu124
   

3. Verify: python -c "import torch; print(torch.__version__)"

Prevention: Use the one-line installer, which handles PyTorch installation automatically.

Error: "Unsloth currently only works on NVIDIA, AMD and Intel GPUs."

Cause: No supported GPU detected, or CUDA/ROCm/XPU drivers are not installed.

Solution:

1. Verify GPU drivers: nvidia-smi (NVIDIA) or rocm-smi (AMD)
2. Check PyTorch GPU support: python -c "import torch; print(torch.cuda.is_available())"
3. If drivers are installed but not detected, reinstall PyTorch with the correct CUDA version

Prevention: Ensure GPU drivers are installed before installing Unsloth.

Error: "llama-server binary not found — cannot load GGUF models. Run setup.sh to build it, or set LLAMA_SERVER_PATH."

Cause: Studio's Chat tab needs the llama-server binary (from llama.cpp) to load GGUF models, but it is not built or not on the expected path.

Solution:

1. Run the Studio setup script to build llama.cpp:

   cd ~/.unsloth/studio
   bash setup.sh
   

2. Or set the path manually:

   export LLAMA_SERVER_PATH=/path/to/llama-server
   

3. For Docker deployments, ensure the binary is included in the image (GitHub issue #5077)

Prevention: The one-line installer builds llama-server automatically. If building from source, ensure CMake and a C++ compiler are available.

Error: "Unsloth: Your tokenizer does not have an EOS token? Please provide one via extra_eos_tokens!"

Cause: The loaded tokenizer has no end-of-sequence token defined, which is required for training.

Solution:

1. Provide extra EOS tokens when constructing the chat template:

   chat_template_str = construct_chat_template(
       tokenizer,
       chat_template="chatml",
       extra_eos_tokens=["<|end|>"],
   )
   

2. Or set it directly on the tokenizer:

   tokenizer.eos_token = "<|endoftext|>"
   

Prevention: Use the built-in chat templates, which set EOS tokens automatically.

Error: "Unsloth: Merging into 4bit will cause your model to lose accuracy..."

Cause: You are saving with save_method="merged_4bit", which merges and requantizes the model.

Solution:

1. Use merged_16bit for highest quality:

   model.save_pretrained_merged("output", tokenizer, save_method="merged_16bit")
   

2. Or export to GGUF with a specific quantization level for a better quality/size tradeoff

Prevention: Only use merged_4bit when deployment constraints demand it.

Error: "Unsloth: Sorry GGUF is currently not supported for vision models!"

Cause: You attempted to export a vision-language model (loaded via FastVisionModel) to GGUF format.

Solution:

1. Save as merged 16-bit instead:

   model.save_pretrained_merged("output", tokenizer, save_method="merged_16bit")
   

2. Or save LoRA adapters only:

   model.save_pretrained_merged("output", tokenizer, save_method="lora")
   

Prevention: Use merged_16bit or lora save methods for vision models.

Error: "Unsloth: You must select one of 3 options when saving models: 'lora', 'merged_16bit', 'merged_4bit'"

Cause: An invalid save_method string was passed.

Solution:

1. Use one of the three valid save methods:

   model.save_pretrained_merged("output", tokenizer, save_method="merged_16bit")
   # or: "merged_4bit", "lora"
   

Prevention: Always pass one of the three documented save method strings.

Error: "Unsloth: Can only load in 4bit or 8bit or 16bit, not a combination!"

Cause: Conflicting quantization flags were passed.

Solution:

1. Choose exactly one quantization option:

   # 4-bit
   model, tokenizer = FastLanguageModel.from_pretrained(model_name, load_in_4bit=True)
   # or full precision
   model, tokenizer = FastLanguageModel.from_pretrained(model_name, load_in_4bit=False)
   

Prevention: Set only one quantization flag per model load.

Error: "Unsloth: Your model needs to call .get_peft_model first!"

Cause: You tried to call a save method that requires LoRA adapters, but get_peft_model() was never called.

Solution:

1. Apply LoRA before saving:

   model = FastLanguageModel.get_peft_model(model, r=16)
   # ... train ...
   model.save_pretrained_merged("output", tokenizer, save_method="merged_16bit")
   

Prevention: Always call get_peft_model() before training and saving with LoRA methods.

Error: transformers version incompatibility

Cause: Some model architectures require specific versions of the transformers library.

Solution:

1. Upgrade transformers:

   pip install --upgrade transformers
   

2. Check Unsloth's requirements for your model architecture
3. The one-line installer pins compatible versions automatically

Prevention: Use the one-line installer or pin versions in your requirements file.

Error: OOM (Out of Memory) during training

Cause: Insufficient GPU VRAM for the model + training configuration.

Solution:

1. Enable 4-bit quantization: load_in_4bit=True
2. Use Unsloth gradient checkpointing: gradient_checkpointing="unsloth"
3. Reduce batch size: per_device_train_batch_size=1
4. Increase gradient accumulation to compensate: gradient_accumulation_steps=8
5. Reduce sequence length: max_seq_length=1024
6. Use a smaller LoRA rank: r=8
7. For very large models (31B+), even 32 GB VRAM may not be enough with GRPO-style training (GitHub issue #4985)

Prevention: Start with load_in_4bit=True and small batch sizes, then increase if VRAM allows.

Error: Windows install.ps1 failure

Cause: Various Windows-specific issues including path problems and execution policy restrictions (GitHub issue #5073).

Solution:

1. Run PowerShell as Administrator
2. Set execution policy: Set-ExecutionPolicy -Scope CurrentUser RemoteSigned
3. Ensure your user profile path does not contain spaces (GitHub issue #4904)
4. Try installing from source as a fallback

Prevention: Use a Windows username without spaces. Run the installer from an Administrator PowerShell.

Error: Windows user names with spaces cause install failure

Cause: The installer constructs file paths using the Windows user profile directory, and spaces break the path parsing (GitHub issue #4904).

Solution:

1. Create a junction or symbolic link to a path without spaces:

   New-Item -ItemType Junction -Path "C:\unsloth_home" -Target "$env:USERPROFILE"
   $env:USERPROFILE = "C:\unsloth_home"
   

2. Then rerun the installer
3. Or install from source into a custom directory without spaces

Prevention: Use a Windows username without spaces.

Error: Studio HIP llama.cpp build failure (AMD ROCm)

Cause: Building llama.cpp with HIP (AMD ROCm) support fails due to missing dependencies or incompatible toolchain (GitHub issue #5093).

Solution:

1. Ensure ROCm is fully installed with development headers
2. Verify hipcc is in your PATH: which hipcc
3. Try building llama.cpp manually:

   cd ~/.unsloth/studio/llama.cpp
   cmake -B build -DGGML_HIP=ON
   cmake --build build --target llama-server
   

4. Set the path manually if the build succeeds elsewhere:

   export LLAMA_SERVER_PATH=/path/to/llama-server
   

Prevention: Install the full ROCm development toolkit before running the Unsloth installer.

Error: Failing to manually build llama.cpp

Cause: Missing build tools (CMake, C++ compiler) or incompatible CUDA toolkit version (GitHub issue #5086).

Solution:

1. Install build prerequisites:

   # Ubuntu/Debian
   sudo apt install cmake build-essential
   # macOS
   brew install cmake
   

2. Ensure CUDA toolkit matches your driver version
3. Check the llama.cpp build log for specific errors

Prevention: Install CMake and a C++ compiler before running the installer.

Error: Studio Docker — llama-server binary not found

Cause: The Docker image does not include a pre-built llama-server binary (GitHub issue #5077).

Solution:

1. Build llama-server inside the container or mount it from the host
2. Set LLAMA_SERVER_PATH environment variable in your Docker compose/run configuration

Prevention: Include llama-server in your Dockerfile build steps.

Error: IndexError in _prepare_dataset with SFTTrainer

Cause: Dataset formatting issue, particularly with large models like Gemma-4-31B (GitHub issue #4833).

Solution:

1. Verify your dataset has the expected columns and format
2. Try with a smaller sample first to isolate the issue:

   small_dataset = dataset.select(range(100))
   

3. Ensure your chat template matches the model

Prevention: Validate dataset format before training. Use standardize_data_formats() to normalize.

Error: Vision LoRA receives zero gradients

Cause: Known issue with some vision models where LoRA adapters on certain layers receive zero gradients (GitHub issue #5039).

Solution:

1. Verify target modules are correct for the vision model
2. Try adjusting the target modules list
3. Check for model-specific instructions in the Unsloth documentation

Prevention: Test with a small dataset and verify loss is decreasing before full training runs.

Error: Incorrect tokenizer behavior

Cause: Tokenizer mismatch between expected and actual format (GitHub issue #4996).

Solution:

1. Ensure the chat template matches the model:

   tokenizer = get_chat_template(tokenizer, chat_template="llama-3.1")
   

2. Verify tokenizer output:

   print(tokenizer.encode("Hello, world!"))
   print(tokenizer.decode([1, 15043, 29892, 3186, 29991]))
   

Prevention: Always apply the correct chat template for your model family.

Error: Gemma 3 GGUF export writes tokens as NORMAL instead of CONTROL

Cause: Token type metadata not correctly set during GGUF export for Gemma 3 models (GitHub issue #5070).

Solution:

1. This is a known issue — check for updates to Unsloth that may include a fix
2. As a workaround, manually patch the GGUF metadata after export if needed

Prevention: Monitor Unsloth releases for Gemma 3 GGUF fixes.

Error: Different behaviour between transformers and unsloth for Gemma-4

Cause: Unsloth's optimized kernels may produce slightly different outputs compared to the reference transformers implementation (GitHub issue #5087).

Solution:

1. For exact reference outputs, use the standard transformers path
2. For training, the differences are typically within acceptable tolerance
3. Report significant discrepancies as issues

Prevention: Validate outputs against reference implementations when precision matters.

Error: Unable to resume from existing checkpoint with UnslothTrainer

Cause: Checkpoint resumption with UnslothTrainer may fail on certain architectures (GitHub issue #5082).

Solution:

1. Ensure the checkpoint directory contains all required files
2. Try resuming with the standard SFTTrainer instead of UnslothTrainer
3. Verify the model architecture is fully supported for checkpoint resumption

Prevention: Save checkpoints frequently and test resumption early in long training runs.

Error: Audio dataset preview fails

Cause: Version mismatch with torchcodec library (GitHub issue #4900).

Solution:

1. Pin torchcodec to a compatible version:

   pip install torchcodec==0.1.0
   

2. Or skip audio preview and work with the dataset directly

Prevention: Pin audio processing library versions in your requirements.

Error: Studio Colab doesn't give correct URL

Cause: Colab's network routing does not expose the Studio server correctly (GitHub issue #4516).

Solution:

1. Use ngrok or localtunnel to expose the port:

   pip install pyngrok
   

   from pyngrok import ngrok
   public_url = ngrok.connect(8888)
   print(public_url)
   

Prevention: Use the recommended Colab setup instructions from the Unsloth documentation.

Error: "Copy" buttons in Studio don't work

Cause: Clipboard API requires HTTPS or localhost context; Studio running on a remote server over HTTP cannot access the clipboard (GitHub issue #5097).

Solution:

1. Access Studio via localhost (local machine) or HTTPS
2. Manually select and copy text as a workaround

Prevention: Access Studio via localhost or set up an HTTPS proxy.

Error: General install issues

Cause: Dependency conflicts, incompatible Python version, or network issues (GitHub issue #4983).

Solution:

1. Use the one-line installer (handles all dependencies)
2. Verify Python version: python --version (must be 3.11–3.13)
3. Create a fresh virtual environment:

   python -m venv unsloth-env
   source unsloth-env/bin/activate  # Linux/macOS
   # or: unsloth-env\Scripts\activate  # Windows
   pip install unsloth
   

4. If behind a corporate proxy, configure pip:

   pip install --proxy http://proxy:port unsloth
   

Prevention: Use a clean virtual environment with a supported Python version.

Issue: VRAM usage higher than expected

Cause: Gradient checkpointing is not using the Unsloth mode, or batch size is too large.

Solution:

1. Verify Unsloth gradient checkpointing is active:

   args = TrainingArguments(
       gradient_checkpointing=True,
       gradient_checkpointing_kwargs={"use_reentrant": False},
   )
   

2. Set gradient_checkpointing: unsloth in YAML config
3. Reduce per_device_train_batch_size to 1

Prevention: Always use gradient_checkpointing: unsloth in your configuration.

Issue: Training loss not decreasing

Cause: Learning rate too high or too low, dataset quality issues, or incorrect chat template.

Solution:

1. Try the default learning rate: 2e-4
2. Verify dataset format matches the expected template
3. Check that the correct chat template is applied
4. Ensure train_on_responses_only masks are correct (if used)
5. Try a smaller LoRA rank (8 or 16) to reduce overfitting risk

Prevention: Start with default hyperparameters and adjust one at a time.

Issue: Slow training speed

Cause: Unsloth optimizations not active, or suboptimal batch configuration.

Solution:

1. Verify Unsloth is actually loaded (check for "Unsloth:" prefixed log messages)
2. Enable packing for short sequences: --packing
3. Increase batch size if VRAM allows
4. Ensure you are using adamw_8bit optimizer

Prevention: Use the Unsloth-provided model loaders (FastLanguageModel, etc.) rather than loading directly with transformers.

Issue: Model outputs gibberish after training

Cause: Overfitting, wrong chat template, or too many training steps.

Solution:

1. Reduce max_steps or num_train_epochs
2. Verify the chat template matches the base model
3. Check training loss — if it drops below 0.1, you are likely overfitting
4. Use a larger dataset or apply regularization (increase lora_dropout slightly)

Prevention: Monitor training loss and stop early if it plateaus or drops too low.

Issue: Saved model does not load correctly

Cause: Incomplete save, wrong save method, or missing tokenizer files.

Solution:

1. Ensure both model and tokenizer are saved:

   model.save_pretrained_merged("output", tokenizer, save_method="merged_16bit")
   

2. Check the output directory contains all expected files
3. Try loading with the same Unsloth version used for saving

Prevention: Always pass the tokenizer when saving. Verify the output directory after saving.

7. FAQ

Q: What GPU do I need to run Unsloth?

You need an NVIDIA (CUDA 11.8–13.0), AMD (ROCm), or Intel (XPU) GPU. NVIDIA GPUs are the best-tested path. CPU-only training is not supported, though GGUF-only workflows can run on CPU with the --no-torch installer flag.

Q: How much VRAM do I need?

With 4-bit quantization and Unsloth gradient checkpointing, a 7B/8B model fits in ~6 GB VRAM with batch size 1. A 13B model needs ~10 GB. Larger models (30B+) typically require 24–48 GB or more. Unsloth reduces VRAM usage by ~60% compared to standard fine-tuning.

Q: What is the difference between LoRA and QLoRA?

LoRA (Low-Rank Adaptation) adds small trainable adapter matrices to frozen model weights. QLoRA adds 4-bit quantization of the base model on top, further reducing VRAM. In Unsloth, set load_in_4bit=True for QLoRA or load_in_4bit=False for standard LoRA.

Q: What is the difference between LoRA and full fine-tuning?

LoRA trains only adapter parameters (~1–5% of total), keeping the base model frozen. Full fine-tuning updates all parameters, requiring much more VRAM and compute but potentially achieving better results on large datasets. Use LoRA for most tasks; reserve full fine-tuning for domain-specific specialization with large datasets.

Q: What LoRA rank should I use?

Start with r=16 for general-purpose fine-tuning. Use r=8 for quick experiments or small datasets. Use r=32 or r=64 for complex tasks or large datasets. Higher ranks use more VRAM and increase overfitting risk on small datasets.

Q: What is padding-free packing?

Packing concatenates multiple training samples into single sequences, eliminating wasted padding tokens. This improves training throughput, especially when your dataset has variable-length examples. Enable it with --packing on the CLI or packing=True in the trainer.

Q: What dataset formats does Unsloth support?

Unsloth supports Alpaca format (instruction/input/output), ShareGPT format (multi-turn conversations), ChatML format, and raw text files for continual pretraining. The format_type: auto setting auto-detects the format.

Q: How do I use a custom dataset?

Load your dataset with HuggingFace datasets, then format it with Unsloth's data utilities:

from datasets import load_dataset
dataset = load_dataset("json", data_files="my_data.json", split="train")

Then apply the appropriate chat template or format conversion.

Q: What is the Verified Facts Sheet mentioned in the training output?

This is an internal Unsloth mechanism for ensuring training data quality. You do not need to interact with it directly.

Q: Can I fine-tune vision models?

Yes. Use FastVisionModel for Llama 3.2 Vision, Qwen2-VL, Gemma 3, and other vision-language models. Note that GGUF export is not supported for vision models.

Q: Can I fine-tune sentence transformers / embedding models?

Yes. Use FastSentenceTransformer for BERT, ModernBERT, and other sentence transformer models. Install sentence-transformers first.

Q: What models does Unsloth support?

Unsloth supports Llama (3.x), Mistral, Gemma (1–4), Gemma 3N, Qwen 2/2.5/3/3 MoE, Phi (3/3.5/4), DeepSeek (V3, R1), Cohere, Falcon H1, Granite, GLM-4 MoE, GPT-OSS, OLMo (2/3), Pixtral, Whisper, BERT/ModernBERT, and any transformers-compatible model via the FastModel fallback.

Q: What is Unsloth Studio?

Unsloth Studio is a local web UI for training, inference, export, and synthetic data generation. It runs on port 8888 by default and provides a visual alternative to the CLI and Python API. Launch it with unsloth studio.

Q: Is Unsloth Studio free?

The core Unsloth library is Apache-2.0 licensed and free. Unsloth Studio and MoE kernels are AGPL-3.0 licensed, which is also free for most uses but requires source disclosure if you distribute modified versions as a network service.

Q: How do I access Studio remotely?

Studio binds to 0.0.0.0 by default, making it accessible on your local network. For remote access over the internet, use SSH tunneling, a reverse proxy with HTTPS, or tools like ngrok.

Q: What is the Studio bootstrap password?

On first launch, Studio creates an admin account and saves the bootstrap password to ~/.unsloth/studio/.bootstrap_password. Use unsloth studio reset-password to reset it.

Q: Can I run Unsloth in Docker?

Yes. See the Admin Guide for Docker deployment instructions. Note that the llama-server binary must be built or mounted separately for Studio Chat functionality (GitHub issue #5077).

Q: Can I run Unsloth on Google Colab?

Yes, but Studio URL routing can be tricky on Colab (GitHub issue #4516). Use ngrok or localtunnel to expose the Studio port. The CLI and Python API work directly in Colab notebooks.

Q: How do I use Unsloth with Ollama?

Export your model to GGUF format, then use the generated Modelfile with ollama create. See section 5.7 for the complete workflow.

Q: How do I use Unsloth with llama.cpp?

Export to GGUF and use the .gguf file directly with llama.cpp's llama-server or llama-cli.

Q: What GGUF quantization should I choose?

For the best quality-size balance, use q8_0. For smaller deployments, q4_k_m or q5_k_m are good choices. Avoid q2_k unless extreme compression is required. The iq2 family is NOT supported.

Q: How do I monitor training progress?

Use WandB (use_wandb: true), TensorBoard (use_tensorboard: true), or Studio's Training tab which shows live loss charts. Training logs also print to the console.

Q: Can I train on multiple GPUs?

Unsloth focuses on single-GPU efficiency. For multi-GPU setups, use the standard HuggingFace training pipeline with data parallelism.

Q: What is the seed: 3407 default?

This is a well-known seed value in the ML community (from the paper "Torch.manual_seed(3407) is all you need"). It is a reasonable default for reproducibility.

Q: Can I use Unsloth with FSDP or DeepSpeed?

Unsloth's custom kernels are designed for single-GPU optimization. For distributed training, use standard PyTorch/HuggingFace parallelism strategies.

Q: How do I push my model to HuggingFace Hub?

Use model.push_to_hub_merged() or model.push_to_hub_gguf(). Set HF_TOKEN or pass token= directly. See section 4.6 for examples.

Q: What is gradient_checkpointing: unsloth?

Unsloth's custom gradient checkpointing implementation saves ~60% more VRAM than standard gradient checkpointing by recomputing activations more efficiently. Always use this setting.

Q: What is the embedding_learning_rate in UnslothTrainingArguments?

It allows you to set a separate (typically lower) learning rate for the embedding layers, which can improve training stability when the embeddings need gentle updates.

Q: What is Q-GaLore?

Q-GaLore is an advanced optimizer option available through UnslothTrainingArguments. It combines quantization with gradient low-rank projection for memory-efficient training. Configure it via the q_galore_config parameter.

Q: How do I train only on model responses, not instructions?

Use train_on_responses_only() to mask instruction tokens from the loss. See section 4.1 for the complete example.

Q: Can I use a raw text file for continual pretraining?

Yes. Use RawTextDataLoader in Python or --raw_text_file on the CLI. See section 4.7 for details.

Q: How do I construct a chat template for Ollama?

Use construct_chat_template() which generates both an Ollama-compatible modelfile and a HuggingFace Jinja template. See section 4.7 for the example.

Q: What is the max_seq_length parameter?

It sets the maximum token sequence length for training. Unsloth auto-applies RoPE scaling if the sequence length exceeds the model's native context window. Default is 2048.

Q: My training is using more VRAM than expected. What should I check?

First, verify load_in_4bit=True is set. Then confirm gradient_checkpointing="unsloth" is active. Reduce batch size to 1 and reduce max_seq_length. Lower the LoRA rank. See the troubleshooting section on OOM errors for a complete checklist.