# Unsloth — User Guide

Unsloth is a Python library, CLI tool, and local web UI for fine-tuning large language models. It delivers ~2x faster training with ~60% less VRAM through custom Triton kernels, supports LoRA/QLoRA/full fine-tuning, GGUF export for Ollama and llama.cpp, vLLM-backed fast inference, and a visual data recipe builder called Unsloth Studio. Version 2026.4.6. Licensed under Apache-2.0 (core library) and AGPL-3.0 (Studio, MoE kernels).

---

## 1. Quick Start

Get from zero to a fine-tuned model in under five minutes.

### Step 1: Install Unsloth

```bash
# Linux / macOS
curl -fsSL https://unsloth.ai/install | bash
```

```powershell
# Windows PowerShell
iex ((New-Object System.Net.WebClient).DownloadString('https://unsloth.ai/install.ps1'))
```

Expected output (Linux example):

```text
Detecting GPU... NVIDIA CUDA 12.4 found
Installing uv package manager...
Creating virtual environment at ~/.unsloth/studio/unsloth_studio/
Installing unsloth + dependencies...
Done! Run 'unsloth studio' to launch.
```

> [!TIP]
> The one-line installer auto-detects your GPU (NVIDIA, AMD, or Intel), installs the correct PyTorch version, and sets up a virtual environment. It is the recommended installation method for all platforms.

### Step 2: Fine-Tune a Model (CLI)

```bash
unsloth train \
  --model_name unsloth/llama-3-8b \
  --dataset yahma/alpaca-cleaned \
  --load_in_4bit \
  --max_steps 400 \
  --save_model \
  --save_method merged_16bit
```

Expected output:

```text
Unsloth: Loading unsloth/llama-3-8b with 4bit quantization...
Unsloth: Applying LoRA adapters (r=16, alpha=16)...
Training: 100%|████████████████████████| 400/400 [12:34<00:00]
Training Loss: 0.9823
Saving model to outputs/merged_16bit...
Done!
```

### Step 3: Fine-Tune a Model (Python)

```python
from unsloth import FastLanguageModel
from trl import SFTTrainer
from transformers import TrainingArguments

# 1. Load model
model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="unsloth/llama-3-8b",
    max_seq_length=2048,
    load_in_4bit=True,
)

# 2. Apply LoRA
model = FastLanguageModel.get_peft_model(
    model,
    r=16,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
                     "gate_proj", "up_proj", "down_proj"],
    lora_alpha=16,
    lora_dropout=0,
)

# 3. Train
trainer = SFTTrainer(
    model=model,
    tokenizer=tokenizer,
    train_dataset=dataset,  # your prepared dataset
    args=TrainingArguments(
        per_device_train_batch_size=2,
        gradient_accumulation_steps=4,
        warmup_steps=5,
        max_steps=400,
        learning_rate=2e-4,
        optim="adamw_8bit",
        output_dir="outputs",
        seed=3407,
    ),
)
trainer.train()

# 4. Save
model.save_pretrained_merged("outputs/merged_16bit", tokenizer, save_method="merged_16bit")
```

### Step 4: Export to GGUF for Ollama

```python
model.save_pretrained_gguf("outputs/gguf", tokenizer, quantization_method="q8_0")
```

Expected output:

```text
Unsloth: Converting to GGUF q8_0 format...
Unsloth: Saved to outputs/gguf/unsloth.Q8_0.gguf
```

Then load in Ollama:

```bash
ollama create my-model -f outputs/gguf/Modelfile
ollama run my-model
```

### Step 5: Launch Unsloth Studio

```bash
unsloth studio
```

Expected output:

```text
Unsloth Studio running at http://0.0.0.0:8888
Admin password saved to ~/.unsloth/studio/.bootstrap_password
```

Open `http://localhost:8888` in your browser to access the visual training interface.

---

## 2. Installation

### 2.1 System Requirements

| Component | Requirement |
|-----------|-------------|
| Python | 3.11, 3.12, or 3.13 |
| GPU | NVIDIA (CUDA 11.8–13.0), AMD (ROCm), or Intel (XPU) |
| PyTorch | 2.1.0–2.10.x |
| Node.js | ≥20.19.0 (only for Studio frontend build) |
| OS | Linux, macOS, Windows |

> [!IMPORTANT]
> Unsloth requires a supported GPU. CPU-only training is not supported. For GGUF-only workflows (no training), use the `--no-torch` installer flag to skip GPU detection.

### 2.2 One-Line Installer (Recommended)

The one-line installer creates an isolated virtual environment at `~/.unsloth/studio/unsloth_studio/`, auto-detects your GPU, installs the correct PyTorch build, and uses `uv` as the package manager (auto-installed if missing).

**Linux / macOS:**

```bash
curl -fsSL https://unsloth.ai/install | bash
```

**Windows PowerShell:**

```powershell
iex ((New-Object System.Net.WebClient).DownloadString('https://unsloth.ai/install.ps1'))
```

**Installer flags:**

| Flag | Description |
|------|-------------|
| `--local` | Editable install from a local repo clone |
| `--no-torch` | CPU-only / GGUF-only mode (skip PyTorch + GPU detection) |
| `--package NAME` | Install a specific extra package alongside Unsloth |

> [!WARNING]
> On Windows, user names with spaces in the home directory path can cause the installer to fail (see GitHub issue #4904). Workaround: set the `USERPROFILE` environment variable to a path without spaces, or install from source into a custom directory.

### 2.3 pip Install

```bash
pip install unsloth
```

This installs the core library. You must have PyTorch already installed with the correct CUDA/ROCm/XPU support for your GPU.

### 2.4 Install from Source

```bash
git clone https://github.com/unslothai/unsloth.git
cd unsloth
pip install -e .
```

This gives you an editable install for development or contributing.

### 2.5 Post-Install Verification

```python
from unsloth import FastLanguageModel
print("Unsloth installed successfully!")
```

Expected output:

```text
Unsloth installed successfully!
```

If you see import errors, check that PyTorch is installed and your GPU is detected:

```python
import torch
print(torch.cuda.is_available())   # NVIDIA
print(torch.version.cuda)          # CUDA version
```

Expected output (NVIDIA example):

```text
True
12.4
```

### 2.6 Upgrading

```bash
# Via pip
pip install --upgrade unsloth

# Studio update
unsloth studio update
```

### 2.7 GPU-Specific Notes

**NVIDIA (CUDA):** The most mature path. CUDA 11.8 through 13.0 are supported. The installer picks the correct PyTorch wheel automatically.

**AMD (ROCm / HIP):** Supported via PyTorch ROCm builds. Studio's llama.cpp integration requires building from source with HIP support, which can fail on some configurations (GitHub issue #5093).

**Intel (XPU):** Supported via PyTorch XPU builds. Less commonly tested than NVIDIA/AMD paths.

---

## 3. Configuration

### 3.1 YAML Configuration Files

Unsloth accepts YAML configuration files for repeatable training runs. Save the following as `config.yaml` and pass it to the CLI or load it programmatically.

```yaml
model:
  model_name: unsloth/Qwen3-4B
  max_seq_length: 2048
  dtype: null
  load_in_4bit: true

data:
  format_type: auto  # auto, alpaca, chatml, sharegpt

training:
  training_type: lora  # lora or full
  learning_rate: 0.0002
  per_device_train_batch_size: 2
  gradient_accumulation_steps: 4
  warmup_steps: 5
  num_train_epochs: 3
  max_steps: -1
  gradient_checkpointing: unsloth
  seed: 3407
  output_dir: outputs
  optimizer: adamw_8bit

lora:
  lora_r: 64
  lora_alpha: 16
  lora_dropout: 0.0
  target_modules: q_proj,k_proj,v_proj,o_proj,gate_proj,up_proj,down_proj

logging:
  use_wandb: false
  use_tensorboard: false
```

#### Configuration Fields Reference

**model section:**

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `model_name` | string | `unsloth/llama-3-8b` | HuggingFace model name or local path |
| `max_seq_length` | int | `2048` | Maximum sequence length. Unsloth auto-applies RoPE scaling if needed |
| `dtype` | string/null | `null` | Data type. `null` for auto-detection (recommended) |
| `load_in_4bit` | bool | `true` | Load with 4-bit BitsAndBytes NF4 quantization |

**data section:**

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `format_type` | string | `auto` | Dataset format: `auto`, `alpaca`, `chatml`, or `sharegpt` |

**training section:**

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `training_type` | string | `lora` | `lora` for LoRA/QLoRA, `full` for full fine-tuning |
| `learning_rate` | float | `0.0002` | Learning rate |
| `per_device_train_batch_size` | int | `2` | Batch size per GPU |
| `gradient_accumulation_steps` | int | `4` | Steps to accumulate before optimizer update |
| `warmup_steps` | int | `5` | Linear warmup steps |
| `num_train_epochs` | int | `3` | Number of training epochs (ignored if `max_steps > 0`) |
| `max_steps` | int | `-1` | Override epoch count with fixed step count. `-1` means use epochs |
| `gradient_checkpointing` | string | `unsloth` | Set to `unsloth` for 60% VRAM savings |
| `seed` | int | `3407` | Random seed for reproducibility |
| `output_dir` | string | `outputs` | Directory for checkpoints and final model |
| `optimizer` | string | `adamw_8bit` | Optimizer type |

**lora section:**

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `lora_r` | int | `64` | LoRA rank. Common values: 8, 16, 32, 64, 128 |
| `lora_alpha` | int | `16` | LoRA alpha scaling parameter |
| `lora_dropout` | float | `0.0` | Dropout rate for LoRA layers |
| `target_modules` | string | See above | Comma-separated list of modules to apply LoRA to |

**logging section:**

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `use_wandb` | bool | `false` | Enable Weights & Biases logging |
| `use_tensorboard` | bool | `false` | Enable TensorBoard logging |

### 3.2 Environment Variables

| Variable | Default | Description |
|----------|---------|-------------|
| `HF_TOKEN` | unset | HuggingFace Hub token for gated models and pushing |
| `UNSLOTH_USE_MODELSCOPE` | `False` | Use ModelScope instead of HuggingFace Hub for model downloads |
| `UNSLOTH_STRICT_CHAT_TEMPLATE` | `0` | Set to `1` to enforce strict chat template validation |
| `UNSLOTH_RETURN_LOGITS` | `0` | Set to `1` to return logits during training |
| `UNSLOTH_DISABLE_AUTO_PADDING_FREE` | unset | Set to disable automatic padding-free training |
| `UNSLOTH_ENABLE_FLEX_ATTENTION` | `1` | Enable flex attention (enabled by default) |
| `UNSLOTH_ENABLE_LOGGING` | `0` | Set to `1` for verbose internal logging |

### 3.3 LoRA Configuration Guide

LoRA rank (`r`) controls the capacity of the adapter. Higher values learn more but use more VRAM and risk overfitting on small datasets.

| LoRA Rank | VRAM Impact | Best For |
|-----------|-------------|----------|
| 8 | Minimal | Small datasets, quick experiments |
| 16 | Low | General-purpose fine-tuning (default) |
| 32 | Moderate | Larger datasets, more complex tasks |
| 64 | Higher | Domain-specific specialization |
| 128 | Significant | Maximum adapter capacity |

The `target_modules` parameter controls which layers receive LoRA adapters. The default set covers all attention projections plus the MLP gate/up/down projections, which is the recommended configuration for most use cases.

### 3.4 Chat Templates

Unsloth includes 26+ built-in chat templates. Set the template to match your base model's expected format:

```python
from unsloth.chat_templates import get_chat_template

tokenizer = get_chat_template(
    tokenizer,
    chat_template="llama-3.1",
    system_message="You are a helpful assistant.",
)
```

**Available templates:**

| Template Name | Models |
|--------------|--------|
| `unsloth` | Unsloth default |
| `llama-1`, `llama-2`, `llama-3`, `llama-3.1`, `llama-3.2`, `llama-3.3` | Meta Llama family |
| `mistral` | Mistral models |
| `gemma-1`, `gemma-2`, `gemma-3`, `gemma-3n`, `gemma-4`, `gemma-4-thinking` | Google Gemma family |
| `phi-3`, `phi-3.5`, `phi-4` | Microsoft Phi family |
| `qwen-2.5`, `qwen-3`, `qwen3-instruct`, `qwen3-thinking` | Alibaba Qwen family |
| `chatml` | ChatML format (many models) |
| `zephyr` | Zephyr-style models |
| `vicuna`, `vicuna_old` | Vicuna models |
| `alpaca` | Alpaca-style models |
| `starling` | Starling models |
| `yi-chat` | Yi models |
| `gpt-oss` | GPT-OSS models |
| `lfm-2`, `lfm-2.5` | LFM models |
| `gemma_chatml`, `gemma2_chatml` | Gemma with ChatML wrapper |

See the Admin Guide for the complete chat template specification and customization options.

---

## 4. Core Usage

### 4.1 Training with LoRA / QLoRA

LoRA (Low-Rank Adaptation) fine-tunes a small number of adapter parameters while freezing the base model. QLoRA adds 4-bit quantization to further reduce VRAM.

#### Basic QLoRA Training (Python)

```python
from unsloth import FastLanguageModel
from trl import SFTTrainer
from transformers import TrainingArguments
from datasets import load_dataset

# Load model with 4-bit quantization (QLoRA)
model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="unsloth/llama-3-8b",
    max_seq_length=2048,
    dtype=None,          # auto-detect
    load_in_4bit=True,   # QLoRA
)

# Apply LoRA adapters
model = FastLanguageModel.get_peft_model(
    model,
    r=16,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
                     "gate_proj", "up_proj", "down_proj"],
    lora_alpha=16,
    lora_dropout=0,
)

# Load and format dataset
dataset = load_dataset("yahma/alpaca-cleaned", split="train")

# Configure trainer
trainer = SFTTrainer(
    model=model,
    tokenizer=tokenizer,
    train_dataset=dataset,
    args=TrainingArguments(
        per_device_train_batch_size=2,
        gradient_accumulation_steps=4,
        warmup_steps=5,
        max_steps=400,
        learning_rate=2e-4,
        optim="adamw_8bit",
        output_dir="outputs",
        seed=3407,
    ),
)

# Train
trainer_stats = trainer.train()
print(f"Training loss: {trainer_stats.training_loss:.4f}")
```

Expected output:

```text
Unsloth: Loading unsloth/llama-3-8b with 4bit quantization...
Unsloth: Applying LoRA adapters...
Training: 100%|████████████████████████| 400/400 [12:34<00:00]
Training loss: 0.9823
```

#### Basic QLoRA Training (CLI)

```bash
unsloth train \
  --model_name unsloth/llama-3-8b \
  --dataset yahma/alpaca-cleaned \
  --load_in_4bit \
  --r 16 \
  --lora_alpha 16 \
  --per_device_train_batch_size 2 \
  --gradient_accumulation_steps 4 \
  --learning_rate 0.0002 \
  --max_steps 400 \
  --optim adamw_8bit \
  --save_model
```

> [!TIP]
> Set `gradient_checkpointing` to `unsloth` (the default) for ~60% VRAM savings compared to standard gradient checkpointing. This is Unsloth's custom implementation that recomputes activations more efficiently.

#### Training with Padding-Free Packing

Packing combines multiple short samples into single sequences, eliminating wasted padding tokens and improving throughput:

```bash
unsloth train \
  --model_name unsloth/llama-3-8b \
  --dataset yahma/alpaca-cleaned \
  --load_in_4bit \
  --packing \
  --max_steps 400 \
  --save_model
```

#### Training on Responses Only

Train the model only on response tokens, masking instruction/system tokens from the loss:

```python
from unsloth.chat_templates import train_on_responses_only

trainer = SFTTrainer(
    model=model,
    tokenizer=tokenizer,
    train_dataset=dataset,
    args=training_args,
)

# Mask everything except responses
trainer = train_on_responses_only(
    trainer,
    instruction_part="<|start_header_id|>user<|end_header_id|>\n\n",
    response_part="<|start_header_id|>assistant<|end_header_id|>\n\n",
)

trainer.train()
```

### 4.2 Full Fine-Tuning

Full fine-tuning updates all model parameters. It requires significantly more VRAM than LoRA but can achieve better results on large datasets.

```python
from unsloth import FastLanguageModel

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="unsloth/llama-3-8b",
    max_seq_length=2048,
    dtype=None,
    load_in_4bit=False,  # Full precision for full fine-tuning
)

# No get_peft_model call — train all parameters
trainer = SFTTrainer(
    model=model,
    tokenizer=tokenizer,
    train_dataset=dataset,
    args=TrainingArguments(
        per_device_train_batch_size=1,
        gradient_accumulation_steps=8,
        learning_rate=5e-5,
        max_steps=200,
        output_dir="outputs-full",
    ),
)
trainer.train()
```

Via YAML config:

```yaml
training:
  training_type: full
  learning_rate: 0.00005
```

### 4.3 Vision Model Fine-Tuning

Fine-tune vision-language models (Llama 3.2 Vision, Qwen2-VL, Gemma 3):

```python
from unsloth import FastVisionModel

model, tokenizer = FastVisionModel.from_pretrained(
    model_name="unsloth/Llama-3.2-11B-Vision-Instruct",
    max_seq_length=2048,
    load_in_4bit=True,
)

model = FastVisionModel.get_peft_model(
    model,
    r=16,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
                     "gate_proj", "up_proj", "down_proj"],
    lora_alpha=16,
)
```

> [!WARNING]
> GGUF export is not supported for vision models. You will see: `"Unsloth: Sorry GGUF is currently not supported for vision models!"` Use `merged_16bit` or `lora` save methods instead.

### 4.4 Sentence Transformer Fine-Tuning

Fine-tune BERT, ModernBERT, and other sentence transformers for embedding tasks:

```python
from unsloth import FastSentenceTransformer

model, tokenizer = FastSentenceTransformer.from_pretrained(
    model_name="BAAI/bge-base-en-v1.5",
    max_seq_length=512,
)

model = FastSentenceTransformer.get_peft_model(
    model,
    r=16,
)
```

> [!NOTE]
> Sentence transformer fine-tuning requires the `sentence-transformers` package: `pip install sentence-transformers`.

### 4.5 Inference

#### Switch to Inference Mode (Python)

After training, switch the model to inference mode for generation:

```python
FastLanguageModel.for_inference(model)

inputs = tokenizer("What is machine learning?", return_tensors="pt").to("cuda")
outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```

Expected output:

```text
What is machine learning?

Machine learning is a subset of artificial intelligence that enables systems
to learn and improve from experience without being explicitly programmed...
```

To switch back to training mode:

```python
FastLanguageModel.for_training(model)
```

#### Inference via CLI

```bash
unsloth inference \
  --model_name unsloth/llama-3-8b \
  --load_in_4bit
```

Default inference parameters: `temperature=0.7`, `top_p=0.9`, `top_k=40`, `max_new_tokens=256`, `repetition_penalty=1.1`.

#### Inference via Studio Chat

Launch Studio and use the Chat tab to load a GGUF model for interactive inference with an OpenAI-compatible API endpoint. See section 4.8 for Studio details.

### 4.6 Exporting Models

#### Save as Merged 16-bit

Merges LoRA adapters into the base model at full precision:

```python
model.save_pretrained_merged(
    "outputs/merged_16bit",
    tokenizer,
    save_method="merged_16bit",
)
```

#### Save as Merged 4-bit

Merges and quantizes to 4-bit:

```python
model.save_pretrained_merged(
    "outputs/merged_4bit",
    tokenizer,
    save_method="merged_4bit",
)
```

> [!CAUTION]
> Merging into 4-bit causes accuracy loss. The model warns: `"Unsloth: Merging into 4bit will cause your model to lose accuracy..."` Use `merged_16bit` when quality matters, or export to GGUF with a specific quantization level for a better quality/size tradeoff.

#### Save LoRA Adapters Only

Saves just the adapter weights (smallest file size):

```python
model.save_pretrained_merged(
    "outputs/lora_adapters",
    tokenizer,
    save_method="lora",
)
```

#### Export to GGUF

Convert to GGUF format for use with Ollama, llama.cpp, and other local inference tools:

```python
model.save_pretrained_gguf(
    "outputs/gguf",
    tokenizer,
    quantization_method="q8_0",
)
```

Expected output:

```text
Unsloth: Converting to GGUF q8_0 format...
Unsloth: Saved to outputs/gguf/unsloth.Q8_0.gguf
```

**Available GGUF quantization methods:**

| Method | Size | Quality | Use Case |
|--------|------|---------|----------|
| `f32` | Largest | Highest | Research, debugging |
| `f16` | Very large | Very high | High-quality inference |
| `bf16` | Very large | Very high | BFloat16 systems |
| `q8_0` | Large | High | Best quality-size balance |
| `q6_k` | Medium-large | Good | Quality-focused deployment |
| `q5_k_m` | Medium | Good | Balanced deployment |
| `q5_k_s` | Medium | Good | Slightly smaller than q5_k_m |
| `q5_0` | Medium | Good | Alternative q5 format |
| `q4_k_m` | Small | Decent | Memory-constrained deployment |
| `q4_k_s` | Small | Decent | Slightly smaller than q4_k_m |
| `q4_0` | Small | Decent | Alternative q4 format |
| `q3_k_l` | Very small | Lower | Aggressive compression |
| `q3_k_m` | Very small | Lower | More aggressive compression |
| `q3_k_s` | Very small | Lower | Most aggressive q3 |
| `q2_k` | Smallest | Lowest | Extreme compression only |

> [!IMPORTANT]
> The `iq2` family of quantization types is NOT supported. Stick to the `q` and `f` types listed above.

#### Export via CLI

```bash
unsloth export \
  --save_method merged_16bit

unsloth export \
  --save_method gguf \
  --quantization q8_0
```

#### Push to HuggingFace Hub

```python
# Push merged model
model.push_to_hub_merged(
    "your-username/my-model",
    tokenizer,
    token="hf_...",
    save_method="merged_16bit",
)

# Push GGUF
model.push_to_hub_gguf(
    "your-username/my-model-gguf",
    tokenizer,
    quantization_method="q8_0",
    token="hf_...",
)
```

Via CLI:

```bash
unsloth train \
  --model_name unsloth/llama-3-8b \
  --dataset yahma/alpaca-cleaned \
  --load_in_4bit \
  --max_steps 400 \
  --push_model \
  --hub_path your-username/my-model \
  --hub_token hf_...
```

### 4.7 Data Preparation

#### Alpaca Format

The simplest format — instruction/input/output triples:

```python
from datasets import load_dataset

dataset = load_dataset("yahma/alpaca-cleaned", split="train")
# Columns: instruction, input, output
```

#### ShareGPT Format

Multi-turn conversations:

```python
from unsloth.chat_templates import to_sharegpt, standardize_data_formats, apply_chat_template

# Convert dataset to ShareGPT format
dataset = to_sharegpt(dataset, merged_prompt="{instruction}\n{input}")

# Standardize
dataset = standardize_data_formats(dataset)

# Apply chat template
dataset = apply_chat_template(dataset, tokenizer)
```

#### Applying Chat Templates to Datasets

```python
from unsloth.chat_templates import get_chat_template

tokenizer = get_chat_template(
    tokenizer,
    chat_template="llama-3.1",
    mapping={"role": "from", "content": "value", "user": "human", "assistant": "gpt"},
    system_message="You are a helpful assistant.",
)
```

#### Raw Text Training (Continual Pretraining)

Train on unstructured text files:

```python
from unsloth import RawTextDataLoader

dataloader = RawTextDataLoader(
    file_path="my_corpus.txt",
    tokenizer=tokenizer,
    max_seq_length=2048,
)
```

Via CLI:

```bash
unsloth train \
  --model_name unsloth/llama-3-8b \
  --raw_text_file my_corpus.txt \
  --load_in_4bit \
  --max_steps 400
```

#### Text Preprocessing

Clean and validate text data before training:

```python
from unsloth import TextPreprocessor

preprocessor = TextPreprocessor()
cleaned_text = preprocessor.clean(raw_text)
```

#### Constructing Custom Chat Templates

Build both an Ollama-compatible modelfile and a HuggingFace Jinja template:

```python
from unsloth.chat_templates import construct_chat_template

chat_template_str = construct_chat_template(
    tokenizer,
    chat_template="llama-3.1",
    default_system_message="You are a helpful assistant.",
    extra_eos_tokens=None,
)
```

### 4.8 Unsloth Studio

Unsloth Studio is a local web UI for training, inference, export, and data recipe building.

#### Launching Studio

```bash
unsloth studio
```

Expected output:

```text
Unsloth Studio running at http://0.0.0.0:8888
```

Studio flags:

| Flag | Default | Description |
|------|---------|-------------|
| `--host` | `0.0.0.0` | Host to bind to |
| `--port` | `8888` | Port (auto-increments if busy) |
| `--frontend` | (built-in) | Path to custom frontend build |
| `--silent` | off | Suppress console output |

One-liner to start and immediately load a model:

```bash
unsloth studio run
```

#### First-Time Setup

On first launch, Studio creates a default admin account. The bootstrap admin password is saved to:

```text
~/.unsloth/studio/.bootstrap_password
```

An onboarding wizard guides you through model selection, dataset choice, and hyperparameter configuration.

#### Training Tab

The Training tab provides a visual interface for fine-tuning:

- Select a base model from HuggingFace or a local path
- Choose a dataset
- Configure hyperparameters via form fields
- View live charts during training: training loss, eval loss, gradient norm, learning rate
- Browse training history

#### Chat Tab

The Chat tab loads GGUF models via llama.cpp for local inference:

- Load any GGUF model (HuggingFace or local file)
- OpenAI-compatible and Anthropic-compatible API endpoint
- Streaming responses
- Context usage bar showing token utilization
- Model selector dropdown

#### Export Tab

Export trained checkpoints in multiple formats:

- Merged 16-bit
- Merged 4-bit
- GGUF (select quantization method)
- LoRA adapters only
- Push directly to HuggingFace Hub

#### Data Recipes (Recipe Studio)

The visual node-based pipeline builder for synthetic data generation:

- **Samplers:** Category, uniform, gaussian, Bernoulli, datetime, UUID, person
- **LLM Columns:** Text generation, structured output, code generation, judge scoring
- **Validators:** Code validator, OXC validator
- **Expressions:** Custom transform expressions
- **Seed Datasets:** Load from HuggingFace, local files, or unstructured sources
- **MCP Tool Integration:** Connect external tools
- **Execution Tracking:** Monitor pipeline runs
- **Publish:** Push generated datasets to HuggingFace Hub

#### Studio API Access

Studio exposes an API for programmatic access. Generate API keys in Settings for automation.

#### Settings

- API key management
- Appearance: dark/light theme
- Chat settings
- Studio update instructions

#### Stopping Studio

```bash
unsloth studio stop
```

#### Resetting Studio Password

```bash
unsloth studio reset-password
```

### 4.9 Using the Generic FastModel Loader

For architectures not covered by `FastLanguageModel` or `FastVisionModel`, use the generic `FastModel` loader:

```python
from unsloth import FastModel

model, tokenizer = FastModel.from_pretrained(
    model_name="some-org/some-model",
    max_seq_length=2048,
    load_in_4bit=True,
)

model = FastModel.get_peft_model(
    model,
    r=16,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
                     "gate_proj", "up_proj", "down_proj"],
)
```

`FastModel` falls back to `transformers.AutoModel` for loading, so it works with any transformers-compatible architecture.

### 4.10 UnslothTrainer and UnslothTrainingArguments

Unsloth provides enhanced versions of the standard HuggingFace trainer:

```python
from unsloth import UnslothTrainer, UnslothTrainingArguments

args = UnslothTrainingArguments(
    per_device_train_batch_size=2,
    gradient_accumulation_steps=4,
    max_steps=400,
    learning_rate=2e-4,
    output_dir="outputs",
    embedding_learning_rate=1e-5,  # Separate LR for embeddings
)

trainer = UnslothTrainer(
    model=model,
    tokenizer=tokenizer,
    train_dataset=dataset,
    args=args,
)
trainer.train()
```

**Additional parameters beyond standard `TrainingArguments`:**

| Parameter | Description |
|-----------|-------------|
| `embedding_learning_rate` | Separate learning rate for embedding layers |
| `q_galore_config` | Configuration for Q-GaLore optimizer |

---

## 5. Common Tasks

### 5.1 Fine-Tune Llama 3 for Chat

```python
from unsloth import FastLanguageModel
from unsloth.chat_templates import get_chat_template, apply_chat_template
from trl import SFTTrainer
from transformers import TrainingArguments
from datasets import load_dataset

# Load
model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="unsloth/llama-3-8b",
    max_seq_length=2048,
    load_in_4bit=True,
)

# LoRA
model = FastLanguageModel.get_peft_model(
    model, r=16,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
                     "gate_proj", "up_proj", "down_proj"],
)

# Chat template
tokenizer = get_chat_template(tokenizer, chat_template="llama-3")

# Dataset
dataset = load_dataset("yahma/alpaca-cleaned", split="train")
dataset = apply_chat_template(dataset, tokenizer)

# Train
trainer = SFTTrainer(
    model=model, tokenizer=tokenizer, train_dataset=dataset,
    args=TrainingArguments(
        per_device_train_batch_size=2,
        gradient_accumulation_steps=4,
        max_steps=400,
        learning_rate=2e-4,
        optim="adamw_8bit",
        output_dir="outputs",
    ),
)
trainer.train()

# Save
model.save_pretrained_merged("outputs/llama3-chat", tokenizer, save_method="merged_16bit")
```

### 5.2 Fine-Tune Qwen 3 for Chat

```python
model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="unsloth/Qwen3-4B",
    max_seq_length=2048,
    load_in_4bit=True,
)

model = FastLanguageModel.get_peft_model(model, r=16)
tokenizer = get_chat_template(tokenizer, chat_template="qwen-3")
```

### 5.3 Fine-Tune Gemma 4 for Chat

```python
model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="google/gemma-4-12b-it",
    max_seq_length=2048,
    load_in_4bit=True,
)

model = FastLanguageModel.get_peft_model(model, r=16)
tokenizer = get_chat_template(tokenizer, chat_template="gemma-4")
```

### 5.4 Fine-Tune Phi-4 for Chat

```python
model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="microsoft/phi-4",
    max_seq_length=2048,
    load_in_4bit=True,
)

model = FastLanguageModel.get_peft_model(model, r=16)
tokenizer = get_chat_template(tokenizer, chat_template="phi-4")
```

### 5.5 Fine-Tune Mistral for Chat

```python
model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="mistralai/Mistral-7B-v0.3",
    max_seq_length=2048,
    load_in_4bit=True,
)

model = FastLanguageModel.get_peft_model(model, r=16)
tokenizer = get_chat_template(tokenizer, chat_template="mistral")
```

### 5.6 Fine-Tune DeepSeek R1 for Reasoning

```python
model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="deepseek-ai/DeepSeek-R1-Distill-Llama-8B",
    max_seq_length=4096,
    load_in_4bit=True,
)

model = FastLanguageModel.get_peft_model(model, r=32)
```

### 5.7 Export to Ollama

Complete workflow from training to Ollama:

```python
# After training...

# Export to GGUF
model.save_pretrained_gguf("outputs/gguf", tokenizer, quantization_method="q8_0")
```

The export generates a `Modelfile` in the output directory. Use it with Ollama:

```bash
cd outputs/gguf
ollama create my-finetuned-model -f Modelfile
ollama run my-finetuned-model
```

Expected output:

```text
>>> Hello!
Hi there! I'm your fine-tuned assistant. How can I help you today?
```

### 5.8 Export Multiple GGUF Quantizations

Export several quantization levels to compare quality vs. size:

```python
for quant in ["q4_k_m", "q5_k_m", "q8_0"]:
    model.save_pretrained_gguf(
        f"outputs/gguf_{quant}",
        tokenizer,
        quantization_method=quant,
    )
```

### 5.9 List Saved Checkpoints

```bash
unsloth list-checkpoints
```

Expected output:

```text
Found checkpoints in outputs/:
  outputs/checkpoint-100/
  outputs/checkpoint-200/
  outputs/checkpoint-400/
  outputs/merged_16bit/
```

### 5.10 Use WandB for Training Monitoring

```yaml
logging:
  use_wandb: true
```

Or set the environment variable:

```bash
export WANDB_API_KEY=your_key_here
```

Then training metrics will be logged to your Weights & Biases dashboard.

### 5.11 Use TensorBoard for Training Monitoring

```yaml
logging:
  use_tensorboard: true
```

Then view logs:

```bash
tensorboard --logdir outputs/runs
```

### 5.12 Load a Gated HuggingFace Model

Some models require accepting license terms on HuggingFace. Set your token:

```bash
export HF_TOKEN=hf_your_token_here
```

Or pass it directly:

```python
model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="meta-llama/Llama-3-8B",
    max_seq_length=2048,
    load_in_4bit=True,
    token="hf_your_token_here",
)
```

### 5.13 Use ModelScope Instead of HuggingFace

For users in regions where HuggingFace is slow or blocked:

```bash
export UNSLOTH_USE_MODELSCOPE=True
```

Then use ModelScope model IDs in `model_name`.

### 5.14 Resume Training from a Checkpoint

```python
trainer.train(resume_from_checkpoint="outputs/checkpoint-200")
```

> [!NOTE]
> Checkpoint resumption with `UnslothTrainer` on some architectures (notably Gemma-4) has known issues. See GitHub issue #5082 if you encounter problems.

### 5.15 Train with Synthetic Data (Studio Data Recipes)

Use Studio's Recipe Studio to build synthetic datasets visually:

1. Launch Studio: `unsloth studio`
2. Navigate to the Data Recipes tab
3. Add a seed dataset node (HuggingFace, local, or unstructured)
4. Add sampler nodes to generate variety (category, gaussian, datetime, etc.)
5. Add LLM column nodes for text/code/structured generation
6. Add validator nodes to ensure quality
7. Connect nodes to build your pipeline
8. Execute the recipe and preview results
9. Publish the dataset to HuggingFace or save locally

---

## 6. Troubleshooting

### Error: "Unsloth: Pytorch is not installed. Go to https://pytorch.org/."

**Cause:** PyTorch is not installed in the current Python environment, or it is installed but not importable.

**Solution:**
1. Activate your Unsloth virtual environment
2. Install PyTorch for your GPU:
   ```bash
   # NVIDIA CUDA 12.4
   pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu124
   ```
3. Verify: `python -c "import torch; print(torch.__version__)"`

**Prevention:** Use the one-line installer, which handles PyTorch installation automatically.

### Error: "Unsloth currently only works on NVIDIA, AMD and Intel GPUs."

**Cause:** No supported GPU detected, or CUDA/ROCm/XPU drivers are not installed.

**Solution:**
1. Verify GPU drivers: `nvidia-smi` (NVIDIA) or `rocm-smi` (AMD)
2. Check PyTorch GPU support: `python -c "import torch; print(torch.cuda.is_available())"`
3. If drivers are installed but not detected, reinstall PyTorch with the correct CUDA version

**Prevention:** Ensure GPU drivers are installed before installing Unsloth.

### Error: "llama-server binary not found — cannot load GGUF models. Run setup.sh to build it, or set LLAMA_SERVER_PATH."

**Cause:** Studio's Chat tab needs the llama-server binary (from llama.cpp) to load GGUF models, but it is not built or not on the expected path.

**Solution:**
1. Run the Studio setup script to build llama.cpp:
   ```bash
   cd ~/.unsloth/studio
   bash setup.sh
   ```
2. Or set the path manually:
   ```bash
   export LLAMA_SERVER_PATH=/path/to/llama-server
   ```
3. For Docker deployments, ensure the binary is included in the image (GitHub issue #5077)

**Prevention:** The one-line installer builds llama-server automatically. If building from source, ensure CMake and a C++ compiler are available.

### Error: "Unsloth: Your tokenizer does not have an EOS token? Please provide one via extra_eos_tokens!"

**Cause:** The loaded tokenizer has no end-of-sequence token defined, which is required for training.

**Solution:**
1. Provide extra EOS tokens when constructing the chat template:
   ```python
   chat_template_str = construct_chat_template(
       tokenizer,
       chat_template="chatml",
       extra_eos_tokens=["<|end|>"],
   )
   ```
2. Or set it directly on the tokenizer:
   ```python
   tokenizer.eos_token = "<|endoftext|>"
   ```

**Prevention:** Use the built-in chat templates, which set EOS tokens automatically.

### Error: "Unsloth: Merging into 4bit will cause your model to lose accuracy..."

**Cause:** You are saving with `save_method="merged_4bit"`, which merges and requantizes the model.

**Solution:**
1. Use `merged_16bit` for highest quality:
   ```python
   model.save_pretrained_merged("output", tokenizer, save_method="merged_16bit")
   ```
2. Or export to GGUF with a specific quantization level for a better quality/size tradeoff

**Prevention:** Only use `merged_4bit` when deployment constraints demand it.

### Error: "Unsloth: Sorry GGUF is currently not supported for vision models!"

**Cause:** You attempted to export a vision-language model (loaded via `FastVisionModel`) to GGUF format.

**Solution:**
1. Save as merged 16-bit instead:
   ```python
   model.save_pretrained_merged("output", tokenizer, save_method="merged_16bit")
   ```
2. Or save LoRA adapters only:
   ```python
   model.save_pretrained_merged("output", tokenizer, save_method="lora")
   ```

**Prevention:** Use `merged_16bit` or `lora` save methods for vision models.

### Error: "Unsloth: You must select one of 3 options when saving models: 'lora', 'merged_16bit', 'merged_4bit'"

**Cause:** An invalid `save_method` string was passed.

**Solution:**
1. Use one of the three valid save methods:
   ```python
   model.save_pretrained_merged("output", tokenizer, save_method="merged_16bit")
   # or: "merged_4bit", "lora"
   ```

**Prevention:** Always pass one of the three documented save method strings.

### Error: "Unsloth: Can only load in 4bit or 8bit or 16bit, not a combination!"

**Cause:** Conflicting quantization flags were passed.

**Solution:**
1. Choose exactly one quantization option:
   ```python
   # 4-bit
   model, tokenizer = FastLanguageModel.from_pretrained(model_name, load_in_4bit=True)
   # or full precision
   model, tokenizer = FastLanguageModel.from_pretrained(model_name, load_in_4bit=False)
   ```

**Prevention:** Set only one quantization flag per model load.

### Error: "Unsloth: Your model needs to call .get_peft_model first!"

**Cause:** You tried to call a save method that requires LoRA adapters, but `get_peft_model()` was never called.

**Solution:**
1. Apply LoRA before saving:
   ```python
   model = FastLanguageModel.get_peft_model(model, r=16)
   # ... train ...
   model.save_pretrained_merged("output", tokenizer, save_method="merged_16bit")
   ```

**Prevention:** Always call `get_peft_model()` before training and saving with LoRA methods.

### Error: transformers version incompatibility

**Cause:** Some model architectures require specific versions of the `transformers` library.

**Solution:**
1. Upgrade transformers:
   ```bash
   pip install --upgrade transformers
   ```
2. Check Unsloth's requirements for your model architecture
3. The one-line installer pins compatible versions automatically

**Prevention:** Use the one-line installer or pin versions in your requirements file.

### Error: OOM (Out of Memory) during training

**Cause:** Insufficient GPU VRAM for the model + training configuration.

**Solution:**
1. Enable 4-bit quantization: `load_in_4bit=True`
2. Use Unsloth gradient checkpointing: `gradient_checkpointing="unsloth"`
3. Reduce batch size: `per_device_train_batch_size=1`
4. Increase gradient accumulation to compensate: `gradient_accumulation_steps=8`
5. Reduce sequence length: `max_seq_length=1024`
6. Use a smaller LoRA rank: `r=8`
7. For very large models (31B+), even 32 GB VRAM may not be enough with GRPO-style training (GitHub issue #4985)

**Prevention:** Start with `load_in_4bit=True` and small batch sizes, then increase if VRAM allows.

### Error: Windows install.ps1 failure

**Cause:** Various Windows-specific issues including path problems and execution policy restrictions (GitHub issue #5073).

**Solution:**
1. Run PowerShell as Administrator
2. Set execution policy: `Set-ExecutionPolicy -Scope CurrentUser RemoteSigned`
3. Ensure your user profile path does not contain spaces (GitHub issue #4904)
4. Try installing from source as a fallback

**Prevention:** Use a Windows username without spaces. Run the installer from an Administrator PowerShell.

### Error: Windows user names with spaces cause install failure

**Cause:** The installer constructs file paths using the Windows user profile directory, and spaces break the path parsing (GitHub issue #4904).

**Solution:**
1. Create a junction or symbolic link to a path without spaces:
   ```powershell
   New-Item -ItemType Junction -Path "C:\unsloth_home" -Target "$env:USERPROFILE"
   $env:USERPROFILE = "C:\unsloth_home"
   ```
2. Then rerun the installer
3. Or install from source into a custom directory without spaces

**Prevention:** Use a Windows username without spaces.

### Error: Studio HIP llama.cpp build failure (AMD ROCm)

**Cause:** Building llama.cpp with HIP (AMD ROCm) support fails due to missing dependencies or incompatible toolchain (GitHub issue #5093).

**Solution:**
1. Ensure ROCm is fully installed with development headers
2. Verify `hipcc` is in your PATH: `which hipcc`
3. Try building llama.cpp manually:
   ```bash
   cd ~/.unsloth/studio/llama.cpp
   cmake -B build -DGGML_HIP=ON
   cmake --build build --target llama-server
   ```
4. Set the path manually if the build succeeds elsewhere:
   ```bash
   export LLAMA_SERVER_PATH=/path/to/llama-server
   ```

**Prevention:** Install the full ROCm development toolkit before running the Unsloth installer.

### Error: Failing to manually build llama.cpp

**Cause:** Missing build tools (CMake, C++ compiler) or incompatible CUDA toolkit version (GitHub issue #5086).

**Solution:**
1. Install build prerequisites:
   ```bash
   # Ubuntu/Debian
   sudo apt install cmake build-essential
   # macOS
   brew install cmake
   ```
2. Ensure CUDA toolkit matches your driver version
3. Check the llama.cpp build log for specific errors

**Prevention:** Install CMake and a C++ compiler before running the installer.

### Error: Studio Docker — llama-server binary not found

**Cause:** The Docker image does not include a pre-built llama-server binary (GitHub issue #5077).

**Solution:**
1. Build llama-server inside the container or mount it from the host
2. Set `LLAMA_SERVER_PATH` environment variable in your Docker compose/run configuration

**Prevention:** Include llama-server in your Dockerfile build steps.

### Error: IndexError in _prepare_dataset with SFTTrainer

**Cause:** Dataset formatting issue, particularly with large models like Gemma-4-31B (GitHub issue #4833).

**Solution:**
1. Verify your dataset has the expected columns and format
2. Try with a smaller sample first to isolate the issue:
   ```python
   small_dataset = dataset.select(range(100))
   ```
3. Ensure your chat template matches the model

**Prevention:** Validate dataset format before training. Use `standardize_data_formats()` to normalize.

### Error: Vision LoRA receives zero gradients

**Cause:** Known issue with some vision models where LoRA adapters on certain layers receive zero gradients (GitHub issue #5039).

**Solution:**
1. Verify target modules are correct for the vision model
2. Try adjusting the target modules list
3. Check for model-specific instructions in the Unsloth documentation

**Prevention:** Test with a small dataset and verify loss is decreasing before full training runs.

### Error: Incorrect tokenizer behavior

**Cause:** Tokenizer mismatch between expected and actual format (GitHub issue #4996).

**Solution:**
1. Ensure the chat template matches the model:
   ```python
   tokenizer = get_chat_template(tokenizer, chat_template="llama-3.1")
   ```
2. Verify tokenizer output:
   ```python
   print(tokenizer.encode("Hello, world!"))
   print(tokenizer.decode([1, 15043, 29892, 3186, 29991]))
   ```

**Prevention:** Always apply the correct chat template for your model family.

### Error: Gemma 3 GGUF export writes tokens as NORMAL instead of CONTROL

**Cause:** Token type metadata not correctly set during GGUF export for Gemma 3 models (GitHub issue #5070).

**Solution:**
1. This is a known issue — check for updates to Unsloth that may include a fix
2. As a workaround, manually patch the GGUF metadata after export if needed

**Prevention:** Monitor Unsloth releases for Gemma 3 GGUF fixes.

### Error: Different behaviour between transformers and unsloth for Gemma-4

**Cause:** Unsloth's optimized kernels may produce slightly different outputs compared to the reference transformers implementation (GitHub issue #5087).

**Solution:**
1. For exact reference outputs, use the standard transformers path
2. For training, the differences are typically within acceptable tolerance
3. Report significant discrepancies as issues

**Prevention:** Validate outputs against reference implementations when precision matters.

### Error: Unable to resume from existing checkpoint with UnslothTrainer

**Cause:** Checkpoint resumption with `UnslothTrainer` may fail on certain architectures (GitHub issue #5082).

**Solution:**
1. Ensure the checkpoint directory contains all required files
2. Try resuming with the standard `SFTTrainer` instead of `UnslothTrainer`
3. Verify the model architecture is fully supported for checkpoint resumption

**Prevention:** Save checkpoints frequently and test resumption early in long training runs.

### Error: Audio dataset preview fails

**Cause:** Version mismatch with `torchcodec` library (GitHub issue #4900).

**Solution:**
1. Pin `torchcodec` to a compatible version:
   ```bash
   pip install torchcodec==0.1.0
   ```
2. Or skip audio preview and work with the dataset directly

**Prevention:** Pin audio processing library versions in your requirements.

### Error: Studio Colab doesn't give correct URL

**Cause:** Colab's network routing does not expose the Studio server correctly (GitHub issue #4516).

**Solution:**
1. Use `ngrok` or `localtunnel` to expose the port:
   ```bash
   pip install pyngrok
   ```
   ```python
   from pyngrok import ngrok
   public_url = ngrok.connect(8888)
   print(public_url)
   ```

**Prevention:** Use the recommended Colab setup instructions from the Unsloth documentation.

### Error: "Copy" buttons in Studio don't work

**Cause:** Clipboard API requires HTTPS or localhost context; Studio running on a remote server over HTTP cannot access the clipboard (GitHub issue #5097).

**Solution:**
1. Access Studio via `localhost` (local machine) or HTTPS
2. Manually select and copy text as a workaround

**Prevention:** Access Studio via localhost or set up an HTTPS proxy.

### Error: General install issues

**Cause:** Dependency conflicts, incompatible Python version, or network issues (GitHub issue #4983).

**Solution:**
1. Use the one-line installer (handles all dependencies)
2. Verify Python version: `python --version` (must be 3.11–3.13)
3. Create a fresh virtual environment:
   ```bash
   python -m venv unsloth-env
   source unsloth-env/bin/activate  # Linux/macOS
   # or: unsloth-env\Scripts\activate  # Windows
   pip install unsloth
   ```
4. If behind a corporate proxy, configure pip:
   ```bash
   pip install --proxy http://proxy:port unsloth
   ```

**Prevention:** Use a clean virtual environment with a supported Python version.

### Issue: VRAM usage higher than expected

**Cause:** Gradient checkpointing is not using the Unsloth mode, or batch size is too large.

**Solution:**
1. Verify Unsloth gradient checkpointing is active:
   ```python
   args = TrainingArguments(
       gradient_checkpointing=True,
       gradient_checkpointing_kwargs={"use_reentrant": False},
   )
   ```
2. Set `gradient_checkpointing: unsloth` in YAML config
3. Reduce `per_device_train_batch_size` to 1

**Prevention:** Always use `gradient_checkpointing: unsloth` in your configuration.

### Issue: Training loss not decreasing

**Cause:** Learning rate too high or too low, dataset quality issues, or incorrect chat template.

**Solution:**
1. Try the default learning rate: `2e-4`
2. Verify dataset format matches the expected template
3. Check that the correct chat template is applied
4. Ensure `train_on_responses_only` masks are correct (if used)
5. Try a smaller LoRA rank (8 or 16) to reduce overfitting risk

**Prevention:** Start with default hyperparameters and adjust one at a time.

### Issue: Slow training speed

**Cause:** Unsloth optimizations not active, or suboptimal batch configuration.

**Solution:**
1. Verify Unsloth is actually loaded (check for "Unsloth:" prefixed log messages)
2. Enable packing for short sequences: `--packing`
3. Increase batch size if VRAM allows
4. Ensure you are using `adamw_8bit` optimizer

**Prevention:** Use the Unsloth-provided model loaders (`FastLanguageModel`, etc.) rather than loading directly with transformers.

### Issue: Model outputs gibberish after training

**Cause:** Overfitting, wrong chat template, or too many training steps.

**Solution:**
1. Reduce `max_steps` or `num_train_epochs`
2. Verify the chat template matches the base model
3. Check training loss — if it drops below 0.1, you are likely overfitting
4. Use a larger dataset or apply regularization (increase `lora_dropout` slightly)

**Prevention:** Monitor training loss and stop early if it plateaus or drops too low.

### Issue: Saved model does not load correctly

**Cause:** Incomplete save, wrong save method, or missing tokenizer files.

**Solution:**
1. Ensure both model and tokenizer are saved:
   ```python
   model.save_pretrained_merged("output", tokenizer, save_method="merged_16bit")
   ```
2. Check the output directory contains all expected files
3. Try loading with the same Unsloth version used for saving

**Prevention:** Always pass the tokenizer when saving. Verify the output directory after saving.

---

## 7. FAQ

**Q: What GPU do I need to run Unsloth?**

You need an NVIDIA (CUDA 11.8–13.0), AMD (ROCm), or Intel (XPU) GPU. NVIDIA GPUs are the best-tested path. CPU-only training is not supported, though GGUF-only workflows can run on CPU with the `--no-torch` installer flag.

**Q: How much VRAM do I need?**

With 4-bit quantization and Unsloth gradient checkpointing, a 7B/8B model fits in ~6 GB VRAM with batch size 1. A 13B model needs ~10 GB. Larger models (30B+) typically require 24–48 GB or more. Unsloth reduces VRAM usage by ~60% compared to standard fine-tuning.

**Q: What is the difference between LoRA and QLoRA?**

LoRA (Low-Rank Adaptation) adds small trainable adapter matrices to frozen model weights. QLoRA adds 4-bit quantization of the base model on top, further reducing VRAM. In Unsloth, set `load_in_4bit=True` for QLoRA or `load_in_4bit=False` for standard LoRA.

**Q: What is the difference between LoRA and full fine-tuning?**

LoRA trains only adapter parameters (~1–5% of total), keeping the base model frozen. Full fine-tuning updates all parameters, requiring much more VRAM and compute but potentially achieving better results on large datasets. Use LoRA for most tasks; reserve full fine-tuning for domain-specific specialization with large datasets.

**Q: What LoRA rank should I use?**

Start with `r=16` for general-purpose fine-tuning. Use `r=8` for quick experiments or small datasets. Use `r=32` or `r=64` for complex tasks or large datasets. Higher ranks use more VRAM and increase overfitting risk on small datasets.

**Q: What is padding-free packing?**

Packing concatenates multiple training samples into single sequences, eliminating wasted padding tokens. This improves training throughput, especially when your dataset has variable-length examples. Enable it with `--packing` on the CLI or `packing=True` in the trainer.

**Q: What dataset formats does Unsloth support?**

Unsloth supports Alpaca format (instruction/input/output), ShareGPT format (multi-turn conversations), ChatML format, and raw text files for continual pretraining. The `format_type: auto` setting auto-detects the format.

**Q: How do I use a custom dataset?**

Load your dataset with HuggingFace `datasets`, then format it with Unsloth's data utilities:

```python
from datasets import load_dataset
dataset = load_dataset("json", data_files="my_data.json", split="train")
```

Then apply the appropriate chat template or format conversion.

**Q: What is the Verified Facts Sheet mentioned in the training output?**

This is an internal Unsloth mechanism for ensuring training data quality. You do not need to interact with it directly.

**Q: Can I fine-tune vision models?**

Yes. Use `FastVisionModel` for Llama 3.2 Vision, Qwen2-VL, Gemma 3, and other vision-language models. Note that GGUF export is not supported for vision models.

**Q: Can I fine-tune sentence transformers / embedding models?**

Yes. Use `FastSentenceTransformer` for BERT, ModernBERT, and other sentence transformer models. Install `sentence-transformers` first.

**Q: What models does Unsloth support?**

Unsloth supports Llama (3.x), Mistral, Gemma (1–4), Gemma 3N, Qwen 2/2.5/3/3 MoE, Phi (3/3.5/4), DeepSeek (V3, R1), Cohere, Falcon H1, Granite, GLM-4 MoE, GPT-OSS, OLMo (2/3), Pixtral, Whisper, BERT/ModernBERT, and any transformers-compatible model via the `FastModel` fallback.

**Q: What is Unsloth Studio?**

Unsloth Studio is a local web UI for training, inference, export, and synthetic data generation. It runs on port 8888 by default and provides a visual alternative to the CLI and Python API. Launch it with `unsloth studio`.

**Q: Is Unsloth Studio free?**

The core Unsloth library is Apache-2.0 licensed and free. Unsloth Studio and MoE kernels are AGPL-3.0 licensed, which is also free for most uses but requires source disclosure if you distribute modified versions as a network service.

**Q: How do I access Studio remotely?**

Studio binds to `0.0.0.0` by default, making it accessible on your local network. For remote access over the internet, use SSH tunneling, a reverse proxy with HTTPS, or tools like ngrok.

**Q: What is the Studio bootstrap password?**

On first launch, Studio creates an admin account and saves the bootstrap password to `~/.unsloth/studio/.bootstrap_password`. Use `unsloth studio reset-password` to reset it.

**Q: Can I run Unsloth in Docker?**

Yes. See the Admin Guide for Docker deployment instructions. Note that the llama-server binary must be built or mounted separately for Studio Chat functionality (GitHub issue #5077).

**Q: Can I run Unsloth on Google Colab?**

Yes, but Studio URL routing can be tricky on Colab (GitHub issue #4516). Use ngrok or localtunnel to expose the Studio port. The CLI and Python API work directly in Colab notebooks.

**Q: How do I use Unsloth with Ollama?**

Export your model to GGUF format, then use the generated Modelfile with `ollama create`. See section 5.7 for the complete workflow.

**Q: How do I use Unsloth with llama.cpp?**

Export to GGUF and use the `.gguf` file directly with llama.cpp's `llama-server` or `llama-cli`.

**Q: What GGUF quantization should I choose?**

For the best quality-size balance, use `q8_0`. For smaller deployments, `q4_k_m` or `q5_k_m` are good choices. Avoid `q2_k` unless extreme compression is required. The `iq2` family is NOT supported.

**Q: How do I monitor training progress?**

Use WandB (`use_wandb: true`), TensorBoard (`use_tensorboard: true`), or Studio's Training tab which shows live loss charts. Training logs also print to the console.

**Q: Can I train on multiple GPUs?**

Unsloth focuses on single-GPU efficiency. For multi-GPU setups, use the standard HuggingFace training pipeline with data parallelism.

**Q: What is the `seed: 3407` default?**

This is a well-known seed value in the ML community (from the paper "Torch.manual_seed(3407) is all you need"). It is a reasonable default for reproducibility.

**Q: Can I use Unsloth with FSDP or DeepSpeed?**

Unsloth's custom kernels are designed for single-GPU optimization. For distributed training, use standard PyTorch/HuggingFace parallelism strategies.

**Q: How do I push my model to HuggingFace Hub?**

Use `model.push_to_hub_merged()` or `model.push_to_hub_gguf()`. Set `HF_TOKEN` or pass `token=` directly. See section 4.6 for examples.

**Q: What is `gradient_checkpointing: unsloth`?**

Unsloth's custom gradient checkpointing implementation saves ~60% more VRAM than standard gradient checkpointing by recomputing activations more efficiently. Always use this setting.

**Q: What is the `embedding_learning_rate` in UnslothTrainingArguments?**

It allows you to set a separate (typically lower) learning rate for the embedding layers, which can improve training stability when the embeddings need gentle updates.

**Q: What is Q-GaLore?**

Q-GaLore is an advanced optimizer option available through `UnslothTrainingArguments`. It combines quantization with gradient low-rank projection for memory-efficient training. Configure it via the `q_galore_config` parameter.

**Q: How do I train only on model responses, not instructions?**

Use `train_on_responses_only()` to mask instruction tokens from the loss. See section 4.1 for the complete example.

**Q: Can I use a raw text file for continual pretraining?**

Yes. Use `RawTextDataLoader` in Python or `--raw_text_file` on the CLI. See section 4.7 for details.

**Q: How do I construct a chat template for Ollama?**

Use `construct_chat_template()` which generates both an Ollama-compatible modelfile and a HuggingFace Jinja template. See section 4.7 for the example.

**Q: What is the `max_seq_length` parameter?**

It sets the maximum token sequence length for training. Unsloth auto-applies RoPE scaling if the sequence length exceeds the model's native context window. Default is `2048`.

**Q: My training is using more VRAM than expected. What should I check?**

First, verify `load_in_4bit=True` is set. Then confirm `gradient_checkpointing="unsloth"` is active. Reduce batch size to 1 and reduce `max_seq_length`. Lower the LoRA rank. See the troubleshooting section on OOM errors for a complete checklist.

---

_Generated May 2026 from unslothai/unsloth at d1f9ab6. AI-generated documentation. The library is checked by sampling (three repositories so far); this guide was not individually reviewed. Verify against the project before relying on it. Unofficial; not affiliated with or reviewed by the project._