# LLMs-from-scratch User Guide

## 1. Quick Start

**LLMs-from-scratch** is the official companion code repository for the book *Build a Large Language Model From Scratch* by Sebastian Raschka (Manning Publications). It provides complete, working implementations of a GPT-style large language model built step-by-step from raw text processing through pretraining, fine-tuning for classification, and instruction-following — all in pure PyTorch.

**Key features:**

- Build a GPT-2-class transformer from scratch in PyTorch
- Implement tokenization using OpenAI's `tiktoken` byte-pair encoding
- Pretrain on custom text corpora (Project Gutenberg, custom datasets)
- Load and use official OpenAI GPT-2 pretrained weights (124M to 1.5B parameters)
- Fine-tune for text classification (spam detection, IMDB sentiment)
- Fine-tune for instruction following with supervised fine-tuning (SFT)
- Implement advanced architectures: KV cache, grouped-query attention (GQA), multi-head latent attention (MLA), sliding window attention (SWA), mixture-of-experts (MoE), DeltaNet
- Extend to modern LLMs: LLaMA 3.2, Qwen3, Gemma3, Gemma4, OLMo3, Qwen3.5, Tiny-Aya

**Prerequisites:**

- Python 3.9 or newer
- A working `pip` or `uv` installation
- NVIDIA GPU recommended for training (CPU works for inference and small experiments)

**Quickest path to running inference with pretrained GPT-2 weights:**

```bash
git clone https://github.com/rasbt/LLMs-from-scratch.git
cd LLMs-from-scratch
pip install -r requirements.txt
pip install -e .
python ch05/01_main-chapter-code/gpt_generate.py --prompt "Hello, world" --device auto
```

```text
Every effort moves you closer to the next level of understanding...
```

> [!TIP]
> If you just want to explore the concepts interactively, open the Jupyter notebooks directly. Each chapter folder contains `ch0X.ipynb` files that walk through the code with explanations.

**What to try next:**

- Run the chapter 4 notebook (`ch04/01_main-chapter-code/ch04.ipynb`) to build the full GPT architecture
- Fine-tune for spam classification in chapter 6
- Build an instruction-following chatbot in chapter 7
- See the Admin Guide for architecture deep-dives and module reference

---

## 2. Installation

### System Requirements

| Requirement | Minimum | Recommended |
|---|---|---|
| Python | 3.9 | 3.10-3.12 |
| RAM | 8 GB | 16 GB+ |
| GPU | None (CPU works) | NVIDIA with CUDA support |
| Disk | 2 GB (code + deps) | 10 GB+ (with model weights) |
| OS | Linux, macOS, Windows | Any with PyTorch support |

### Install via pip (recommended)

```bash
git clone https://github.com/rasbt/LLMs-from-scratch.git
cd LLMs-from-scratch
pip install -r requirements.txt
```

> [!IMPORTANT]
> The `requirements.txt` includes `tensorflow` (used for loading OpenAI's original GPT-2 checkpoint format in chapters 5-7). On Intel macOS, a version-pinned `torch` and `tensorflow-cpu` variant are automatically selected via environment markers.

### Install the companion package

The repository includes an installable Python package (`llms_from_scratch`) that consolidates all implementations from every chapter into reusable modules:

```bash
pip install -e .
```

This makes all chapter modules importable:

```python
from llms_from_scratch.ch04 import GPTModel
from llms_from_scratch.ch05 import generate, text_to_token_ids, token_ids_to_text
```

### Install with pixi (alternative)

The repo ships a `pixi.toml` for reproducible environments using [pixi](https://pixi.sh):

```bash
pixi install
pixi run jupyter lab
```

The pixi environment pins Python 3.10, `pytorch-cpu >=2.6.0`, and all book dependencies. To include testing dependencies:

```bash
pixi install -e tests
```

### Install with uv (alternative)

```bash
uv pip install -r requirements.txt
uv pip install -e .
```

### Docker environment

The `setup/03_optional-docker-environment/` folder contains Docker setup instructions for running the code in a containerized environment. See the folder's `README.md` for details.

### AWS SageMaker

A CloudFormation template is provided at `setup/04_optional-aws-sagemaker-notebook/cloudformation-template.yml` for launching a SageMaker notebook instance pre-configured with all dependencies.

### Verify installation

Run the environment check script to verify all dependencies are correctly installed:

```bash
python setup/02_installing-python-libraries/python_environment_check.py
```

```text
[OK] Your Python version is 3.10.12
[OK] torch 2.5.1
[OK] jupyterlab 4.3.5
[OK] tiktoken 0.9.0
[OK] matplotlib 3.10.0
[OK] tensorflow 2.18.0
[OK] tqdm 4.67.1
[OK] numpy 1.26.4
[OK] pandas 2.2.3
[OK] psutil 5.9.8
```

> [!WARNING]
> If any line shows `[FAIL]`, install the correct version before proceeding. Mismatched versions — especially for `torch` and `tensorflow` — can cause subtle errors during weight loading.

### Chapter-specific extras

Some bonus sections require additional packages. Each folder that needs them includes a `requirements-extra.txt`:

| Chapter/Section | Extra Dependencies | Install |
|---|---|---|
| `ch02/02_bonus_bytepair-encoder` | GPT-2 BPE reference | `pip install -r ch02/02_bonus_bytepair-encoder/requirements-extra.txt` |
| `ch05/06_user_interface` | Chainlit | `pip install -r ch05/06_user_interface/requirements-extra.txt` |
| `ch05/07_gpt_to_llama` | safetensors, sentencepiece | `pip install -r ch05/07_gpt_to_llama/requirements-extra.txt` |
| `ch06/03_bonus_imdb-classification` | HuggingFace Transformers | `pip install -r ch06/03_bonus_imdb-classification/requirements-extra.txt` |
| `ch06/04_user_interface` | Chainlit | `pip install -r ch06/04_user_interface/requirements-extra.txt` |
| `ch07/02_dataset-utilities` | Dataset tools | `pip install -r ch07/02_dataset-utilities/requirements-extra.txt` |
| `ch07/03_model-evaluation` | Evaluation tools | `pip install -r ch07/03_model-evaluation/requirements-extra.txt` |
| `ch07/05_dataset-generation` | Generation tools | `pip install -r ch07/05_dataset-generation/requirements-extra.txt` |
| `ch07/06_user_interface` | Chainlit | `pip install -r ch07/06_user_interface/requirements-extra.txt` |

---

## 3. Core Usage

The repository follows the book's chapter structure. Each chapter builds on the previous one, progressively constructing a complete LLM pipeline.

### Chapter 1 — Understanding Large Language Models

Introductory chapter. Contains reading recommendations at `ch01/reading-recommendations.md`. No code to run.

### Chapter 2 — Working with Text Data

Covers tokenization and data loading for LLM training.

**Main notebook:** `ch02/01_main-chapter-code/ch02.ipynb`

Key implementations:

- `GPTDatasetV1` — sliding window dataset that creates input/target pairs from tokenized text
- `create_dataloader_v1()` — convenience function wrapping the dataset with a PyTorch DataLoader
- Token and positional embeddings

```python
import tiktoken
from llms_from_scratch.ch02 import create_dataloader_v1

tokenizer = tiktoken.get_encoding("gpt2")
dataloader = create_dataloader_v1(
    txt="Your training text here",
    batch_size=4,
    max_length=256,
    stride=128
)
```

**Bonus sections:**

| Folder | Topic |
|---|---|
| `ch02/02_bonus_bytepair-encoder` | OpenAI GPT-2 BPE tokenizer reference implementation |
| `ch02/03_bonus_embedding-vs-matmul` | Equivalence of embedding lookup and matrix multiplication |
| `ch02/04_bonus_dataloader-intuition` | Visual intuition for dataloader sliding window |
| `ch02/05_bpe-from-scratch` | Build a BPE tokenizer from scratch |

### Chapter 3 — Coding Attention Mechanisms

Implements self-attention from first principles through multi-head attention.

**Main notebook:** `ch03/01_main-chapter-code/ch03.ipynb`

Key classes (all in `llms_from_scratch.ch03`):

| Class | Description |
|---|---|
| `SelfAttention_v1` | Basic self-attention with raw parameter matrices |
| `SelfAttention_v2` | Self-attention using `nn.Linear` layers |
| `CausalAttention` | Masked self-attention with dropout |
| `MultiHeadAttentionWrapper` | Naive multi-head via list of `CausalAttention` |
| `MultiHeadAttention` | Efficient multi-head attention with fused projections |
| `PyTorchMultiHeadAttention` | Using `F.scaled_dot_product_attention` |

> [!NOTE]
> The progression from `SelfAttention_v1` to `PyTorchMultiHeadAttention` is intentionally pedagogical. Each version adds one concept: learnable projections, causal masking, dropout, multi-head splitting, and finally PyTorch's optimized kernel.

**Bonus sections:**

| Folder | Topic |
|---|---|
| `ch03/02_bonus_efficient-multihead-attention` | Benchmarking different MHA implementations |
| `ch03/03_understanding-buffers` | PyTorch buffer mechanics for attention masks |

### Chapter 4 — Implementing a GPT Model from Scratch

Assembles the full GPT architecture: `LayerNorm`, `GELU`, `FeedForward`, `TransformerBlock`, and `GPTModel`.

**Main notebook:** `ch04/01_main-chapter-code/ch04.ipynb`

The complete GPT-2 124M configuration:

```python
GPT_CONFIG_124M = {
    "vocab_size": 50257,
    "context_length": 1024,
    "emb_dim": 768,
    "n_heads": 12,
    "n_layers": 12,
    "drop_rate": 0.1,
    "qkv_bias": True
}
```

**Bonus sections — advanced attention and architecture variants:**

| Folder | Topic | Key Script |
|---|---|---|
| `ch04/02_performance-analysis` | FLOPs analysis of the GPT model | `flops-analysis.ipynb` |
| `ch04/03_kv-cache` | KV caching for faster inference | `gpt_with_kv_cache.py`, `gpt_with_kv_cache_optimized.py` |
| `ch04/04_gqa` | Grouped-query attention | `gpt_with_kv_gqa.py` |
| `ch04/05_mla` | Multi-head latent attention (DeepSeek-style) | `gpt_with_kv_mla.py` |
| `ch04/06_swa` | Sliding window attention | `gpt_with_kv_swa.py` |
| `ch04/07_moe` | Mixture-of-experts feed-forward layers | `gpt_with_kv_moe.py` |
| `ch04/08_deltanet` | DeltaNet (gated linear attention) | Memory estimation scripts |

> [!TIP]
> Each advanced architecture variant in `ch04/` includes standalone Python scripts with CLI arguments for experimentation. These scripts implement generation with KV caching and accept arguments like `--emb_dim`, `--n_heads`, `--n_layers`, and `--max_new_tokens`.

**Running the KV-cache GPT variant:**

```bash
python ch04/03_kv-cache/gpt_with_kv_cache.py
```

**Running grouped-query attention with custom parameters:**

```bash
python ch04/04_gqa/gpt_with_kv_gqa.py --emb_dim 768 --n_heads 12 --n_layers 12 --n_kv_groups 4 --max_new_tokens 200
```

**Running mixture-of-experts:**

```bash
python ch04/07_moe/gpt_with_kv_moe.py --num_experts 8 --num_experts_per_tok 2 --emb_dim 768 --n_layers 12
```

### Chapter 5 — Pretraining on Unlabeled Data

Covers GPT-2 weight loading from OpenAI checkpoints, pretraining from scratch, and extending to modern architectures.

**Main notebook:** `ch05/01_main-chapter-code/ch05.ipynb`

**Generate text with pretrained GPT-2 weights:**

```bash
python ch05/01_main-chapter-code/gpt_generate.py --prompt "Every effort moves you" --device auto
```

| Flag | Type | Default | Description |
|---|---|---|---|
| `--prompt` | string | `"Every effort moves you"` | Prompt text to seed generation |
| `--device` | string | `cpu` | Device: `cpu`, `cuda`, `mps`, or `auto` |

**Pretraining on Project Gutenberg:**

```bash
# Step 1: Prepare the dataset
python ch05/03_bonus_pretraining_on_gutenberg/prepare_dataset.py \
    --data_dir gutenberg/data/raw \
    --output_dir gutenberg_preprocessed \
    --max_size_mb 500

# Step 2: Run pretraining
python ch05/03_bonus_pretraining_on_gutenberg/pretraining_simple.py \
    --data_dir gutenberg/data \
    --output_dir model_checkpoints \
    --n_epochs 1 \
    --batch_size 4 \
    --lr 0.0005
```

**Hyperparameter tuning:**

```bash
python ch05/05_bonus_hparam_tuning/hparam_search.py
```

**Extending to other LLM architectures:**

| Folder | Architecture | Notes |
|---|---|---|
| `ch05/07_gpt_to_llama` | LLaMA 2 then LLaMA 3.2 | Converting GPT architecture to LLaMA, loading Meta weights |
| `ch05/11_qwen3` | Qwen3 (dense + MoE) | Standalone implementation with KV cache |
| `ch05/12_gemma3` | Gemma 3 | Google's open model |
| `ch05/13_olmo3` | OLMo 3 | AI2's open model |
| `ch05/15_tiny-aya` | Tiny-Aya | Cohere's multilingual model |
| `ch05/16_qwen3.5` | Qwen3.5 (GatedDeltaNet) | With DeltaNet attention via `Qwen3_5GatedDeltaNet` |
| `ch05/17_gemma4` | Gemma 4 | Latest Google model |

**Training speed optimization (ch05/10):**

Three progressive scripts demonstrate optimization techniques:

| Script | Optimizations |
|---|---|
| `00_orig.py` | Baseline (no optimizations) |
| `01_opt_single_gpu.py` | `PyTorchMultiHeadAttention` with FlashAttention via `F.scaled_dot_product_attention`, `torch.compile` |
| `02_opt_multi_gpu_ddp.py` | All single-GPU optimizations + DistributedDataParallel (DDP) multi-GPU |

**Chainlit chat interface:**

```bash
pip install -r ch05/06_user_interface/requirements-extra.txt
chainlit run ch05/06_user_interface/app_orig.py
```

### Chapter 6 — Fine-Tuning for Classification

Fine-tunes GPT-2 as a binary text classifier (spam detection).

**Main notebook:** `ch06/01_main-chapter-code/ch06.ipynb`

**Run the standalone training script:**

```bash
python ch06/01_main-chapter-code/gpt_class_finetune.py
```

| Flag | Type | Default | Description |
|---|---|---|---|
| `--test_mode` | flag | `False` | Run with reduced dataset for testing |

The script downloads a spam SMS dataset, fine-tunes a GPT-2 124M model for binary classification, and saves the model as `review_classifier.pth`.

**IMDB classification bonus (ch06/03):**

Trains various models on IMDB sentiment data with extensive CLI options:

```bash
# Train GPT-2 on IMDB
python ch06/03_bonus_imdb-classification/train_gpt.py \
    --model_size "gpt2-small (124M)" \
    --weights pretrained \
    --trainable_layers last_block \
    --num_epochs 1

# Train BERT/DistilBERT/RoBERTa on IMDB
python ch06/03_bonus_imdb-classification/train_bert_hf.py \
    --model distilbert \
    --trainable_layers all \
    --num_epochs 1
```

**Full CLI reference for `train_gpt.py`:**

| Flag | Type | Default | Description |
|---|---|---|---|
| `--model_size` | str | `"gpt2-small (124M)"` | GPT-2 variant: `"gpt2-small (124M)"`, `"gpt2-medium (355M)"`, `"gpt2-large (774M)"`, `"gpt2-xl (1558M)"` |
| `--weights` | str | `pretrained` | `pretrained` or `random` |
| `--trainable_layers` | str | `last_block` | `all`, `last_block`, `last_layer` |
| `--trainable_token_pos` | str | `last` | `first` or `last` |
| `--average_embeddings` | flag | `False` | Average all token embeddings instead of using one position |
| `--context_length` | str | `256` | `longest_training_example`, `model_context_length`, or integer |
| `--num_epochs` | int | `1` | Training epochs |
| `--learning_rate` | float | `5e-05` | Learning rate |
| `--compile` | flag | off | Enable `torch.compile` |

**Full CLI reference for `train_bert_hf.py`:**

| Flag | Type | Default | Description |
|---|---|---|---|
| `--model` | str | `distilbert` | `distilbert`, `bert`, `roberta`, `modernbert-base/-large`, `deberta-v3-base` |
| `--trainable_layers` | str | `all` | `all`, `last_block`, `last_layer` |
| `--use_attention_mask` | str | `true` | `true` or `false` |
| `--num_epochs` | int | `1` | Training epochs |
| `--learning_rate` | float | `5e-06` | Learning rate |

> [!NOTE]
> The `additional_experiments.py` script in `ch06/02_bonus_additional-experiments/` supports LoRA fine-tuning via `--trainable_layers lora` with configurable `--lora_rank` (default 8) and `--lora_alpha` (default 8) parameters. It also supports gradient accumulation via `--accumulation_steps`.

**Classification chat UI:**

After training the classifier in chapter 6, launch the Chainlit interface:

```bash
pip install -r ch06/04_user_interface/requirements-extra.txt
chainlit run ch06/04_user_interface/app.py
```

### Chapter 7 — Fine-Tuning to Follow Instructions

Implements supervised fine-tuning (SFT) for instruction following, preference tuning with DPO, and model evaluation.

**Main notebook:** `ch07/01_main-chapter-code/ch07.ipynb`

**Run instruction fine-tuning:**

```bash
python ch07/01_main-chapter-code/gpt_instruction_finetuning.py
```

| Flag | Type | Default | Description |
|---|---|---|---|
| `--test_mode` | flag | `False` | Run with reduced dataset for testing |

This trains a GPT-2 355M model on the bundled `instruction-data.json` dataset and saves the result as `gpt2-medium355M-sft.pth`.

**Evaluate model outputs using Ollama:**

```bash
python ch07/01_main-chapter-code/ollama_evaluate.py \
    --file_path instruction-data-with-response.json
```

> [!WARNING]
> The evaluation script requires Ollama to be running locally. If Ollama is not running, the script raises `RuntimeError: Ollama not running. Launch ollama before proceeding.`

**Dataset utilities (ch07/02):**

Find and remove near-duplicate entries in instruction datasets:

```bash
python ch07/02_dataset-utilities/find-near-duplicates.py \
    --json_file instruction-examples.json \
    --threshold 0.9 \
    --remove_duplicates \
    --json_output_file cleaned-data.json
```

| Flag | Type | Default | Description |
|---|---|---|---|
| `--json_file` | str | required | Path to input JSON dataset |
| `--threshold` | float | `0.9` | Similarity threshold (0-1, 1 is strictest) |
| `--remove_duplicates` | flag | `False` | Remove duplicates and save cleaned file |
| `--json_output_file` | str | required with `--remove_duplicates` | Path for cleaned output |

**Preference tuning with DPO (ch07/04):**

The `dpo-from-scratch.ipynb` notebook implements Direct Preference Optimization from scratch, using preference data generated via Ollama in `create-preference-data-ollama.ipynb`.

**Dataset generation (ch07/05):**

Generate synthetic instruction datasets using LLaMA 3 via Ollama or GPT-4 via the OpenAI API.

**Instruction-following chat UI:**

```bash
pip install -r ch07/06_user_interface/requirements-extra.txt
chainlit run ch07/06_user_interface/app.py
```

### Appendices

| Appendix | Topic | Key Content |
|---|---|---|
| **A** | Multi-GPU Training with DDP | `DDP-script.py`, `DDP-script-torchrun.py` |
| **B** | References and Further Reading | Reading list |
| **C** | Exercise Solutions Pointers | Links to solutions |
| **D** | Pretraining with Learning Rate Warmup and Cosine Annealing | `appendix-D.ipynb` |
| **E** | Parameter-Efficient Fine-Tuning with LoRA | `appendix-E.ipynb` |

**Running multi-GPU DDP training (Appendix A):**

```bash
# Using torchrun (recommended)
torchrun --nproc_per_node=2 appendix-A/01_main-chapter-code/DDP-script-torchrun.py

# Using CUDA_VISIBLE_DEVICES
CUDA_VISIBLE_DEVICES=0,1 python appendix-A/01_main-chapter-code/DDP-script.py
```

> [!IMPORTANT]
> The DDP scripts are designed for 2 GPUs by default. To use a different number of GPUs, uncomment the configuration code on lines 103-107 in the respective script.

---

## 4. Common Tasks

### Load pretrained GPT-2 weights and generate text

```python
from llms_from_scratch.ch04 import GPTModel
from llms_from_scratch.ch05 import (
    download_and_load_gpt2,
    load_weights_into_gpt,
    generate,
    text_to_token_ids,
    token_ids_to_text
)
import tiktoken
import torch

# Configuration for GPT-2 124M
GPT_CONFIG_124M = {
    "vocab_size": 50257,
    "context_length": 1024,
    "emb_dim": 768,
    "n_heads": 12,
    "n_layers": 12,
    "drop_rate": 0.0,
    "qkv_bias": True
}

# Download and load weights
settings, params = download_and_load_gpt2("124M", "gpt2")
model = GPTModel(GPT_CONFIG_124M)
load_weights_into_gpt(model, params)
model.eval()

# Generate
tokenizer = tiktoken.get_encoding("gpt2")
token_ids = generate(
    model=model,
    idx=text_to_token_ids("Hello, I am", tokenizer),
    max_new_tokens=50,
    context_size=GPT_CONFIG_124M["context_length"],
    eos_id=50256
)
print(token_ids_to_text(token_ids, tokenizer))
```

### Fine-tune GPT-2 for binary text classification

1. Run the chapter 6 notebook or standalone script to train:
   ```bash
   python ch06/01_main-chapter-code/gpt_class_finetune.py
   ```
2. The script automatically downloads the SMS Spam dataset, creates balanced train/val/test splits, and fine-tunes GPT-2 124M
3. The trained model is saved as `ch06/01_main-chapter-code/review_classifier.pth`

### Fine-tune GPT-2 for instruction following

1. Run the chapter 7 notebook or standalone script:
   ```bash
   python ch07/01_main-chapter-code/gpt_instruction_finetuning.py
   ```
2. Uses the bundled `instruction-data.json` (1,100 instruction/response pairs)
3. Saves the fine-tuned GPT-2 355M model as `gpt2-medium355M-sft.pth`

### Run a Chainlit chat interface

Four chat UIs are available:

| UI | Command | Model | Purpose |
|---|---|---|---|
| GPT-2 text generation | `chainlit run ch05/06_user_interface/app_orig.py` | GPT-2 pretrained | Open-ended generation |
| Spam classifier | `chainlit run ch06/04_user_interface/app.py` | Fine-tuned classifier | Classify text as spam/not-spam |
| Instruction follower | `chainlit run ch07/06_user_interface/app.py` | SFT fine-tuned | Answer instructions |
| Qwen3 chat | `chainlit run ch05/11_qwen3/qwen3-chat-interface/qwen3-chat-interface.py` | Qwen3 0.6B | Thinking-mode chat |

> [!TIP]
> The Qwen3 chat interface at `ch05/11_qwen3/qwen3-chat-interface/` also has a multi-turn variant (`qwen3-chat-interface-multiturn.py`) that remembers conversation history.

### Compare KV cache memory across attention mechanisms

Each architecture variant in `ch04/` includes a memory estimator script:

```bash
# GQA memory estimation
python ch04/04_gqa/memory_estimator_gqa.py --emb_dim 4096 --n_heads 32 --n_layers 32 --n_kv_groups 8

# MLA memory estimation
python ch04/05_mla/memory_estimator_mla.py --emb_dim 4096 --n_heads 32 --n_layers 32 --latent_dim 512

# SWA memory estimation
python ch04/06_swa/memory_estimator_swa.py --emb_dim 4096 --n_heads 32 --n_layers 32 --sliding_window_size 4096

# MoE memory estimation
python ch04/07_moe/memory_estimator_moe.py --emb_dim 4096 --hidden_dim 11008 --num_experts 8 --top_k 2
```

### Plot memory comparison charts

```bash
python ch04/04_gqa/plot_memory_estimates_gqa.py
python ch04/05_mla/plot_memory_estimates_mla.py
python ch04/06_swa/plot_memory_estimates_swa.py --emb_dim 4096 --n_heads 32 --n_layers 32 --sliding_window_size 4096 --swa_ratio 5:1
python ch04/07_moe/plot_memory_estimates_moe.py --emb_dim 4096 --hidden_dim 11008
python ch04/08_deltanet/plot_memory_estimates_gated_deltanet.py --emb_dim 2048 --n_heads 16 --n_layers 48
```

### Build a BPE tokenizer from scratch

```bash
jupyter lab ch02/05_bpe-from-scratch/bpe-from-scratch.ipynb
```

The notebook implements the complete byte-pair encoding algorithm step-by-step, without any external tokenizer library.

### Pretrain on a large text corpus

```bash
# Download and prepare Project Gutenberg data
python ch05/03_bonus_pretraining_on_gutenberg/prepare_dataset.py \
    --data_dir gutenberg/data/raw \
    --output_dir gutenberg_preprocessed

# Run pretraining with checkpointing
python ch05/03_bonus_pretraining_on_gutenberg/pretraining_simple.py \
    --data_dir gutenberg/data \
    --output_dir model_checkpoints \
    --n_epochs 1 \
    --batch_size 4 \
    --lr 0.0005 \
    --eval_freq 100 \
    --save_ckpt_freq 100000 \
    --print_sample_iter 1000
```

### Load weights for modern LLMs (LLaMA, Qwen3, Gemma)

Each model family in `ch05/` has standalone notebooks:

```bash
# LLaMA 3.2
jupyter lab ch05/07_gpt_to_llama/standalone-llama32.ipynb

# Qwen3 (with and without KV cache)
jupyter lab ch05/11_qwen3/standalone-qwen3.ipynb
jupyter lab ch05/11_qwen3/standalone-qwen3-plus-kvcache.ipynb

# Qwen3 MoE variant
jupyter lab ch05/11_qwen3/standalone-qwen3-moe.ipynb

# Gemma 3
jupyter lab ch05/12_gemma3/standalone-gemma3.ipynb

# Gemma 4
jupyter lab ch05/17_gemma4/standalone-gemma4.ipynb

# OLMo 3
jupyter lab ch05/13_olmo3/standalone-olmo3.ipynb

# Tiny-Aya
jupyter lab ch05/15_tiny-aya/standalone-tiny-aya.ipynb
```

### Implement DPO (Direct Preference Optimization) from scratch

```bash
# Step 1: Generate preference data using Ollama
jupyter lab ch07/04_preference-tuning-with-dpo/create-preference-data-ollama.ipynb

# Step 2: Run DPO training
jupyter lab ch07/04_preference-tuning-with-dpo/dpo-from-scratch.ipynb
```

### Run chapter 5 pretraining with other LLM architectures

```bash
# Run ch05 pretraining flow with LLaMA 3.2 instead of GPT-2
jupyter lab ch05/14_ch05_with_other_llms/ch05-llama32.ipynb

# Run ch05 pretraining flow with Qwen3
jupyter lab ch05/14_ch05_with_other_llms/ch05-qwen3.ipynb
```

---

## 5. Troubleshooting

### Diagnostic decision tree

```text
Problem?
├── Installation issue
│   └── Run python_environment_check.py, check Python >= 3.9
├── Download failure
│   └── Check internet, try backup URLs, see discussions/273
├── GPU/device error
│   └── Verify CUDA/MPS availability, try --device cpu
├── Shape mismatch error
│   └── Ensure model config matches downloaded weight size
├── DDP script error
│   └── Verify 2+ GPUs available, check CUDA_VISIBLE_DEVICES
└── Ollama evaluation fails
    └── Ensure Ollama is running locally (ollama serve)
```

### Error: `Model size not in ('124M', '355M', '774M', '1558M')`

**Cause:** An invalid GPT-2 model size string was passed to `download_and_load_gpt2()`.

**Solution:**
Use one of the four supported sizes exactly as shown:

```python
settings, params = download_and_load_gpt2("124M", "gpt2")
```

**Prevention:** The allowed sizes are `"124M"`, `"355M"`, `"774M"`, and `"1558M"`. These correspond to GPT-2 Small, Medium, Large, and XL respectively.

### Error: `Shape mismatch. Left: {shape}, Right: {shape}`

**Cause:** The model configuration dictionary does not match the pretrained weights being loaded. For example, trying to load GPT-2 355M weights into a model configured for 124M.

**Solution:**
Ensure your `GPT_CONFIG` matches the model size you downloaded:

```python
# For 355M weights, use 355M config
GPT_CONFIG_355M = {
    "vocab_size": 50257,
    "context_length": 1024,
    "emb_dim": 1024,
    "n_heads": 16,
    "n_layers": 24,
    "drop_rate": 0.0,
    "qkv_bias": True
}
```

**Prevention:** Always match config parameters to the model size string passed to `download_and_load_gpt2()`.

### Error: `RuntimeError: Ollama not running. Launch ollama before proceeding.`

**Cause:** The `ollama_evaluate.py` script checks for a running Ollama instance and fails if none is found.

**Solution:**

1. Install Ollama from [ollama.ai](https://ollama.ai)
2. Start the Ollama service: `ollama serve`
3. Pull a model: `ollama pull llama3`
4. Re-run the evaluation script

**Prevention:** Start Ollama before running any evaluation or dataset generation scripts.

### Error: `ZeroDivisionError` in DDP scripts

**Cause:** The DDP training scripts in Appendix A are designed for exactly 2 GPUs. Running with fewer causes a division error.

**Solution:**

```bash
# For torchrun
torchrun --nproc_per_node=2 DDP-script-torchrun.py

# For manual launch
CUDA_VISIBLE_DEVICES=0,1 python DDP-script.py
```

To use a different GPU count, uncomment lines 103-107 in the script.

**Prevention:** Always specify the correct number of available GPUs.

### Error: `find_freq_pair` crashes when `token_ids` length < 2

**Cause:** Known bug (GitHub issue #1002) where the BPE `find_freq_pair` function crashes on very short input sequences with fewer than 2 tokens.

**Solution:** Ensure input text is long enough to produce at least 2 tokens after encoding.

### SWA window size problem

**Cause:** Known issue (GitHub issue #1008) with sliding window attention window size calculation.

**Solution:** Check the `--sliding_window_size` parameter matches your intended window. The default is `1024`.

> [!CAUTION]
> When using the memory estimator scripts with `--swa_ratio`, the format must be `a:b` with non-negative integers where `a+b > 0`. Invalid formats like `1:` or `:1` will raise `ValueError: --swa_ratio must be in the form 'a:b' with nonnegative integers and a+b>0`.

### GPT-2 weight download fails

**Cause:** Network issues reaching the OpenAI checkpoint server at `openaipublic.blob.core.windows.net`.

**Solution:**
The download function automatically falls back to a Backblaze B2 backup URL. If both fail:

1. Check your internet connection
2. Visit [GitHub Discussions #273](https://github.com/rasbt/LLMs-from-scratch/discussions/273) for alternative download methods
3. Try downloading manually and placing files in the `gpt2/{size}/` directory

> [!WARNING]
> The GPT-2 checkpoint download requires `tensorflow` to parse the original TensorFlow checkpoint format. Ensure TensorFlow is installed even if you only use PyTorch for training.

### `[FAIL]` in environment check output

**Cause:** A dependency version does not match `requirements.txt` specifications.

**Solution:**

```bash
pip install -r requirements.txt --upgrade
```

Then re-run `python setup/02_installing-python-libraries/python_environment_check.py`.

### Error: `n_kv_groups must divide n_heads exactly`

**Cause:** The GQA, MLA, or SWA memory estimator was given an `--n_kv_groups` value that does not evenly divide `--n_heads`.

**Solution:** Choose `n_kv_groups` values that are divisors of `n_heads`. For example, with `--n_heads 32`, valid groups are 1, 2, 4, 8, 16, or 32.

### Error: `Invalid --trainable_layers argument`

**Cause:** An unsupported value was passed to `--trainable_layers` in a classification training script.

**Solution:**
Valid options depend on the script:
- `train_gpt.py`: `all`, `last_block`, `last_layer`
- `additional_experiments.py`: `all`, `last_block`, `last_two_blocks`, `last_layer`, `lora`, `lora_alternative`
- `train_bert_hf.py`: `all`, `last_block`, `last_layer`

---

## 6. FAQ

**Q: Do I need a GPU to use this repository?**

No. All code runs on CPU, though training will be significantly slower. For pretraining and fine-tuning, an NVIDIA GPU with CUDA support is strongly recommended. Apple Silicon (MPS) works for many operations but is not yet fully supported across all scripts (see GitHub issue #977).

**Q: Which GPT-2 model size should I start with?**

Start with GPT-2 Small (124M). It trains fastest, requires the least VRAM (~2 GB), and demonstrates all the same concepts as the larger models. Move to 355M for fine-tuning experiments in chapters 6-7.

**Q: Do I need the book to use the code?**

The code is fully functional without the book, and each chapter folder includes a detailed `README.md` with explanations. The book provides deeper theoretical background and step-by-step narrative.

**Q: How do I use the `llms_from_scratch` package vs. the chapter scripts?**

The `pkg/llms_from_scratch/` package consolidates all implementations into importable modules (e.g., `from llms_from_scratch.ch04 import GPTModel`). The chapter scripts (e.g., `ch04/01_main-chapter-code/gpt.py`) are standalone files that include all code inline for educational clarity. Use the package for building on top of the implementations; use the chapter scripts for learning.

**Q: Can I extend the code to other model architectures?**

Yes. Chapter 5 bonus sections demonstrate converting the GPT architecture to LLaMA 3.2, Qwen3, Gemma3, Gemma4, OLMo3, Tiny-Aya, and Qwen3.5 — each with standalone notebooks and from-scratch implementations.

**Q: How do I run the Jupyter notebooks?**

```bash
pip install -r requirements.txt
jupyter lab
```

Then navigate to the desired chapter folder and open the `.ipynb` file.

**Q: What is the `previous_chapters.py` file in each chapter?**

Each chapter's `previous_chapters.py` consolidates all classes and functions from prior chapters so the current chapter's code can import them. This avoids cross-chapter import dependencies while keeping each chapter self-contained.

**Q: How do I train on my own text data?**

Replace the text file in the data loading step. For pretraining, modify `ch05/03_bonus_pretraining_on_gutenberg/prepare_dataset.py` to point to your data directory. For fine-tuning, create a JSON file matching the format in `ch07/01_main-chapter-code/instruction-data.json`.

**Q: What are the `exercise-solutions.ipynb` files?**

These notebooks contain solutions to the exercises at the end of each chapter in the book. They are useful for checking your work or exploring alternative implementations.

**Q: What is `pixi.toml` and do I need it?**

`pixi.toml` is an alternative environment specification for the [pixi](https://pixi.sh) package manager. It pins Python 3.10 and `pytorch-cpu >=2.6.0` on conda-forge. You do not need pixi if you are using `pip` or `uv`. It provides a fully reproducible environment across Linux x64, macOS ARM64, and Windows x64.

**Q: What is the `reasoning-from-scratch/` folder?**

This is a bonus directory at the repository root for experimental reasoning implementations. Consult its README for current details.

---

_Generated May 2026 from rasbt/LLMs-from-scratch at 28c65cd. AI-generated documentation. The library is checked by sampling (three repositories so far); this guide was not individually reviewed. Verify against the project before relying on it. Unofficial; not affiliated with or reviewed by the project._