LLMs-from-scratch User Guide

1. Quick Start

LLMs-from-scratch is the official companion code repository for the book Build a Large Language Model From Scratch by Sebastian Raschka (Manning Publications). It provides complete, working implementations of a GPT-style large language model built step-by-step from raw text processing through pretraining, fine-tuning for classification, and instruction-following — all in pure PyTorch.

Key features:

• Build a GPT-2-class transformer from scratch in PyTorch
• Implement tokenization using OpenAI's tiktoken byte-pair encoding
• Pretrain on custom text corpora (Project Gutenberg, custom datasets)
• Load and use official OpenAI GPT-2 pretrained weights (124M to 1.5B parameters)
• Fine-tune for text classification (spam detection, IMDB sentiment)
• Fine-tune for instruction following with supervised fine-tuning (SFT)
• Implement advanced architectures: KV cache, grouped-query attention (GQA), multi-head latent attention (MLA), sliding window attention (SWA), mixture-of-experts (MoE), DeltaNet
• Extend to modern LLMs: LLaMA 3.2, Qwen3, Gemma3, Gemma4, OLMo3, Qwen3.5, Tiny-Aya

Prerequisites:

• Python 3.9 or newer
• A working pip or uv installation
• NVIDIA GPU recommended for training (CPU works for inference and small experiments)

Quickest path to running inference with pretrained GPT-2 weights:

git clone https://github.com/rasbt/LLMs-from-scratch.git
cd LLMs-from-scratch
pip install -r requirements.txt
pip install -e .
python ch05/01_main-chapter-code/gpt_generate.py --prompt "Hello, world" --device auto

Every effort moves you closer to the next level of understanding...

Tip

If you just want to explore the concepts interactively, open the Jupyter notebooks directly. Each chapter folder contains ch0X.ipynb files that walk through the code with explanations.

What to try next:

• Run the chapter 4 notebook (ch04/01_main-chapter-code/ch04.ipynb) to build the full GPT architecture
• Fine-tune for spam classification in chapter 6
• Build an instruction-following chatbot in chapter 7
• See the Admin Guide for architecture deep-dives and module reference

2. Installation

System Requirements

Install via pip (recommended)

git clone https://github.com/rasbt/LLMs-from-scratch.git
cd LLMs-from-scratch
pip install -r requirements.txt

Important

The requirements.txt includes tensorflow (used for loading OpenAI's original GPT-2 checkpoint format in chapters 5-7). On Intel macOS, a version-pinned torch and tensorflow-cpu variant are automatically selected via environment markers.

Install the companion package

The repository includes an installable Python package (llms_from_scratch) that consolidates all implementations from every chapter into reusable modules:

pip install -e .

This makes all chapter modules importable:

from llms_from_scratch.ch04 import GPTModel
from llms_from_scratch.ch05 import generate, text_to_token_ids, token_ids_to_text

Install with pixi (alternative)

The repo ships a pixi.toml for reproducible environments using pixi:

pixi install
pixi run jupyter lab

The pixi environment pins Python 3.10, pytorch-cpu >=2.6.0, and all book dependencies. To include testing dependencies:

pixi install -e tests

Install with uv (alternative)

uv pip install -r requirements.txt
uv pip install -e .

Docker environment

The setup/03_optional-docker-environment/ folder contains Docker setup instructions for running the code in a containerized environment. See the folder's README.md for details.

AWS SageMaker

A CloudFormation template is provided at setup/04_optional-aws-sagemaker-notebook/cloudformation-template.yml for launching a SageMaker notebook instance pre-configured with all dependencies.

Verify installation

Run the environment check script to verify all dependencies are correctly installed:

python setup/02_installing-python-libraries/python_environment_check.py

[OK] Your Python version is 3.10.12
[OK] torch 2.5.1
[OK] jupyterlab 4.3.5
[OK] tiktoken 0.9.0
[OK] matplotlib 3.10.0
[OK] tensorflow 2.18.0
[OK] tqdm 4.67.1
[OK] numpy 1.26.4
[OK] pandas 2.2.3
[OK] psutil 5.9.8

Warning

If any line shows [FAIL], install the correct version before proceeding. Mismatched versions — especially for torch and tensorflow — can cause subtle errors during weight loading.

Chapter-specific extras

Some bonus sections require additional packages. Each folder that needs them includes a requirements-extra.txt:

3. Core Usage

The repository follows the book's chapter structure. Each chapter builds on the previous one, progressively constructing a complete LLM pipeline.

Chapter 1 — Understanding Large Language Models

Introductory chapter. Contains reading recommendations at ch01/reading-recommendations.md. No code to run.

Chapter 2 — Working with Text Data

Covers tokenization and data loading for LLM training.

Main notebook: ch02/01_main-chapter-code/ch02.ipynb

Key implementations:

• GPTDatasetV1 — sliding window dataset that creates input/target pairs from tokenized text
• create_dataloader_v1() — convenience function wrapping the dataset with a PyTorch DataLoader
• Token and positional embeddings

import tiktoken
from llms_from_scratch.ch02 import create_dataloader_v1

tokenizer = tiktoken.get_encoding("gpt2")
dataloader = create_dataloader_v1(
    txt="Your training text here",
    batch_size=4,
    max_length=256,
    stride=128
)

Bonus sections:

Chapter 3 — Coding Attention Mechanisms

Implements self-attention from first principles through multi-head attention.

Main notebook: ch03/01_main-chapter-code/ch03.ipynb

Key classes (all in llms_from_scratch.ch03):

Note

The progression from SelfAttention_v1 to PyTorchMultiHeadAttention is intentionally pedagogical. Each version adds one concept: learnable projections, causal masking, dropout, multi-head splitting, and finally PyTorch's optimized kernel.

Bonus sections:

Chapter 4 — Implementing a GPT Model from Scratch

Assembles the full GPT architecture: LayerNorm, GELU, FeedForward, TransformerBlock, and GPTModel.

Main notebook: ch04/01_main-chapter-code/ch04.ipynb

The complete GPT-2 124M configuration:

GPT_CONFIG_124M = {
    "vocab_size": 50257,
    "context_length": 1024,
    "emb_dim": 768,
    "n_heads": 12,
    "n_layers": 12,
    "drop_rate": 0.1,
    "qkv_bias": True
}

Bonus sections — advanced attention and architecture variants:

Tip

Each advanced architecture variant in ch04/ includes standalone Python scripts with CLI arguments for experimentation. These scripts implement generation with KV caching and accept arguments like --emb_dim, --n_heads, --n_layers, and --max_new_tokens.

Running the KV-cache GPT variant:

python ch04/03_kv-cache/gpt_with_kv_cache.py

Running grouped-query attention with custom parameters:

python ch04/04_gqa/gpt_with_kv_gqa.py --emb_dim 768 --n_heads 12 --n_layers 12 --n_kv_groups 4 --max_new_tokens 200

Running mixture-of-experts:

python ch04/07_moe/gpt_with_kv_moe.py --num_experts 8 --num_experts_per_tok 2 --emb_dim 768 --n_layers 12

Chapter 5 — Pretraining on Unlabeled Data

Covers GPT-2 weight loading from OpenAI checkpoints, pretraining from scratch, and extending to modern architectures.

Main notebook: ch05/01_main-chapter-code/ch05.ipynb

Generate text with pretrained GPT-2 weights:

python ch05/01_main-chapter-code/gpt_generate.py --prompt "Every effort moves you" --device auto

Pretraining on Project Gutenberg:

# Step 1: Prepare the dataset
python ch05/03_bonus_pretraining_on_gutenberg/prepare_dataset.py \
    --data_dir gutenberg/data/raw \
    --output_dir gutenberg_preprocessed \
    --max_size_mb 500

# Step 2: Run pretraining
python ch05/03_bonus_pretraining_on_gutenberg/pretraining_simple.py \
    --data_dir gutenberg/data \
    --output_dir model_checkpoints \
    --n_epochs 1 \
    --batch_size 4 \
    --lr 0.0005

Hyperparameter tuning:

python ch05/05_bonus_hparam_tuning/hparam_search.py

Extending to other LLM architectures:

Training speed optimization (ch05/10):

Three progressive scripts demonstrate optimization techniques:

Chainlit chat interface:

pip install -r ch05/06_user_interface/requirements-extra.txt
chainlit run ch05/06_user_interface/app_orig.py

Chapter 6 — Fine-Tuning for Classification

Fine-tunes GPT-2 as a binary text classifier (spam detection).

Main notebook: ch06/01_main-chapter-code/ch06.ipynb

Run the standalone training script:

python ch06/01_main-chapter-code/gpt_class_finetune.py

The script downloads a spam SMS dataset, fine-tunes a GPT-2 124M model for binary classification, and saves the model as review_classifier.pth.

IMDB classification bonus (ch06/03):

Trains various models on IMDB sentiment data with extensive CLI options:

# Train GPT-2 on IMDB
python ch06/03_bonus_imdb-classification/train_gpt.py \
    --model_size "gpt2-small (124M)" \
    --weights pretrained \
    --trainable_layers last_block \
    --num_epochs 1

# Train BERT/DistilBERT/RoBERTa on IMDB
python ch06/03_bonus_imdb-classification/train_bert_hf.py \
    --model distilbert \
    --trainable_layers all \
    --num_epochs 1

Full CLI reference for train_gpt.py:

Full CLI reference for train_bert_hf.py:

Note

The additional_experiments.py script in ch06/02_bonus_additional-experiments/ supports LoRA fine-tuning via --trainable_layers lora with configurable --lora_rank (default 8) and --lora_alpha (default 8) parameters. It also supports gradient accumulation via --accumulation_steps.

Classification chat UI:

After training the classifier in chapter 6, launch the Chainlit interface:

pip install -r ch06/04_user_interface/requirements-extra.txt
chainlit run ch06/04_user_interface/app.py

Chapter 7 — Fine-Tuning to Follow Instructions

Implements supervised fine-tuning (SFT) for instruction following, preference tuning with DPO, and model evaluation.

Main notebook: ch07/01_main-chapter-code/ch07.ipynb

Run instruction fine-tuning:

python ch07/01_main-chapter-code/gpt_instruction_finetuning.py

This trains a GPT-2 355M model on the bundled instruction-data.json dataset and saves the result as gpt2-medium355M-sft.pth.

Evaluate model outputs using Ollama:

python ch07/01_main-chapter-code/ollama_evaluate.py \
    --file_path instruction-data-with-response.json

Warning

The evaluation script requires Ollama to be running locally. If Ollama is not running, the script raises RuntimeError: Ollama not running. Launch ollama before proceeding.

Dataset utilities (ch07/02):

Find and remove near-duplicate entries in instruction datasets:

python ch07/02_dataset-utilities/find-near-duplicates.py \
    --json_file instruction-examples.json \
    --threshold 0.9 \
    --remove_duplicates \
    --json_output_file cleaned-data.json

Preference tuning with DPO (ch07/04):

The dpo-from-scratch.ipynb notebook implements Direct Preference Optimization from scratch, using preference data generated via Ollama in create-preference-data-ollama.ipynb.

Dataset generation (ch07/05):

Generate synthetic instruction datasets using LLaMA 3 via Ollama or GPT-4 via the OpenAI API.

Instruction-following chat UI:

pip install -r ch07/06_user_interface/requirements-extra.txt
chainlit run ch07/06_user_interface/app.py

Appendices

Running multi-GPU DDP training (Appendix A):

# Using torchrun (recommended)
torchrun --nproc_per_node=2 appendix-A/01_main-chapter-code/DDP-script-torchrun.py

# Using CUDA_VISIBLE_DEVICES
CUDA_VISIBLE_DEVICES=0,1 python appendix-A/01_main-chapter-code/DDP-script.py

Important

The DDP scripts are designed for 2 GPUs by default. To use a different number of GPUs, uncomment the configuration code on lines 103-107 in the respective script.

4. Common Tasks

Load pretrained GPT-2 weights and generate text

from llms_from_scratch.ch04 import GPTModel
from llms_from_scratch.ch05 import (
    download_and_load_gpt2,
    load_weights_into_gpt,
    generate,
    text_to_token_ids,
    token_ids_to_text
)
import tiktoken
import torch

# Configuration for GPT-2 124M
GPT_CONFIG_124M = {
    "vocab_size": 50257,
    "context_length": 1024,
    "emb_dim": 768,
    "n_heads": 12,
    "n_layers": 12,
    "drop_rate": 0.0,
    "qkv_bias": True
}

# Download and load weights
settings, params = download_and_load_gpt2("124M", "gpt2")
model = GPTModel(GPT_CONFIG_124M)
load_weights_into_gpt(model, params)
model.eval()

# Generate
tokenizer = tiktoken.get_encoding("gpt2")
token_ids = generate(
    model=model,
    idx=text_to_token_ids("Hello, I am", tokenizer),
    max_new_tokens=50,
    context_size=GPT_CONFIG_124M["context_length"],
    eos_id=50256
)
print(token_ids_to_text(token_ids, tokenizer))

Fine-tune GPT-2 for binary text classification

1. Run the chapter 6 notebook or standalone script to train:

   python ch06/01_main-chapter-code/gpt_class_finetune.py
   

2. The script automatically downloads the SMS Spam dataset, creates balanced train/val/test splits, and fine-tunes GPT-2 124M
3. The trained model is saved as ch06/01_main-chapter-code/review_classifier.pth

Fine-tune GPT-2 for instruction following

1. Run the chapter 7 notebook or standalone script:

   python ch07/01_main-chapter-code/gpt_instruction_finetuning.py
   

2. Uses the bundled instruction-data.json (1,100 instruction/response pairs)
3. Saves the fine-tuned GPT-2 355M model as gpt2-medium355M-sft.pth

Run a Chainlit chat interface

Four chat UIs are available:

Tip

The Qwen3 chat interface at ch05/11_qwen3/qwen3-chat-interface/ also has a multi-turn variant (qwen3-chat-interface-multiturn.py) that remembers conversation history.

Compare KV cache memory across attention mechanisms

Each architecture variant in ch04/ includes a memory estimator script:

# GQA memory estimation
python ch04/04_gqa/memory_estimator_gqa.py --emb_dim 4096 --n_heads 32 --n_layers 32 --n_kv_groups 8

# MLA memory estimation
python ch04/05_mla/memory_estimator_mla.py --emb_dim 4096 --n_heads 32 --n_layers 32 --latent_dim 512

# SWA memory estimation
python ch04/06_swa/memory_estimator_swa.py --emb_dim 4096 --n_heads 32 --n_layers 32 --sliding_window_size 4096

# MoE memory estimation
python ch04/07_moe/memory_estimator_moe.py --emb_dim 4096 --hidden_dim 11008 --num_experts 8 --top_k 2

Plot memory comparison charts

python ch04/04_gqa/plot_memory_estimates_gqa.py
python ch04/05_mla/plot_memory_estimates_mla.py
python ch04/06_swa/plot_memory_estimates_swa.py --emb_dim 4096 --n_heads 32 --n_layers 32 --sliding_window_size 4096 --swa_ratio 5:1
python ch04/07_moe/plot_memory_estimates_moe.py --emb_dim 4096 --hidden_dim 11008
python ch04/08_deltanet/plot_memory_estimates_gated_deltanet.py --emb_dim 2048 --n_heads 16 --n_layers 48

Build a BPE tokenizer from scratch

jupyter lab ch02/05_bpe-from-scratch/bpe-from-scratch.ipynb

The notebook implements the complete byte-pair encoding algorithm step-by-step, without any external tokenizer library.

Pretrain on a large text corpus

# Download and prepare Project Gutenberg data
python ch05/03_bonus_pretraining_on_gutenberg/prepare_dataset.py \
    --data_dir gutenberg/data/raw \
    --output_dir gutenberg_preprocessed

# Run pretraining with checkpointing
python ch05/03_bonus_pretraining_on_gutenberg/pretraining_simple.py \
    --data_dir gutenberg/data \
    --output_dir model_checkpoints \
    --n_epochs 1 \
    --batch_size 4 \
    --lr 0.0005 \
    --eval_freq 100 \
    --save_ckpt_freq 100000 \
    --print_sample_iter 1000

Load weights for modern LLMs (LLaMA, Qwen3, Gemma)

Each model family in ch05/ has standalone notebooks:

# LLaMA 3.2
jupyter lab ch05/07_gpt_to_llama/standalone-llama32.ipynb

# Qwen3 (with and without KV cache)
jupyter lab ch05/11_qwen3/standalone-qwen3.ipynb
jupyter lab ch05/11_qwen3/standalone-qwen3-plus-kvcache.ipynb

# Qwen3 MoE variant
jupyter lab ch05/11_qwen3/standalone-qwen3-moe.ipynb

# Gemma 3
jupyter lab ch05/12_gemma3/standalone-gemma3.ipynb

# Gemma 4
jupyter lab ch05/17_gemma4/standalone-gemma4.ipynb

# OLMo 3
jupyter lab ch05/13_olmo3/standalone-olmo3.ipynb

# Tiny-Aya
jupyter lab ch05/15_tiny-aya/standalone-tiny-aya.ipynb

Implement DPO (Direct Preference Optimization) from scratch

# Step 1: Generate preference data using Ollama
jupyter lab ch07/04_preference-tuning-with-dpo/create-preference-data-ollama.ipynb

# Step 2: Run DPO training
jupyter lab ch07/04_preference-tuning-with-dpo/dpo-from-scratch.ipynb

Run chapter 5 pretraining with other LLM architectures

# Run ch05 pretraining flow with LLaMA 3.2 instead of GPT-2
jupyter lab ch05/14_ch05_with_other_llms/ch05-llama32.ipynb

# Run ch05 pretraining flow with Qwen3
jupyter lab ch05/14_ch05_with_other_llms/ch05-qwen3.ipynb

5. Troubleshooting

Diagnostic decision tree

Problem?
├── Installation issue
│   └── Run python_environment_check.py, check Python >= 3.9
├── Download failure
│   └── Check internet, try backup URLs, see discussions/273
├── GPU/device error
│   └── Verify CUDA/MPS availability, try --device cpu
├── Shape mismatch error
│   └── Ensure model config matches downloaded weight size
├── DDP script error
│   └── Verify 2+ GPUs available, check CUDA_VISIBLE_DEVICES
└── Ollama evaluation fails
    └── Ensure Ollama is running locally (ollama serve)

Error: Model size not in ('124M', '355M', '774M', '1558M')

Cause: An invalid GPT-2 model size string was passed to download_and_load_gpt2().

Solution: Use one of the four supported sizes exactly as shown:

settings, params = download_and_load_gpt2("124M", "gpt2")

Prevention: The allowed sizes are "124M", "355M", "774M", and "1558M". These correspond to GPT-2 Small, Medium, Large, and XL respectively.

Error: Shape mismatch. Left: {shape}, Right: {shape}

Cause: The model configuration dictionary does not match the pretrained weights being loaded. For example, trying to load GPT-2 355M weights into a model configured for 124M.

Solution: Ensure your GPT_CONFIG matches the model size you downloaded:

# For 355M weights, use 355M config
GPT_CONFIG_355M = {
    "vocab_size": 50257,
    "context_length": 1024,
    "emb_dim": 1024,
    "n_heads": 16,
    "n_layers": 24,
    "drop_rate": 0.0,
    "qkv_bias": True
}

Prevention: Always match config parameters to the model size string passed to download_and_load_gpt2().

Error: RuntimeError: Ollama not running. Launch ollama before proceeding.

Cause: The ollama_evaluate.py script checks for a running Ollama instance and fails if none is found.

Solution:

1. Install Ollama from ollama.ai
2. Start the Ollama service: ollama serve
3. Pull a model: ollama pull llama3
4. Re-run the evaluation script

Prevention: Start Ollama before running any evaluation or dataset generation scripts.

Error: ZeroDivisionError in DDP scripts

Cause: The DDP training scripts in Appendix A are designed for exactly 2 GPUs. Running with fewer causes a division error.

Solution:

# For torchrun
torchrun --nproc_per_node=2 DDP-script-torchrun.py

# For manual launch
CUDA_VISIBLE_DEVICES=0,1 python DDP-script.py

To use a different GPU count, uncomment lines 103-107 in the script.

Prevention: Always specify the correct number of available GPUs.

Error: find_freq_pair crashes when token_ids length < 2

Cause: Known bug (GitHub issue #1002) where the BPE find_freq_pair function crashes on very short input sequences with fewer than 2 tokens.

Solution: Ensure input text is long enough to produce at least 2 tokens after encoding.

SWA window size problem

Cause: Known issue (GitHub issue #1008) with sliding window attention window size calculation.

Solution: Check the --sliding_window_size parameter matches your intended window. The default is 1024.

Caution

When using the memory estimator scripts with --swa_ratio, the format must be a:b with non-negative integers where a+b > 0. Invalid formats like 1: or :1 will raise ValueError: --swa_ratio must be in the form 'a:b' with nonnegative integers and a+b>0.

GPT-2 weight download fails

Cause: Network issues reaching the OpenAI checkpoint server at openaipublic.blob.core.windows.net.

Solution: The download function automatically falls back to a Backblaze B2 backup URL. If both fail:

1. Check your internet connection
2. Visit GitHub Discussions #273 for alternative download methods
3. Try downloading manually and placing files in the gpt2/{size}/ directory

Warning

The GPT-2 checkpoint download requires tensorflow to parse the original TensorFlow checkpoint format. Ensure TensorFlow is installed even if you only use PyTorch for training.

[FAIL] in environment check output

Cause: A dependency version does not match requirements.txt specifications.

Solution:

pip install -r requirements.txt --upgrade

Then re-run python setup/02_installing-python-libraries/python_environment_check.py.

Error: n_kv_groups must divide n_heads exactly

Cause: The GQA, MLA, or SWA memory estimator was given an --n_kv_groups value that does not evenly divide --n_heads.

Solution: Choose n_kv_groups values that are divisors of n_heads. For example, with --n_heads 32, valid groups are 1, 2, 4, 8, 16, or 32.

Error: Invalid --trainable_layers argument

Cause: An unsupported value was passed to --trainable_layers in a classification training script.

Solution: Valid options depend on the script:

• train_gpt.py: all, last_block, last_layer
• additional_experiments.py: all, last_block, last_two_blocks, last_layer, lora, lora_alternative
• train_bert_hf.py: all, last_block, last_layer

6. FAQ

Q: Do I need a GPU to use this repository?

No. All code runs on CPU, though training will be significantly slower. For pretraining and fine-tuning, an NVIDIA GPU with CUDA support is strongly recommended. Apple Silicon (MPS) works for many operations but is not yet fully supported across all scripts (see GitHub issue #977).

Q: Which GPT-2 model size should I start with?

Start with GPT-2 Small (124M). It trains fastest, requires the least VRAM (~2 GB), and demonstrates all the same concepts as the larger models. Move to 355M for fine-tuning experiments in chapters 6-7.

Q: Do I need the book to use the code?

The code is fully functional without the book, and each chapter folder includes a detailed README.md with explanations. The book provides deeper theoretical background and step-by-step narrative.

Q: How do I use the llms_from_scratch package vs. the chapter scripts?

The pkg/llms_from_scratch/ package consolidates all implementations into importable modules (e.g., from llms_from_scratch.ch04 import GPTModel). The chapter scripts (e.g., ch04/01_main-chapter-code/gpt.py) are standalone files that include all code inline for educational clarity. Use the package for building on top of the implementations; use the chapter scripts for learning.

Q: Can I extend the code to other model architectures?

Yes. Chapter 5 bonus sections demonstrate converting the GPT architecture to LLaMA 3.2, Qwen3, Gemma3, Gemma4, OLMo3, Tiny-Aya, and Qwen3.5 — each with standalone notebooks and from-scratch implementations.

Q: How do I run the Jupyter notebooks?

pip install -r requirements.txt
jupyter lab

Then navigate to the desired chapter folder and open the .ipynb file.

Q: What is the previous_chapters.py file in each chapter?

Each chapter's previous_chapters.py consolidates all classes and functions from prior chapters so the current chapter's code can import them. This avoids cross-chapter import dependencies while keeping each chapter self-contained.

Q: How do I train on my own text data?

Replace the text file in the data loading step. For pretraining, modify ch05/03_bonus_pretraining_on_gutenberg/prepare_dataset.py to point to your data directory. For fine-tuning, create a JSON file matching the format in ch07/01_main-chapter-code/instruction-data.json.

Q: What are the exercise-solutions.ipynb files?

These notebooks contain solutions to the exercises at the end of each chapter in the book. They are useful for checking your work or exploring alternative implementations.

Q: What is pixi.toml and do I need it?

pixi.toml is an alternative environment specification for the pixi package manager. It pins Python 3.10 and pytorch-cpu >=2.6.0 on conda-forge. You do not need pixi if you are using pip or uv. It provides a fully reproducible environment across Linux x64, macOS ARM64, and Windows x64.

Q: What is the reasoning-from-scratch/ folder?

This is a bonus directory at the repository root for experimental reasoning implementations. Consult its README for current details.