# nanochat User Guide

nanochat is the simplest experimental harness for training your own ChatGPT-style large language model (LLM) end to end on a single GPU node. One codebase carries you through every stage — tokenization, pretraining, fine-tuning, reinforcement learning, evaluation, efficient inference, and a ChatGPT-like web UI. You can train a GPT-2-capability model for roughly **$48 in compute (~2 hours on an 8×H100 node)** and then chat with it in your browser.

This guide shows you how to install nanochat, run the headline "speedrun" to build and talk to your own model, drive every command-line script, and troubleshoot the most common problems. It is written entirely from the source code at the documented commit, so every command, flag, and default reflects what the code actually does.

> [!NOTE]
> nanochat is **not** a configurable LLM framework. It is a single, minimal, readable, hackable codebase designed to be forked and modified. There are no plugin systems, model factories, or giant config objects — just Python scripts you run in sequence.

---

## Table of Contents

1. [Quick Start](#1-quick-start)
2. [What You Need](#2-what-you-need)
3. [Installation](#3-installation)
4. [The One Dial You Control: `--depth`](#4-the-one-dial-you-control---depth)
5. [The Full Pipeline at a Glance](#5-the-full-pipeline-at-a-glance)
6. [Everyday Usage: The Scripts](#6-everyday-usage-the-scripts)
7. [Talking to Your Model](#7-talking-to-your-model)
8. [Common Workflows](#8-common-workflows)
9. [Tuning for Your Hardware](#9-tuning-for-your-hardware)
10. [How It Works (Concepts)](#10-how-it-works-concepts)
11. [Troubleshooting](#11-troubleshooting)
12. [FAQ](#12-faq)
13. [Command & Flag Reference](#13-command--flag-reference)

---

## 1. Quick Start

If you have a freshly booted 8×H100 GPU node, the entire pipeline lives in a single script. This trains a GPT-2-grade model and prepares it for chat in about three hours.

```bash
# 1. Clone the repository
git clone https://github.com/karpathy/nanochat.git
cd nanochat

# 2. Run the full speedrun (setup + tokenizer + pretrain + SFT + eval)
bash runs/speedrun.sh
```

Because the run takes hours, launch it inside a `screen` session so it survives a dropped SSH connection:

```bash
screen -L -Logfile runs/speedrun.log -S speedrun bash runs/speedrun.sh
```

When it finishes, serve the chat web UI and open the printed URL in your browser:

```bash
source .venv/bin/activate
python -m scripts.chat_web
```

```text
Starting NanoChat Web Server
Temperature: 0.8, Top-k: 50, Max tokens: 512
Server ready at http://localhost:8000
```

> [!TIP]
> On a cloud box, connect using the node's **public IP** and the port, e.g. `http://209.20.xxx.xxx:8000/`. `localhost` only works if you are on the machine itself or have tunneled the port.

> [!IMPORTANT]
> A speedrun-grade model is a ~4×10¹⁹ FLOPs capability model — "a bit like talking to a kindergartener." It can answer simple factual questions and will happily hallucinate. This is expected; nanochat exists to teach the full training stack, not to compete with frontier models.

---

## 2. What You Need

### Hardware

| Setup | What it's for | Notes |
|-------|---------------|-------|
| **8×H100 GPU node** | The reference speedrun (~3 hours) | What `runs/speedrun.sh` targets. Hopper GPUs unlock Flash Attention 3. |
| **8×A100 GPU node** | Same pipeline, slightly slower | Fully supported; falls back to PyTorch SDPA attention. |
| **Single GPU** | Any stage, ~8× slower | Omit `torchrun`; the code automatically switches to gradient accumulation and produces ~identical results. |
| **CPU / Apple Silicon (MPS)** | Educational tiny runs only | Use `runs/runcpu.sh`. You will not get good results — this is a demo, not a real training run. |

> [!WARNING]
> Training real LLMs requires GPU compute and money. You will not get far on a MacBook. Treat CPU/MPS runs as a way to exercise the code paths, not to produce a usable model.

> [!CAUTION]
> If your GPUs have **less than 80 GB** of VRAM, the default settings will run out of memory (OOM). Reduce `--device-batch-size` (from `32` down to `16`, `8`, `4`, `2`, or `1`) until training fits. See [Tuning for Your Hardware](#9-tuning-for-your-hardware).

### Software

- **Python ≥ 3.10** (the project pins `torch==2.9.1`, targeting CUDA 12.8 or CPU).
- **[uv](https://docs.astral.sh/uv/)** for dependency management (the setup scripts install it automatically if missing).
- A **Linux** environment for GPU training. macOS is supported only for the small CPU/MPS demo.

---

## 3. Installation

nanochat uses **uv** for dependency management. Pick the install method that matches your hardware.

=== "CUDA (A100/H100/etc.)"

    ```bash
    uv sync --extra gpu
    source .venv/bin/activate
    ```

=== "CPU-only / Apple Silicon (MPS)"

    ```bash
    uv sync --extra cpu
    source .venv/bin/activate
    ```

=== "Development (adds test/plotting tools)"

    ```bash
    uv sync --extra gpu --group dev
    source .venv/bin/activate
    ```

The `cpu` and `gpu` extras are **mutually exclusive** — they pull `torch` from different package indexes (`download.pytorch.org/whl/cpu` vs. `.../cu128`). Pick exactly one.

> [!NOTE]
> If `uv` is not installed, the setup scripts run this for you:
> ```bash
> curl -LsSf https://astral.sh/uv/install.sh | sh
> ```
> You can also run it yourself before `uv sync`.

The `dev` group adds `pytest`, `matplotlib`, `ipykernel`, `python-dotenv`, and `transformers` (the last is needed to evaluate HuggingFace baselines like GPT-2).

### Verify your install

After activating the virtual environment, confirm the package imports and check which device and precision nanochat will auto-detect:

```bash
python -c "import nanochat, torch; from nanochat.common import COMPUTE_DTYPE, COMPUTE_DTYPE_REASON; print(torch.__version__); print(COMPUTE_DTYPE, '-', COMPUTE_DTYPE_REASON)"
```

```text
2.9.1+cu128
torch.bfloat16 - auto-detected: CUDA SM 90 (bf16 supported)
```

---

## 4. The One Dial You Control: `--depth`

The single most important concept in nanochat is that you describe the model you want with **one integer**: `--depth`, the number of transformer layers.

Everything else — the model width, number of attention heads, learning rates, training horizon (how many tokens to train on), batch size, and weight decay — is **calculated automatically** to produce a compute-optimal model. You ask for a bigger or smaller model; the code figures out the rest.

| `--depth` | Rough capability | Typical use |
|-----------|------------------|-------------|
| `4`–`6` | Toy model | CPU/MPS demo runs (`runs/runcpu.sh`) |
| `12` | ~GPT-1 sized | Fast research iteration (~5 min pretraining) |
| `20` | The `$100` speedrun model | `runs/speedrun.sh` default |
| `24`–`26` | ~GPT-2 capability | Beating the GPT-2 CORE score |

```bash
# A 12-layer model — the author's favorite scale for quick experiments
torchrun --standalone --nproc_per_node=8 -m scripts.base_train -- --depth=12
```

> [!TIP]
> When you change something in the code and want to know if it helped, re-run a `d12` (or `d16`) and compare. It's the fastest meaningful signal.

> [!IMPORTANT]
> `model_dim = depth × 64` (the `--aspect-ratio`, default 64), rounded up to a clean multiple of the attention head dimension. You normally never set the width directly — increasing `--depth` widens the model for you.

---

## 5. The Full Pipeline at a Glance

nanochat builds a chat model in stages. Each stage is one script you run, and each consumes the artifacts of the previous one. All intermediate files (datasets, tokenizer, checkpoints, reports) are written under a single base directory (default `~/.cache/nanochat`).

```text
   download data        train tokenizer        PRETRAIN              FINE-TUNE            (optional) RL          CHAT
 ┌────────────────┐   ┌────────────────┐   ┌──────────────┐   ┌──────────────────┐   ┌──────────────┐   ┌──────────────┐
 │ nanochat.dataset│──▶│ scripts.tok_train│─▶│scripts.base_ │──▶│ scripts.chat_sft │──▶│scripts.chat_rl│──▶│scripts.chat_ │
 │   (parquet      │   │  (BPE, GPT-4     │   │  train       │   │ (teach chat,     │   │ (GRPO on      │   │  cli / web   │
 │    shards)      │   │   style)         │   │ (base model) │   │  tools, MC)      │   │  GSM8K math)  │   │              │
 └────────────────┘   └────────────────┘   └──────────────┘   └──────────────────┘   └──────────────┘   └──────────────┘
        │                                          │                    │                     │
        │                                    base_eval             chat_eval             chat_eval
        │                                  (CORE, bpb,           (ARC, MMLU,            (re-measure)
        ▼                                   samples)              GSM8K, ...)
  tok_eval (compression)
```

A checkpoint produced by one stage feeds the next via a model **source**:

- **`base`** — the pretrained base model (output of `base_train`).
- **`sft`** — the supervised-fine-tuned chat model (output of `chat_sft`).
- **`rl`** — the reinforcement-learned chat model (output of `chat_rl`).

Chat and evaluation scripts select which one to load with `-i/--source` (e.g. `-i sft`).

> [!NOTE]
> All stage outputs are organized by a **model tag**, which defaults to `d<depth>` (e.g. `d20`). If you run multiple experiments, pass `--model-tag` to keep them separate.

---

## 6. Everyday Usage: The Scripts

All scripts are run as Python modules from the repository root. For multi-GPU training, prefix with `torchrun` and note the `--` separator that divides torchrun's own arguments from the script's arguments:

```bash
torchrun --standalone --nproc_per_node=8 -m scripts.base_train -- --depth=24 --fp8
#                                                              ^^ everything after -- goes to base_train
```

For single-GPU (or CPU/MPS) runs, drop `torchrun` entirely:

```bash
python -m scripts.base_train --depth=24
```

> [!IMPORTANT]
> The `--` after the module name is **required** with `torchrun`. Without it, your `--depth`, `--run`, etc. are swallowed by torchrun instead of reaching the script.

### 6.1 Download the pretraining data — `nanochat.dataset`

Downloads pretraining shards (parquet files) from the ClimbMix-400B dataset on demand. The validation shard is **always** downloaded automatically in addition to the train shards you request.

```bash
# Download 8 shards (~2B characters, enough to train the tokenizer)
python -m nanochat.dataset -n 8

# Download ~170 shards (enough for GPT-2 capability pretraining)
python -m nanochat.dataset -n 170
```

```text
Downloading 9 shards using 4 workers...
Target directory: /root/.cache/nanochat/base_data_climbmix
...
Done! Downloaded: 9/9 shards to /root/.cache/nanochat/base_data_climbmix
```

| Flag | Default | Meaning |
|------|---------|---------|
| `-n`, `--num-files` | `-1` | Number of **train** shards to download. `-1` downloads all 6542 shards (the entire dataset). |
| `-w`, `--num-workers` | `4` | Number of parallel download workers. |

Each shard is ~250M characters (~100 MB compressed). Downloads are resumable — already-present shards are skipped, and failed downloads retry with exponential backoff.

> [!TIP]
> You can kick off a large download in the background while the tokenizer trains, exactly as the speedrun does:
> ```bash
> python -m nanochat.dataset -n 170 &
> python -m scripts.tok_train
> wait
> ```

### 6.2 Train the tokenizer — `scripts.tok_train`

Trains a byte-pair-encoding (BPE) tokenizer in the style of the GPT-4 tokenizer, using the Rust-backed `rustbpe` library. Saves to `~/.cache/nanochat/tokenizer/`.

```bash
python -m scripts.tok_train --max-chars=2000000000 --vocab-size=32768
```

| Flag | Default | Meaning |
|------|---------|---------|
| `--max-chars` | `2000000000` | Maximum characters to train on (2 billion). |
| `--doc-cap` | `10000` | Maximum characters taken from any single document. |
| `--vocab-size` | `32768` | Vocabulary size (2¹⁵). |

After training, the script runs a sanity round-trip (encode → decode must match) and caches a token-bytes table used later for the bits-per-byte metric.

### 6.3 Evaluate the tokenizer — `scripts.tok_eval`

Reports your tokenizer's compression ratio against the GPT-2 and GPT-4 tokenizers across several text types (news, Korean, code, math, science, and held-out training/validation text). Takes no arguments.

```bash
python -m scripts.tok_eval
```

```text
Vocab sizes:
GPT-2: 50257
GPT-4: 100277
Ours: 32768

Comparison with GPT-2:
Text Type  Bytes    Tokens  Ratio   Tokens  Ratio   Diff %
news       ...      ...     ...     ...     ...     +x.x%   Ours
...
```

Higher compression ratio (more bytes per token) is better.

### 6.4 Pretrain the base model — `scripts.base_train`

The compute-heavy stage. Trains the GPT transformer on the pretraining data. This is what the leaderboard measures.

```bash
# The speedrun's pretraining command (GPT-2 grade, d24, with FP8)
torchrun --standalone --nproc_per_node=8 -m scripts.base_train -- \
    --depth=24 --target-param-data-ratio=8 --device-batch-size=16 --fp8 --run=my_run
```

```text
step 00000/21500 (0.00%) | loss: 10.981 | lrm: 0.03 | dt: 412.55ms | tok/sec: 2,541,872 | bf16_mfu: 41.20 | ...
...
Step 21500 | Validation bpb: 0.748331
Step 21500 | CORE metric: 0.2585
```

This script has the largest set of flags. The most important ones:

| Flag | Default | Meaning |
|------|---------|---------|
| `--depth` | `20` | Transformer depth (your one dial). |
| `--run` | `dummy` | wandb run name. `dummy` disables Weights & Biases logging. |
| `--fp8` | off | Enable FP8 training (requires an H100+ GPU). Faster pretraining. |
| `--device-batch-size` | `32` | Per-GPU batch size. **Reduce this to avoid OOM.** |
| `--target-param-data-ratio` | `12` | Tokens-to-parameters ratio that sets the training horizon. Lower = shorter training. |
| `--num-iterations` | `-1` | Explicit step count. Overrides the ratio/FLOPs horizon when set. |
| `--max-seq-len` | `2048` | Context length in tokens. |
| `--eval-every` | `250` | Steps between validation-loss evaluations. |
| `--core-metric-every` | `2000` | Steps between CORE-metric evaluations. |
| `--sample-every` | `2000` | Steps between sampling text from the model. |
| `--save-every` | `-1` | Steps between checkpoint saves. `-1` saves only at the end. |
| `--resume-from-step` | `-1` | Resume training from a saved step. |

See the full list in the [Command & Flag Reference](#13-command--flag-reference).

> [!TIP]
> The training horizon is chosen by precedence: `--num-iterations` (if set) → `--target-flops` (if set) → `--target-param-data-ratio` (the common default). You only set one.

### 6.5 Evaluate the base model — `scripts.base_eval`

Evaluates a base model on three things (any subset, comma-separated via `--eval`): the **CORE** metric, **bits per byte (bpb)** on train/val splits, and text **samples**.

```bash
# Evaluate your nanochat d24 model on 8 GPUs
torchrun --standalone --nproc_per_node=8 -m scripts.base_eval -- --model-tag d24 --device-batch-size=16

# Evaluate the real GPT-2 (124M) baseline for comparison (requires the dev group)
torchrun --nproc_per_node=8 -m scripts.base_eval -- --hf-path openai-community/gpt2

# Quick approximate single-GPU evaluation
python -m scripts.base_eval --model-tag d24 --device-batch-size=16 --max-per-task=100 --split-tokens=524288
```

| Flag | Default | Meaning |
|------|---------|---------|
| `--eval` | `core,bpb,sample` | Which evaluations to run (any of `core`, `bpb`, `sample`). |
| `--hf-path` | `None` | Evaluate a HuggingFace model instead (e.g. `openai-community/gpt2-xl`). |
| `--model-tag` | `None` | Which nanochat checkpoint to load (defaults to the largest available). |
| `--step` | `None` | Which step to load (defaults to the last). |
| `--max-per-task` | `-1` | Examples per CORE task. `-1` = all (slower, more accurate). |
| `--device-batch-size` | `32` | Batch size for the bpb evaluation. |
| `--split-tokens` | `40 × 524288` | Tokens to evaluate per split for bpb. |

The CORE results are written to a CSV under `~/.cache/nanochat/base_eval/`.

### 6.6 Fine-tune for chat — `scripts.chat_sft`

Supervised fine-tuning (SFT) teaches the base model the chat format: conversation special tokens, tool use, and multiple-choice answering. It trains on a mixture that includes SmolTalk conversations, synthetic identity conversations, MMLU, GSM8K, and spelling tasks.

```bash
torchrun --standalone --nproc_per_node=8 -m scripts.chat_sft -- --device-batch-size=16 --run=my_run
```

| Flag | Default | Meaning |
|------|---------|---------|
| `--run` | `dummy` | wandb run name (`dummy` disables logging). |
| `--num-iterations` | `-1` | Steps to train. `-1` = one full epoch over the mixture. |
| `--mmlu-epochs` | `3` | Epochs of MMLU in the training mixture (teaches multiple choice). |
| `--gsm8k-epochs` | `4` | Epochs of GSM8K (teaches math and tool use). |
| `--device-batch-size` | inherited | Per-GPU batch size (defaults to the pretraining value). |
| `--max-seq-len` | inherited | Context length (defaults to the pretraining value). |

> [!NOTE]
> Most SFT hyperparameters (batch sizes, learning rates) are **inherited from the pretrained checkpoint** unless you override them. The console logs each value it inherited or overrode.

The identity conversations give your model its "personality." The speedrun downloads a ready-made set:

```bash
curl -L -o $NANOCHAT_BASE_DIR/identity_conversations.jsonl \
  https://karpathy-public.s3.us-west-2.amazonaws.com/identity_conversations.jsonl
```

### 6.7 Evaluate the chat model — `scripts.chat_eval`

Runs the chat model against a suite of tasks: `ARC-Easy`, `ARC-Challenge`, `MMLU`, `GSM8K`, `HumanEval`, and `SpellingBee`. When all tasks run, it also reports the **ChatCORE** metric (mean centered accuracy, 0 = random baseline, 1 = perfect).

```bash
# Evaluate the SFT model on all tasks
torchrun --nproc_per_node=8 -m scripts.chat_eval -- -i sft

# Evaluate a single task
python -m scripts.chat_eval -i sft -a ARC-Easy

# Multiple specific tasks (pipe-separated)
python -m scripts.chat_eval -i sft -a "MMLU|GSM8K"
```

| Flag | Default | Meaning |
|------|---------|---------|
| `-i`, `--source` | *(required)* | Model to load: `sft` or `rl`. |
| `-a`, `--task-name` | `None` (all) | Task(s) to run. Use `|` to separate multiple. |
| `-t`, `--temperature` | `0.0` | Sampling temperature. |
| `-n`, `--num-samples` | `1` | Samples per problem. |
| `-x`, `--max-problems` | `None` | Cap on problems per task (for quick runs). |

### 6.8 Reinforcement learning (optional) — `scripts.chat_rl`

Improves math ability via a simplified GRPO/REINFORCE loop on GSM8K. This stage is optional and is **not** part of the default speedrun.

```bash
torchrun --standalone --nproc_per_node=8 -m scripts.chat_rl -- --run=my_run
```

| Flag | Default | Meaning |
|------|---------|---------|
| `--num-epochs` | `1` | Epochs over GSM8K. |
| `--num-samples` | `16` | Samples (rollouts) per question. |
| `--temperature` | `1.0` | Sampling temperature for rollouts. |
| `--eval-every` | `60` | Steps between pass@k evaluations. |
| `--save-every` | `60` | Steps between checkpoint saves. |

The RL model is saved under the `rl` source, which you then load with `chat_cli -i rl`, `chat_web -i rl`, or `chat_eval -i rl`.

---

## 7. Talking to Your Model

Once you have an SFT (or RL) checkpoint, there are two ways to chat.

### 7.1 Command line — `scripts.chat_cli`

A simple interactive REPL in your terminal. Intended for single-GPU use.

```bash
# Interactive mode
python -m scripts.chat_cli

# Single-shot: ask one question, get one answer, exit
python -m scripts.chat_cli -p "What is the capital of France?"
```

```text
NanoChat Interactive Mode
--------------------------------------------------
Type 'quit' or 'exit' to end the conversation
Type 'clear' to start a new conversation
--------------------------------------------------

User: Why is the sky blue?

Assistant: The sky appears blue because ...
```

| Flag | Default | Meaning |
|------|---------|---------|
| `-i`, `--source` | `sft` | Model to load: `sft` or `rl`. |
| `-p`, `--prompt` | `''` | Single prompt; prints one response and exits. Omit for interactive chat. |
| `-t`, `--temperature` | `0.6` | Sampling temperature. |
| `-k`, `--top-k` | `50` | Top-k sampling parameter. |
| `-g`, `--model-tag` | `None` | Specific checkpoint tag (defaults to the largest). |
| `-s`, `--step` | `None` | Specific step (defaults to the last). |

In interactive mode, type `clear` to reset the conversation or `quit`/`exit` to leave.

> [!TIP]
> nanochat models sometimes respond better if you greet them ("Hi") before asking your question. Try "What is the capital of France?" — a working speedrun model should answer Paris, and may even know the sky is blue.

### 7.2 Web UI — `scripts.chat_web`

A ChatGPT-style browser interface served by FastAPI. It distributes requests across multiple GPUs (one model replica per GPU).

```bash
# Single GPU (default)
python -m scripts.chat_web

# Spread across 4 GPUs
python -m scripts.chat_web --num-gpus 4
```

| Flag | Default | Meaning |
|------|---------|---------|
| `-n`, `--num-gpus` | `1` | Number of GPUs to load model replicas on. |
| `-i`, `--source` | `sft` | Model to load: `sft` or `rl`. |
| `-p`, `--port` | `8000` | Port to serve on. |
| `--host` | `0.0.0.0` | Bind address. |
| `-t`, `--temperature` | `0.8` | Default generation temperature. |
| `-k`, `--top-k` | `50` | Default top-k. |
| `-m`, `--max-tokens` | `512` | Default max tokens per response. |

Open the printed URL in your browser. The server exposes:

| Endpoint | Purpose |
|----------|---------|
| `GET /` | The chat UI (HTML page). |
| `POST /chat/completions` | Streaming chat API. |
| `GET /health` | Health check + worker pool status. |
| `GET /stats` | Worker pool statistics and GPU utilization. |

> [!CAUTION]
> The web server has **no authentication** and binds to `0.0.0.0` (all interfaces) by default. On a public cloud box, anyone who can reach the port can use your model. See the Admin Guide for hardening guidance before exposing it.

---

## 8. Common Workflows

### 8.1 Train and talk to a GPT-2-grade model (the speedrun)

This is the recommended first experience. Everything is in one script.

- [ ] Boot a fresh 8×H100 (or 8×A100) node.
- [ ] `git clone` the repo and `cd` into it.
- [ ] `bash runs/speedrun.sh` (ideally inside `screen`).
- [ ] Wait ~3 hours; watch `runs/speedrun.log` if you used screen logging.
- [ ] `source .venv/bin/activate && python -m scripts.chat_web`.
- [ ] Open the public URL and chat.

Internally the speedrun runs, in order: data download → tokenizer train/eval → `base_train --depth=24 --fp8` → `base_eval` → download identity data → `chat_sft` → `chat_eval -i sft` → generate the run report.

### 8.2 Run a quick experiment (research loop)

When you want to test a code change quickly, a `d12` model pretrains in ~5 minutes on 8 GPUs:

```bash
OMP_NUM_THREADS=1 torchrun --standalone --nproc_per_node=8 -m scripts.base_train -- \
    --depth=12 \
    --run="d12" \
    --model-tag="d12" \
    --core-metric-every=999999 \
    --sample-every=-1 \
    --save-every=-1
```

This logs to wandb (run name `d12`), evaluates CORE only on the final step, and skips intermediate sampling and checkpointing. Watch `val_bpb` and `core_metric` in wandb to judge whether your change helped.

### 8.3 Try it on a Mac / CPU (no GPU)

`runs/runcpu.sh` shrinks the model dramatically so a tiny run completes in a few tens of minutes. It trains a 4–6 layer model, runs a short SFT, and lets you chat.

```bash
bash runs/runcpu.sh
```

> [!WARNING]
> Results will be poor — this exists to exercise the code, not to produce a usable model. You may prefer to copy-paste the commands from the script one at a time.

### 8.4 Give your model a custom personality

Personality comes from the synthetic **identity conversations** mixed into SFT. To customize it, generate your own conversations (see `dev/gen_synthetic_data.py`) and point SFT at your file via the identity-conversations data path. The author's guides in the repo Discussions walk through this end to end.

### 8.5 Resume an interrupted pretraining run

If a long run was interrupted and you saved checkpoints (`--save-every`), resume from the last saved step:

```bash
torchrun --standalone --nproc_per_node=8 -m scripts.base_train -- \
    --depth=24 --model-tag=d24 --resume-from-step=10000
```

This reloads the model weights, optimizer state, and dataloader position from the checkpoint.

---

## 9. Tuning for Your Hardware

The single most common adjustment is **fitting training into your VRAM**. The lever is `--device-batch-size`.

```bash
# Default is 32. If you OOM, step it down:
torchrun --standalone --nproc_per_node=8 -m scripts.base_train -- --depth=24 --device-batch-size=16
# still OOM? try 8, then 4, 2, 1
```

Reducing `--device-batch-size` does **not** change your effective batch size or results — nanochat compensates with gradient accumulation to reach the same total batch size. It only changes how much memory each forward/backward pass uses (and makes each step a little slower).

> [!TIP]
> The `runs/miniseries.sh` and `runs/scaling_laws.sh` scripts already encode sensible `--device-batch-size` reductions at larger depths (e.g. 16 for depth ≥ 20, 8 for depth ≥ 28). Use them as a reference.

### Precision (dtype)

nanochat manages precision explicitly through a single global compute dtype, auto-detected from your hardware:

| Hardware | Default dtype | Notes |
|----------|--------------|-------|
| CUDA SM 80+ (A100, H100, …) | `bfloat16` | Native bf16 tensor cores. |
| CUDA SM < 80 (V100, T4, …) | `float32` | No bf16. Force fp16 with `NANOCHAT_DTYPE=float16` (enables a GradScaler). |
| CPU / MPS | `float32` | No reduced-precision tensor cores. |

Override with the `NANOCHAT_DTYPE` environment variable:

```bash
NANOCHAT_DTYPE=float32 python -m scripts.chat_cli -p "hello"          # force fp32
NANOCHAT_DTYPE=bfloat16 torchrun --nproc_per_node=8 -m scripts.base_train  # force bf16
```

> [!NOTE]
> Flash Attention 3 (the fast attention path) only activates on **Hopper (H100/H200)** GPUs running in **bf16**. On other GPUs nanochat automatically falls back to PyTorch's SDPA attention — correct, just slower. On the SDPA fallback, prefer `--window-pattern L` (full context) for best utilization.

### Single GPU

Omit `torchrun` and run with plain `python`. The code detects the single-process setting and uses gradient accumulation. Expect roughly 8× the wall-clock time of an 8-GPU node.

---

## 10. How It Works (Concepts)

You don't need this section to use nanochat, but it helps you read the logs and tune intelligently.

**Tokenizer.** A GPT-4-style BPE tokenizer (vocab 32768) trained on your data with the Rust `rustbpe` library, then used at inference via `tiktoken` for speed.

**Base model (pretraining).** A from-scratch GPT transformer with rotary positional embeddings, QK-norm, ReLU² MLPs, untied embedding/unembedding weights, Group-Query Attention, and a sliding-window attention pattern. It learns to predict the next token on a large web text corpus (ClimbMix). The headline quality metric is the **DCLM CORE score**; the loss is reported as **bits per byte (bpb)**, which is invariant to vocabulary size.

**Compute-optimal scaling.** From your `--depth`, nanochat derives the model width and uses scaling-law relationships (Chinchilla-style data:param ratio, batch-size and learning-rate scaling from the Power Lines and μP literature) to set the training horizon and hyperparameters automatically.

**SFT (chat fine-tuning).** Teaches the model the conversation protocol using special tokens (`<|user_start|>`, `<|assistant_start|>`, etc.), how to call a calculator tool, and how to answer multiple-choice questions — by training on a curated mixture (SmolTalk, identity, MMLU, GSM8K, spelling tasks).

**RL (optional).** A simplified GRPO/REINFORCE loop that samples multiple answers per GSM8K problem, rewards correct ones, and nudges the model toward higher accuracy.

**Inference Engine.** An efficient KV-cache generator that prefills the prompt once, then decodes new tokens. It also runs a small **tool-use state machine**: when the model emits a `<|python_start|>` … `<|python_end|>` block, the Engine evaluates the expression with a safe calculator and feeds the result back as `<|output_start|>` … `<|output_end|>` tokens.

**The report.** During a speedrun, each stage appends to a markdown report under the base directory, capturing system info, configs, metrics, and sample outputs — a complete record of the run.

---

## 11. Troubleshooting

### Error: `No dataset parquet files found, did you run dataset.py?`

**Cause:** Training or tokenizer scripts can't find pretraining shards in the base directory.
**Solution:** Download data first: `python -m nanochat.dataset -n 8` (tokenizer) or `-n 170` (full pretrain).
**Prevention:** Always run the dataset download step before tokenizer/base training. The speedrun does this for you.

### Warning: `DATASET UPGRADE REQUIRED`

**Cause:** nanochat switched from FinewebEdu-100B to ClimbMix-400B (March 4 2026). After a `git pull`, your old data directory no longer matches.
**Solution:** Re-download and re-train the tokenizer on the new data:
```bash
python -m nanochat.dataset -n 170
python -m scripts.tok_train
```
**Prevention:** Re-run the data and tokenizer steps whenever you pull a dataset-changing update.

### Out of memory (CUDA OOM) during training

**Cause:** `--device-batch-size` is too large for your VRAM (common below 80 GB).
**Solution:** Lower `--device-batch-size`: `32 → 16 → 8 → 4 → 2 → 1`. Results are unchanged (gradient accumulation compensates).
**Prevention:** At larger depths, start lower (e.g. `16` for depth ≥ 20). The provided run scripts encode good defaults.

### Warning: `Flash Attention 3 not available, using PyTorch SDPA fallback`

**Cause:** You're not on a Hopper (H100/H200) GPU in bf16. FA3 kernels are Hopper-only.
**Solution:** None needed — training is correct, just slower. If you see a further warning about sliding windows under SDPA, add `--window-pattern L`.
**Prevention:** Use H100-class GPUs in bf16 for the fastest path; otherwise accept the fallback.

### `AssertionError` about total batch size divisibility

**Cause:** `--total-batch-size` isn't divisible by `device_batch_size × max_seq_len × num_gpus`.
**Solution:** Use a power-of-two total batch size (e.g. `524288`) or let it auto-compute (`--total-batch-size=-1`), and keep `--device-batch-size` a power of two.
**Prevention:** Prefer the auto-computed batch size unless you have a specific reason to override.

### `AssertionError: Your PyTorch installation is not configured for CUDA but device_type is 'cuda'`

**Cause:** You installed the CPU extra (`uv sync --extra cpu`) but are requesting CUDA.
**Solution:** Reinstall with the GPU extra: `uv sync --extra gpu`. (The extras are mutually exclusive.)
**Prevention:** Match the extra to your hardware at install time.

### `FileNotFoundError: No checkpoints found in ...`

**Cause:** A chat/eval script tried to load a model stage that hasn't been trained yet (e.g. running `chat_cli` before `chat_sft`).
**Solution:** Run the prerequisite stage first, or point `--model-tag`/`-i source` at a checkpoint that exists.
**Prevention:** Follow the pipeline order: base → sft → (rl) before chatting.

### The web UI loads but I can't reach it from my laptop

**Cause:** You're using `localhost` from a remote machine, or the cloud firewall blocks the port.
**Solution:** Use the node's **public IP** and the port (e.g. `http://<public-ip>:8000/`), and open the port in your provider's firewall/security group.
**Prevention:** Note the exact URL the server prints and confirm the port is reachable.

### Chat responses look like gibberish / the model is "dumb"

**Cause:** A speedrun-grade model is genuinely tiny (~4×10¹⁹ FLOPs). Low quality is expected.
**Solution:** Train a deeper model (`--depth=26`), train on more data, or simply accept the toy-scale quality.
**Prevention:** Set expectations — nanochat teaches the stack; it is not a frontier model.

### `assert decoded == test_text` fails during tokenizer training

**Cause:** A round-trip encode/decode mismatch — usually a corrupted or incompatible tokenizer build.
**Solution:** Re-run `scripts.tok_train` cleanly; ensure `rustbpe`/`tiktoken` installed correctly via `uv sync`.
**Prevention:** Don't hand-edit tokenizer artifacts in the base directory.

### Calculator/tool output never appears in answers

**Cause:** The model only invokes the calculator when it emits a Python tool block, and the calculator only accepts simple math or `.count(...)` expressions (it refuses `**`, imports, and other unsafe patterns).
**Solution:** This is by design and safe. Ask arithmetic or letter-counting questions (e.g. "how many r in strawberry") to see tool use.
**Prevention:** None needed.

---

## 12. FAQ

**Q: How much does it cost to train a model?**
A GPT-2-grade speedrun is ~2 hours on an 8×H100 node. At ~$3/GPU/hr (~$24/hr for the node) that's roughly **$48**; on a spot instance it can be closer to **~$15**.

**Q: Do I need 8 GPUs?**
No. Any stage runs on a single GPU — just omit `torchrun`. It takes about 8× longer and produces nearly identical results via gradient accumulation.

**Q: Can I run this on my MacBook?**
Yes, for a tiny educational demo via `runs/runcpu.sh`. You will not get a usable model. Real training needs GPUs.

**Q: What's the difference between the `base`, `sft`, and `rl` models?**
`base` is the pretrained next-token predictor. `sft` is fine-tuned to chat, use tools, and answer multiple choice. `rl` further sharpens math via reinforcement learning. Chat scripts pick one with `-i/--source`.

**Q: What does `--depth` actually control?**
The number of transformer layers. Everything else (width, heads, learning rates, training length, batch size, weight decay) is derived automatically to keep the model compute-optimal.

**Q: Where are all my files stored?**
Under `~/.cache/nanochat` by default. Override with the `NANOCHAT_BASE_DIR` environment variable. This holds the dataset, tokenizer, checkpoints, eval results, and reports.

**Q: What is "CORE" and "bpb"?**
CORE is the DCLM benchmark score (the headline capability metric; GPT-2's is 0.256525). bpb is bits-per-byte validation loss — a vocab-size-invariant loss you watch during training.

**Q: How do I log training to Weights & Biases?**
Run `wandb login` once, then pass `--run=<name>` to a training script. The default `--run=dummy` disables wandb.

**Q: Why are there two attention paths?**
Flash Attention 3 is used on Hopper GPUs (fastest). Everywhere else (Ada, Blackwell, MPS, CPU), nanochat falls back to PyTorch SDPA automatically. Results are equivalent; speed differs.

**Q: Can I evaluate the real GPT-2 to compare?**
Yes: `torchrun --nproc_per_node=8 -m scripts.base_eval -- --hf-path openai-community/gpt2`. This requires the `dev` group (for `transformers`).

**Q: How do I give my model a personality?**
Mix custom synthetic "identity conversations" into SFT. See `dev/gen_synthetic_data.py` and the repo's Discussions guides on infusing identity.

**Q: Is the chat web server safe to expose publicly?**
It has basic abuse limits (message/length/temperature caps) but **no authentication** and binds to all interfaces. Don't expose it to the open internet without putting it behind your own auth/proxy. See the Admin Guide.

**Q: Can I train a bigger-than-GPT-2 model?**
Yes — increase `--depth` beyond 26. You'll likely need to reduce `--device-batch-size` to fit memory, and training will take longer.

**Q: What's the leaderboard?**
A community "GPT-2 speedrun" leaderboard tracking the wall-clock time to reach GPT-2 CORE capability on an 8×H100 node. `runs/speedrun.sh` is always the reference recipe. See `dev/LEADERBOARD.md`.

**Q: How do I resume a crashed run?**
Train with `--save-every=N` so checkpoints exist, then restart with `--resume-from-step=<step>` and the same `--model-tag`.

**Q: Where can I ask deeper questions about the code?**
The repo recommends [DeepWiki](https://deepwiki.com/karpathy/nanochat), the GitHub Discussions tab, and the `#nanochat` Discord channel.

---

## 13. Command & Flag Reference

### Module entry points

| Command | Purpose |
|---------|---------|
| `python -m nanochat.dataset` | Download pretraining data shards. |
| `python -m nanochat.report [reset\|generate]` | Reset or assemble the run report. |
| `python -m scripts.tok_train` | Train the BPE tokenizer. |
| `python -m scripts.tok_eval` | Evaluate tokenizer compression. |
| `python -m scripts.base_train` | Pretrain the base model. |
| `python -m scripts.base_eval` | Evaluate the base model (CORE/bpb/samples). |
| `python -m scripts.chat_sft` | Supervised fine-tuning for chat. |
| `python -m scripts.chat_rl` | Reinforcement learning on GSM8K. |
| `python -m scripts.chat_eval` | Evaluate the chat model. |
| `python -m scripts.chat_cli` | Chat over the command line. |
| `python -m scripts.chat_web` | Serve the chat web UI. |

### Run scripts

| Script | Purpose |
|--------|---------|
| `runs/speedrun.sh` | The reference GPT-2 speedrun (full pipeline). |
| `runs/runcpu.sh` | Tiny CPU/MPS demo run. |
| `runs/miniseries.sh [series_name]` | Train a series of models across depths 12–26. |
| `runs/scaling_laws.sh` | Scaling-law sweep across FLOPs budgets and depths. |

### `base_train` — full flag list

| Flag | Default | Meaning |
|------|---------|---------|
| `--run` | `dummy` | wandb run name (`dummy` = no logging). |
| `--device-type` | `""` | `cuda`/`cpu`/`mps` (empty = autodetect). |
| `--fp8` | off | Enable FP8 training (H100+ only). |
| `--fp8-recipe` | `tensorwise` | FP8 scaling recipe (`tensorwise` or `rowwise`). |
| `--depth` | `20` | Transformer depth. |
| `--aspect-ratio` | `64` | `model_dim = depth × aspect_ratio`. |
| `--head-dim` | `128` | Target attention head dimension. |
| `--max-seq-len` | `2048` | Max context length. |
| `--window-pattern` | `SSSL` | Sliding-window pattern tiled across layers (`L`=full, `S`=quarter). |
| `--num-iterations` | `-1` | Explicit step count (`-1` = disabled). |
| `--target-flops` | `-1.0` | Train to a target FLOPs budget. |
| `--target-param-data-ratio` | `12` | Tokens:params ratio (Chinchilla≈20). |
| `--device-batch-size` | `32` | Per-GPU batch size (reduce to avoid OOM). |
| `--total-batch-size` | `-1` | Total tokens/step (`-1` = auto-optimal). |
| `--embedding-lr` | `0.3` | Embedding LR (Adam). |
| `--unembedding-lr` | `0.008` | Unembedding LR (Adam). |
| `--weight-decay` | `0.28` | Weight decay (Muon). |
| `--matrix-lr` | `0.02` | Matrix-parameter LR (Muon). |
| `--scalar-lr` | `0.5` | Scalar-parameter LR. |
| `--warmup-steps` | `40` | LR warmup steps. |
| `--warmdown-ratio` | `0.65` | Fraction of training spent in LR warmdown. |
| `--final-lr-frac` | `0.05` | Final LR as a fraction of initial. |
| `--resume-from-step` | `-1` | Resume from this saved step. |
| `--eval-every` | `250` | Steps between val-bpb evals. |
| `--eval-tokens` | `80×524288` | Tokens used for val-loss eval. |
| `--core-metric-every` | `2000` | Steps between CORE evals. |
| `--core-metric-max-per-task` | `500` | Examples per task for CORE. |
| `--sample-every` | `2000` | Steps between text samples. |
| `--save-every` | `-1` | Steps between checkpoints (`-1` = end only). |
| `--model-tag` | `None` | Checkpoint directory name (defaults to `d<depth>`). |

> [!NOTE]
> For `chat_sft`, `chat_rl`, `chat_eval`, `base_eval`, `chat_cli`, and `chat_web` flag lists, see the relevant sections above ([6.4](#64-pretrain-the-base-model--scriptsbase_train)–[7.2](#72-web-ui--scriptschat_web)). Flags marked "inherited" default to the value baked into the checkpoint being loaded.

---

_Generated June 2026 from karpathy/nanochat at dc54a1a. AI-generated documentation. The library is checked by sampling (three repositories so far); this guide was not individually reviewed. Verify against the project before relying on it. Unofficial; not affiliated with or reviewed by the project._