autoresearch User Guide

autoresearch is a deliberately minimal, single-GPU LLM pretraining sandbox built by Andrej Karpathy as a derivative of nanochat. The whole point of the repo is to be edited autonomously by a coding agent (Claude Code, OpenAI Codex, or similar) running an overnight loop of: edit train.py, run a 5-minute experiment, keep or revert based on a single validation metric (val_bpb), and try the next idea.

This guide takes you from a fresh GPU box to a working autonomous research loop. It is intentionally short, because the codebase is intentionally short — two Python files (prepare.py, train.py), one markdown file (program.md) that programs the agent, plus an analysis notebook.

Important

autoresearch is a research toy. There are no model checkpoints, no multi-GPU support, no CLI flags on the training script, and no API. Every "knob" the agent touches is a module-level constant inside train.py. If you are looking for a production training framework, look elsewhere.

1. Quick Start

On any single-GPU NVIDIA Linux box with Python 3.10+ and uv installed, four commands stand the project up end-to-end:

git clone https://github.com/karpathy/autoresearch.git
cd autoresearch
uv sync
uv run prepare.py
uv run train.py

The first three commands fetch the repo and install dependencies (PyTorch 2.9.1 on the cu128 index, plus eight other packages). uv run prepare.py downloads 11 Parquet data shards from Hugging Face and trains a BPE tokenizer (~40 s). uv run train.py then trains an 80M-parameter GPT for exactly 5 minutes of steady-state wall-clock time and prints a summary.

Expected tail of the final run:

---
val_bpb:          0.997900
training_seconds: 300.1
total_seconds:    325.9
peak_vram_mb:     45060.2
mfu_percent:      39.80
total_tokens_M:   499.6
num_steps:        953
num_params_M:     50.3
depth:            8

If you see a val_bpb: line at the end, the rig works and you can hand the project off to an autonomous agent (see Section 4).

Tip

The fastest way to confirm a fresh setup is healthy is to run uv run prepare.py --num-shards 2 first. That downloads only the two shards needed to train the tokenizer (~30 s of network), proves your toolchain works, and gives the training run something to read. You can scale up to --num-shards 10 afterwards.

Note

Every command in this guide is shown as uv run <script>.py. That wrapper executes inside the project virtualenv automatically. If you prefer, you can source .venv/bin/activate once and drop the uv run prefix.

2. Installation

2.1 Hardware and platform requirements

Warning

There is no CPU, MPS, ROCm, or multi-GPU code path in this repository. Running on a Mac, AMD GPU, or Windows-without-CUDA will fail at torch.cuda.get_device_capability() very early. Use one of the community forks listed in Section 5.5 instead of filing a bug.

2.2 Software stack

pyproject.toml pins the following minimums:

Important

The PyTorch wheel comes from the pytorch-cu128 index, which means your driver must support CUDA 12.8. Verify with nvidia-smi before uv sync. Older drivers will sync successfully but fail at import torch during prepare.py.

2.3 Install uv

macOS / Linux

curl -LsSf https://astral.sh/uv/install.sh | sh

Windows (PowerShell)

powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"

Already installed

uv --version

2.4 Clone and install dependencies

git clone https://github.com/karpathy/autoresearch.git
cd autoresearch
uv sync

Expected output (abridged):

Resolved 47 packages in 1.24s
Prepared 9 packages in 3.2s
Installed 47 packages in 11s

uv sync reads pyproject.toml, resolves uv.lock, and creates .venv/ with the CUDA 12.8 PyTorch build.

2.5 Verify the GPU is visible

uv run python -c "import torch; print(torch.cuda.is_available(), torch.cuda.get_device_name(0))"

Expected output:

True NVIDIA H100 80GB HBM3

If this prints False, fix your CUDA driver before continuing — every later command depends on the GPU.

3. Configuration

autoresearch has no configuration files, no environment variables, and no CLI flags on train.py. Configuration lives in two places:

1. prepare.py — fixed constants that define the evaluation harness (treat as read-only).
2. train.py — module-level hyperparameters near the bottom of the file (intended to be edited).

The agent edits train.py only. program.md (the agent prompt) declares prepare.py off-limits because it owns the metric.

3.1 The two CLI flags that exist (prepare.py only)

3.2 Fixed constants in prepare.py (do not edit)

These are the evaluation harness. The agent's program.md rule set forbids touching them, because changing any of them silently invalidates comparisons across runs.

3.3 Hyperparameters in train.py (the knobs the agent turns)

These live in a single block under the comment # Hyperparameters (edit these directly, no CLI flags needed):

Note

Learning rates are tuned at n_embd = 768 (DEPTH=8, ASPECT_RATIO=64, HEAD_DIM=128). When you change model size, setup_optimizer rescales all AdamW LRs by (model_dim / 768) ** -0.5, and Muon's per-parameter step adds a max(1.0, rows / cols) ** 0.5 factor. You generally do not need to compensate by hand.

Tip

Because every knob is a constant, every experiment is fully described by a git diff. That is the design — it lets the agent (and you) read run history just by reading commit messages.

4. Core Usage

autoresearch runs in two modes:

1. Manual — you run uv run train.py, read the output, hand-edit constants, re-run. Useful for sanity checks.
2. Autonomous — you point an AI coding agent (Claude Code, OpenAI Codex) at program.md and let it iterate overnight.

The autonomous mode is what the project was built for; the manual mode is mostly a debugging tool.

4.1 One-time data and tokenizer prep

Before any training, you need data shards on disk and a trained tokenizer. Both are produced by prepare.py and cached in ~/.cache/autoresearch/:

uv run prepare.py

Default behaviour:

1. Download 10 training shards plus shard_06542.parquet (validation) from karpathy/climbmix-400b-shuffle.
2. Train a BPE tokenizer with rustbpe on up to 1 GB of training text.
3. Convert merges into a tiktoken.Encoding and pickle to ~/.cache/autoresearch/tokenizer/tokenizer.pkl.
4. Compute and save a token_bytes.pt lookup tensor used for the val_bpb metric.
5. Run a tokenizer round-trip sanity check on "Hello world! Numbers: 123. Unicode: 你好".

Expected output (trimmed):

Cache directory: /home/you/.cache/autoresearch

Data: downloading 11 shards (0 already exist)...
  Downloaded shard_00000.parquet
  ...
  Downloaded shard_06542.parquet
Data: 11/11 shards ready at /home/you/.cache/autoresearch/data

Tokenizer: training BPE tokenizer...
Tokenizer: trained in 38.2s, saved to /home/you/.cache/autoresearch/tokenizer/tokenizer.pkl
Tokenizer: building token_bytes lookup...
Tokenizer: saved token_bytes to /home/you/.cache/autoresearch/tokenizer/token_bytes.pt
Tokenizer: sanity check passed (vocab_size=8192)

Done! Ready to train.

Tip

prepare.py is idempotent. Re-running it skips already-downloaded shards and skips tokenizer training if both tokenizer.pkl and token_bytes.pt already exist. It is safe to re-run after a crash.

Important

tokenizer.pkl and token_bytes.pt are a pair. If only one is present, prepare.py will retrain from scratch. Do not delete just one — either keep both or wipe the whole ~/.cache/autoresearch/tokenizer/ directory.

4.2 Manual training run

After prepare.py succeeds, train manually with:

uv run train.py

The script:

1. Loads the tokenizer from ~/.cache/autoresearch/tokenizer/.
2. Builds a GPT model from the train.py constants (no CLI flags).
3. Compiles with torch.compile(dynamic=False) — the first ~10 steps are excluded from the timing budget.
4. Trains for exactly TIME_BUDGET = 300 seconds of steady-state wall-clock.
5. Runs evaluate_bpb over EVAL_TOKENS of the pinned validation shard.
6. Prints a final summary block.

A typical streaming log:

Vocab size: 8,192
Model config: {'sequence_len': 2048, 'vocab_size': 8192, 'n_layer': 8, 'n_head': 6, 'n_kv_head': 6, 'n_embd': 768, 'window_pattern': 'SSSL'}
Parameter counts:
  wte                     : 6,291,456
  value_embeds            : 25,165,824
  lm_head                 : 6,291,456
  transformer_matrices    : 42,467,328
  scalars                 : 16
  total                   : 80,216,080
Estimated FLOPs per token: 6.03e+08
Scaling AdamW LRs by 1/sqrt(768/768) = 1.000000
Time budget: 300s
Gradient accumulation steps: 2

step 00000 (0.0%) | loss: 9.123456 | lrm: 1.00 | dt: 1034ms | tok/sec: 507,123 | mfu: 39.8% | epoch: 1 | remaining: 300s
...
step 00953 (100.0%) | loss: 1.892100 | lrm: 0.00 | ...

---
val_bpb:          0.997900
...

Tip

Redirect the full log to a file with uv run train.py > run.log 2>&1. Then grep "^val_bpb:" run.log reads only the metric. This is exactly what the autonomous loop does so the streaming progress lines do not flood the agent's context.

4.3 Autonomous loop

In a checkout you do not mind being mutated, launch your coding agent with permissions disabled (so it can edit train.py and shell-execute without prompting), then say something like:

Hi, have a look at program.md and let's kick off a new experiment! Let's do the setup first.

The agent will read program.md, agree on a run tag (e.g. mar5), create a branch autoresearch/mar5, verify data exists, and start looping. Each loop iteration:

1. Inspect git state.
2. Edit train.py with one experimental idea.
3. git commit the change.
4. uv run train.py > run.log 2>&1.
5. grep "^val_bpb:\|^peak_vram_mb:" run.log to read metrics.
6. Append a row to results.tsv (kept untracked).
7. If val_bpb improved (lower is better), advance the branch. Otherwise git reset and try the next idea.

Expected throughput on an H100: roughly 12 experiments per hour, or about 100 experiments per 8-hour overnight session.

Important

program.md instructs the agent never to pause for confirmation once the loop starts. To stop, kill the agent process — it will not stop on its own. This is intentional, so you can leave it running unattended overnight.

Caution

The agent is given autonomous write access to the working tree and shell. It runs git reset regularly and edits train.py freely. Only run it inside a checkout you are willing to lose. Never aim it at a branch with work you care about.

4.4 The program.md "skill file"

program.md is the human-editable programming surface for the agent. The default program.md shipped with the repo is a minimal baseline — one agent, one loop, one metric. Typical evolutions:

• Tell the agent to focus on a specific region of the search space first.
• Add minimum-improvement thresholds before keeping a change.
• Constrain edits to architecture-only or optimizer-only.
• Spawn parallel agents on different branches (autoresearch/mar5-gpu0, autoresearch/mar5-gpu1, …) with different program.md strategies per GPU.

Note

program.md is part of the repo. If you change it, commit the change so future runs are reproducible. It is the only file other than train.py that meaningfully drives experimental behaviour.

5. Common Tasks

5.1 Smoke-test a fresh install

uv run prepare.py --num-shards 2
uv run train.py > run.log 2>&1
grep "^val_bpb:" run.log

The smaller --num-shards 2 value still satisfies the len(parquet_files) >= 2 precondition for tokenizer training and shaves several minutes off the first prep. Expected output:

val_bpb:          0.997900

(The exact number depends on your hardware. On slower GPUs you fit fewer steps in 300 s, so val_bpb will be higher.)

5.2 Resume a crashed agent loop

If your agent process exited or was killed mid-experiment on branch autoresearch/mar5:

git checkout autoresearch/mar5
# Restart your agent and re-prompt it with program.md.

results.tsv is intentionally untracked, so it survives git reset/git checkout and your historical log is preserved across restarts.

5.3 Start a fresh tagged run

The branch name autoresearch/<tag> must not already exist. Pick a new tag:

git checkout master
git pull
# Prompt the agent with a new tag, e.g. "mar5b" or "mar5-v2".

5.4 Tune for smaller GPUs

The default config peaks at ~44 GB VRAM. To run on a 12/16/24 GB card:

Warning

Reducing MAX_SEQ_LEN or VOCAB_SIZE requires editing prepare.py, which program.md declares off-limits to the agent. Do these edits yourself, manually, before starting the agent. After editing prepare.py, wipe ~/.cache/autoresearch/tokenizer/ so the new vocab size is rebuilt.

5.5 Switch to a community fork for non-CUDA hardware

The README lists these forks for hardware the upstream repo does not support:

There is also an open request (#519) to recognize an Ascend-compatible fork in the README's notable-forks list.

5.6 Compare two experiment branches

Each autoresearch/<tag> branch carries its own untracked results.tsv. To diff them, copy out before switching:

git checkout autoresearch/mar5
cp results.tsv /tmp/mar5.tsv
git checkout autoresearch/mar5b
cp results.tsv /tmp/mar5b.tsv
diff /tmp/mar5.tsv /tmp/mar5b.tsv

For a richer comparison, use analysis.ipynb — pandas and matplotlib are already in the environment from uv sync.

6. Troubleshooting

The training loop prints progress on a single carriage-returning line, so failures usually surface either as a Python traceback or as the literal string FAIL. Below are the failure modes confirmed in the source.

Error: torch.cuda.is_available() returns False

Cause: PyTorch cannot see a CUDA device. Either no GPU is attached, the NVIDIA driver is missing, or the driver is older than what the pinned torch==2.9.1+cu128 wheel requires.

Solution: Install or update the NVIDIA driver. Verify with nvidia-smi. Do not downgrade PyTorch — the Muon and AdamW fused steps rely on torch.compile behaviour from a recent version.

Prevention: Keep your driver current. If you are on a shared cluster, check that CUDA_VISIBLE_DEVICES is not set to the empty string in your shell.

Error: kernel-loading failure inside kernels.get_kernel(...)

Cause: train.py picks varunneal/flash-attention-3 for Hopper ((9, 0)) and kernels-community/flash-attn3 for everything else. If neither builds for your GPU, the import fails.

Solution: Run nvidia-smi --query-gpu=compute_cap --format=csv,noheader. If it reports below sm_80 (Ampere), this repo cannot run on your hardware. Use a fork from Section 5.5 or switch GPUs.

Prevention: This repo targets H100 first, A100 second. Older datacenter GPUs (V100) and pre-Ampere consumer cards (RTX 20-series and earlier) are not supported.

Error: ValueError: Invalid input type: <class '...'>

Cause: Raised at prepare.py:241 (line numbers approximate). Your code called Tokenizer.encode(x) where x is neither str nor list. The dataloader passes lists, so this only appears in custom code that calls tokenizer.encode with a generator, numpy array, or similar.

Solution: Coerce to str or list[str] before passing.

Prevention: Stick to the documented input types.

Error: Tokenizer: need at least 2 data shards (1 train + 1 val). Download more data first.

Cause: prepare.py requires at least 2 parquet files to train. You ran --num-shards 0, or your downloads all failed.

Solution:

uv run prepare.py --num-shards 2

Prevention: Default --num-shards 10 already protects you. Avoid setting it below 2.

Error: AssertionError: No parquet files found. Run prepare.py first.

Cause: train.py opened a dataloader before any data was on disk.

Solution: Run uv run prepare.py before uv run train.py. Required after every fresh checkout or after wiping ~/.cache/autoresearch/.

Prevention: Always run prepare.py first on a new machine. The Quick Start in this guide does this.

Error: download retries exhausted; some shard_NNNNN.parquet files missing

Cause: download_single_shard retries 5 times with exponential backoff (2^attempt seconds — so 2, 4, 8, 16 s between attempts). Hugging Face rate limits or 5xx errors can exhaust those attempts. Issue #215 tracks hardening this further (e.g. checksumming the cached shards).

Solution: Re-run prepare.py; only missing shards are re-fetched. For large jobs, lower the worker count to avoid rate-limit triggers:

uv run prepare.py --num-shards 10 --download-workers 2

Prevention: Use --download-workers 2 for --num-shards -1 (full dataset).

Error: training prints FAIL and exits with code 1

Cause: A guard at the end of each step: if train_loss is NaN or greater than 100, the run prints FAIL and exits. Triggered by numerically unstable hyperparameters (LR far too high, broken normalization, etc.).

Solution: This is the intended behaviour in autonomous mode — the agent reads an empty val_bpb line, marks the experiment as crash, and git resets. If you are running manually, lower the offending LR or revert the change.

Prevention: Make small LR moves; reserve aggressive architecture changes for cases where the baseline is stable.

peak_vram_mb climbs across long runs

Cause: Python's GC is disabled after step 0 (gc.disable() in the training loop) to avoid GC-induced stalls, with a manual gc.collect() every 5,000 steps. At default config, 5-minute runs finish in ~1,000 steps so this never matters — but if you cut batch size dramatically you may run far more steps without ever hitting a manual GC.

Solution: Either raise DEVICE_BATCH_SIZE (fewer steps), or change the (step + 1) % 5000 == 0 condition in train.py to a smaller interval.

Prevention: Don't increase total step count by an order of magnitude without also tightening the manual GC schedule.

mfu_percent very low (< 20%) on H100

Cause: On H100 the expected MFU is roughly 40%. Much lower usually means the FA3 kernel didn't load and attention fell back to a slower path, or torch.compile is recompiling because of dynamic shapes the agent introduced.

Solution: Check the startup log for kernels lines indicating which FA3 backend loaded. If it failed, run uv sync --reinstall and retry. If torch.compile is recompiling repeatedly, find the agent commit that introduced dynamic shapes and revert.

Prevention: Keep tensor shapes static. The fused steps are decorated @torch.compile(dynamic=False, fullgraph=True); do not relax that.

Tokenizer hangs at the sanity-check assertion

Cause: The end of train_tokenizer() does enc.decode(enc.encode_ordinary("Hello world! Numbers: 123. Unicode: 你好")) == .... A hang there usually means rustbpe failed to link or tiktoken failed to import — not a real round-trip failure.

Solution: uv sync --reinstall.

Prevention: Don't mix pip install with uv sync in the same virtualenv.

Issue #35 — "Why not fixed FLOPs budget?"

Status: Open. The chosen design fixes wall-clock time rather than FLOPs, which means slower hardware sees fewer tokens and reaches a higher val_bpb than fast hardware on identical code. Discussion is in the upstream issue; the choice is intentional and tied to the autonomous-loop comparability story (Section 7 FAQ).

7. FAQ

Q: Can I run this on a MacBook or M-series chip?

Not directly. The code hardcodes torch.device("cuda") and Flash Attention 3. Use miolini/autoresearch-macos or trevin-creator/autoresearch-mlx.

Q: Can I run this on AMD?

Not directly. Use andyluo7/autoresearch.

Q: Can I run multiple GPUs?

No. The repo is explicitly single-GPU. The parent nanochat repo supports distributed training; use that if you need DDP/FSDP.

Q: Why is the training time hardcoded to 5 minutes?

So that every experiment uses the same wall-clock budget on a given machine. Tradeoffs between depth, batch size, and step count become directly comparable. The downside is that results are not directly comparable to other people on different hardware — but that is not the goal. The goal is to give an autonomous agent apples-to-apples decisions on your GPU.

Q: Why val_bpb and not val_loss?

Bits-per-byte is independent of vocabulary size. If the agent ever changed VOCAB_SIZE (which prepare.py rules out), cross-entropy in nats would no longer be a fair metric. BPB converts via loss / (ln(2) * total_bytes), which is comparable across tokenizer designs.

Q: Why is prepare.py off-limits to the agent?

Because it owns the metric. It defines evaluate_bpb, the pinned validation shard, and the fixed time budget. Letting the agent edit any of those is letting it improve the metric by changing the metric.

Q: Why does train.py have no CLI flags?

Because the agent's interaction model is "edit source, commit, run". CLI flags would be a second source of truth. Module-level constants mean git diff is the complete description of an experiment.

Q: Can I run two agents in parallel on one GPU?

Not recommended — they would contend for VRAM and torch.compile cache. The intended pattern is one agent per GPU. On a multi-GPU box, give each agent its own checkout (or git worktree) and its own branch tag.

Q: Where are model checkpoints saved?

They aren't. train.py trains for 5 minutes, evaluates, and exits. No checkpoints, no intermediate saves. The agent is optimizing a metric, not producing a deployable model.

Q: The dataset is 6,542 shards. Do I need all of it?

No. With the default 5-minute budget on H100 you process ~500M tokens, which is less than one shard's worth. --num-shards 10 is plenty of headroom. Use --num-shards -1 only when training a much larger model for much longer.

Q: Does the tokenizer need to be retrained per experiment?

No, and you can't — the tokenizer lives in prepare.py, which the agent cannot edit. The tokenizer is a fixed artifact of the eval harness; every experiment sees the same encoding.

Q: Can I commit results.tsv?

You can, but program.md asks the agent to keep it untracked so it survives git reset. If you want to share results, copy the file to a gist or a separate results repo.

Q: Is there a license?

README.md lists MIT at the bottom. Issue #523 (closed) and #395 (closed) tracked the request to add a separate LICENSE file — check the current repo state if you need a file-level licence artefact.

Q: What is WINDOW_PATTERN = "SSSL"?

A per-layer attention window pattern. L is full context (MAX_SEQ_LEN = 2048). S is half (1024). SSSL repeats across layers; the very last layer is always overridden to L so the final readout sees the full context. Changing this is one of the more impactful architecture knobs the agent can turn.

Q: Can the agent modify the optimizer?

Yes. The MuonAdamW class is defined inside train.py, so it is fair game. Optimizer changes are explicitly allowed by program.md.

Q: The agent keeps reverting crashed runs — is that normal?

Yes. Crashes (FAIL, OOM, Python exceptions) are logged with status=crash and reverted. It is only a bug if it happens on every run from the baseline, which means your environment is broken — re-do Section 2.

Q: How do I know my run actually hit the time budget?

Check training_seconds in the summary block. It should be very close to 300.0. Much less means the run exited early (usually FAIL). Much more is unreachable in normal flow because the loop exit condition is total_training_time >= TIME_BUDGET.

Q: Why are the first ~10 steps not counted?

The training loop guards if step > 10 before adding to total_training_time. torch.compile does graph capture and autotuning during those steps, which takes seconds. Excluding them keeps the measured "training time" reflective of steady state, not compilation.

8. Where to Look Next

For internals — how the model, optimizer, and dataloader actually work; how to scale the autonomous loop across multiple GPUs; security considerations for letting an agent edit your code overnight — see the companion karpathy__autoresearch__AdminGuide.md in this directory.