# Llama 2 Reference Implementation — User Guide

The `meta-llama/llama` repository is Meta's original reference implementation
for the **Llama 2** family of large language models (7B, 13B, 70B, and their
`-chat` variants). It contains the model architecture in pure PyTorch, a
SentencePiece tokenizer wrapper, model-parallel weight loading via
`fairscale`, a `download.sh` script for fetching weights after license
acceptance, and two example scripts (`example_text_completion.py`,
`example_chat_completion.py`) that exercise the public `Llama` class.

> [!IMPORTANT]
> **This repository is the Llama 2 reference release and is now in
> maintenance mode.** Meta's active development has moved to the
> [`llama-models`](https://github.com/meta-llama/llama-models) repository
> for model definitions and to [`llama-stack`](https://github.com/meta-llama/llama-stack)
> for the production inference toolchain. New projects targeting Llama 3,
> Llama 3.1, Llama 3.2, or Llama 3.3 should use those instead. The code in
> this repo only loads Llama 2 era checkpoints (7B / 13B / 70B + chat
> variants) and should be treated as a learning artifact and a faithful
> minimal implementation of the Llama 2 architecture.

---

## 1. Quick Start

The fastest path from a fresh clone to a running text-completion call is
five steps: clone, install, request a download URL from Meta, run the
download script, and launch one of the example scripts under `torchrun`.

```bash
# 1. Clone
git clone https://github.com/meta-llama/llama.git
cd llama

# 2. Install in editable mode (pulls torch, fairscale, fire, sentencepiece)
pip install -e .

# 3. Request a license-bound download URL from Meta:
#    https://ai.meta.com/resources/models-and-libraries/llama-downloads/
#    You will receive an email containing a presigned URL.

# 4. Run the download helper (it will prompt for the URL and the model size)
./download.sh

# 5. Run a one-line text completion against the downloaded 7B base model
torchrun --nproc_per_node 1 example_text_completion.py \
    --ckpt_dir llama-2-7b/ \
    --tokenizer_path tokenizer.model \
    --max_seq_len 128 --max_batch_size 4
```

```text
Loaded in 12.42 seconds
I believe the meaning of life is
> to be happy, to find joy in the small things, and to live each day to the fullest.

==================================
```

> [!TIP]
> The 7B base model is the lowest-friction starting point — it loads on a
> single GPU (model-parallel size 1) and finishes a 64-token completion in a
> few seconds on consumer-grade Ampere/Ada hardware. Save the chat
> variants and 13B / 70B for after you have the toolchain working.

### What you get

- A faithful PyTorch implementation of the Llama 2 architecture: pre-norm,
  RMSNorm, rotary positional embeddings, grouped-query attention with KV
  caching, SwiGLU feed-forward.
- A model-parallel build path layered on top of `fairscale` so the same
  code runs on 1 GPU (7B), 2 GPUs (13B), or 8 GPUs (70B) by changing
  `--nproc_per_node`.
- A minimal public surface: one class (`Llama`), three methods
  (`build`, `text_completion`, `chat_completion`).
- Two ready-to-run examples that double as integration tests for your
  install.

### Prerequisites checklist

- [x] Python 3.8 or newer (3.10 works well; tested by community on 3.11)
- [x] CUDA-capable NVIDIA GPU(s) with at least 16 GB VRAM for 7B
- [x] PyTorch with CUDA + NCCL support (the default CUDA wheels)
- [x] `wget` and `md5sum` (or `md5` on macOS arm64) available in `PATH`
- [x] A presigned URL from Meta's license-acceptance flow

> [!NOTE]
> The download script verifies every checkpoint and the tokenizer with
> MD5 checksums (`md5sum -c checklist.chk` on Linux / `md5 checklist.chk`
> on macOS arm64). A failed checksum means the download was truncated —
> rerun `./download.sh`; `wget --continue` resumes partial transfers.

---

## 2. Installation

### 2.1 Minimum-friction install (recommended)

```bash
git clone https://github.com/meta-llama/llama.git
cd llama
pip install -e .
```

The `setup.py` declares the package name `llama` at version `0.0.1` and
reads runtime requirements from `requirements.txt`:

```text
torch
fairscale
fire
sentencepiece
```

> [!IMPORTANT]
> The `requirements.txt` deliberately does not pin versions. The repo was
> validated against `torch >= 2.0` and `fairscale >= 0.4.13`. If your
> distributed launch hangs or NCCL fails to initialize, the most common
> root cause is a torch / fairscale combination that disagrees about
> the model-parallel API. A clean conda environment usually resolves it.

### 2.2 Conda environment (clean baseline)

```bash
conda create -n llama python=3.10 -y
conda activate llama
git clone https://github.com/meta-llama/llama.git
cd llama
pip install -e .
```

### 2.3 GPU / driver requirements

| Model | Min GPUs | Recommended VRAM (each) | Model-parallel size |
|-------|---------:|------------------------:|--------------------:|
| `llama-2-7b` / `llama-2-7b-chat` | 1 | 16 GB | 1 |
| `llama-2-13b` / `llama-2-13b-chat` | 2 | 24 GB | 2 |
| `llama-2-70b` / `llama-2-70b-chat` | 8 | 80 GB (A100/H100 class) | 8 |

The model-parallel size is fixed by the **number of `consolidated.NN.pth`
shards** that ship with each checkpoint: 1 shard for 7B, 2 for 13B, 8 for
70B. `Llama.build` asserts that `model_parallel_size == len(checkpoints)` —
if you launch with the wrong `--nproc_per_node`, you will see the error
documented in Troubleshooting below.

### 2.4 Verifying the install

```bash
python -c "from llama import Llama, Dialog; from llama.model import ModelArgs, Transformer; from llama.tokenizer import Tokenizer; print('ok')"
```

```text
ok
```

> [!WARNING]
> Importing `llama` only verifies that the Python package is on the path —
> it does NOT initialize CUDA or load weights. The first real check is
> running an example script under `torchrun`. If `pip install -e .`
> succeeded but `import torch; torch.cuda.is_available()` returns
> `False`, fix your CUDA-enabled PyTorch install before downloading
> any weights.

---

## 3. Configuration

Llama's reference implementation is a research-grade library: there is no
config file, no environment-variable layer above what `torch.distributed`
already reads, and no plugin system. All configuration is **runtime
parameters passed at launch**.

### 3.1 Distributed runtime variables

These are read from the environment by `Llama.build` and are normally
populated for you by `torchrun`:

| Variable | Default | Read by | Purpose |
|----------|--------:|---------|---------|
| `WORLD_SIZE` | `1` | `Llama.build` | Total number of model-parallel ranks. Derives `model_parallel_size` when not passed explicitly. |
| `LOCAL_RANK` | `0` | `Llama.build` | Per-host rank used to call `torch.cuda.set_device()`. |

> [!NOTE]
> `torchrun --nproc_per_node N` sets `WORLD_SIZE=N` and `LOCAL_RANK` per
> spawned process automatically. You should not have to set them by hand
> unless you are launching the example script outside `torchrun`.

### 3.2 Build parameters (`Llama.build`)

| Parameter | Type | Default | Notes |
|-----------|------|--------:|-------|
| `ckpt_dir` | `str` | required | Directory containing `consolidated.*.pth` and `params.json`. |
| `tokenizer_path` | `str` | required | Path to the SentencePiece model file (e.g. `tokenizer.model`). |
| `max_seq_len` | `int` | required | Maximum input sequence length the model will accept. |
| `max_batch_size` | `int` | required | Maximum batch size — pre-allocates KV cache. |
| `model_parallel_size` | `Optional[int]` | `None` | If `None`, falls back to `WORLD_SIZE`. |
| `seed` | `int` | `1` | Manual seed; must be identical across ranks. |

### 3.3 Sampling parameters (per call)

| Parameter | Type | `text_completion` default | `chat_completion` default |
|-----------|------|--------------------------:|--------------------------:|
| `temperature` | `float` | `0.6` | `0.6` |
| `top_p` | `float` | `0.9` | `0.9` |
| `max_gen_len` | `Optional[int]` | `max_seq_len - 1` | `max_seq_len - 1` |
| `logprobs` | `bool` | `False` | `False` |
| `echo` | `bool` | `False` | n/a |

> [!TIP]
> Setting `temperature=0` switches the sampler from nucleus sampling
> (`sample_top_p`) to greedy `argmax`. Use this for deterministic-style
> evaluation runs where you want repeatable outputs.

For the full structural configuration of the model itself
(`ModelArgs.dim`, `n_layers`, `n_heads`, `multiple_of`, `ffn_dim_multiplier`,
`norm_eps`), see the **Configuration Reference** in the Administrator &
Developer Guide.

---

## 4. Core Usage

The two example scripts are the canonical entry points and they show every
moving part of the public surface.

### 4.1 `example_text_completion.py`

A four-prompt batch run that demonstrates the base (non-chat) text
completion API.

**Signature** (from `main` in `example_text_completion.py`):

```python
def main(
    ckpt_dir: str,
    tokenizer_path: str,
    temperature: float = 0.6,
    top_p: float = 0.9,
    max_seq_len: int = 128,
    max_gen_len: int = 64,
    max_batch_size: int = 4,
):
```

**Launch** (single GPU, 7B base):

```bash
torchrun --nproc_per_node 1 example_text_completion.py \
    --ckpt_dir llama-2-7b/ \
    --tokenizer_path tokenizer.model \
    --max_seq_len 128 \
    --max_gen_len 64 \
    --max_batch_size 4
```

The script feeds four prompts through `generator.text_completion(...)` —
including a few-shot English-to-French translation prompt — and prints the
generation for each.

> [!NOTE]
> The `fire.Fire(main)` wrapper at the bottom of each example means every
> argument of `main` becomes a `--flag` on the command line. There are no
> hidden options — what you see in the function signature is the entire
> CLI for the example.

### 4.2 `example_chat_completion.py`

A six-dialog batch that exercises the chat templating logic, including
system-prompted dialogs, multi-turn dialogs, and the unsafe-tag guardrail.

**Signature** (from `main` in `example_chat_completion.py`):

```python
def main(
    ckpt_dir: str,
    tokenizer_path: str,
    temperature: float = 0.6,
    top_p: float = 0.9,
    max_seq_len: int = 512,
    max_batch_size: int = 8,
    max_gen_len: Optional[int] = None,
):
```

**Launch** (single GPU, 7B chat):

```bash
torchrun --nproc_per_node 1 example_chat_completion.py \
    --ckpt_dir llama-2-7b-chat/ \
    --tokenizer_path tokenizer.model \
    --max_seq_len 512 \
    --max_batch_size 8
```

The dialogs include a `[/INST]`-injection probe to demonstrate the
`UNSAFE_ERROR` path. When special tags (`[INST]`, `[/INST]`, `<<SYS>>`,
`<</SYS>>`) appear inside user content, the model still generates an
output, but `chat_completion` replaces the assistant content with the
literal string `"Error: special tags are not allowed as part of the
prompt."` before returning.

> [!WARNING]
> Chat completion enforces a strict role alternation: the dialog must
> start with `system` (optional) or `user`, must alternate `user` /
> `assistant` / `user` ..., and the **last** message must be from `user`.
> Violations raise `AssertionError` with the message `"Last message must
> be from user, got <role>"` or `"model only supports 'system', 'user'
> and 'assistant' roles..."`.

### 4.3 Multi-GPU launch (13B and 70B)

```bash
# 13B base — 2 GPUs
torchrun --nproc_per_node 2 example_text_completion.py \
    --ckpt_dir llama-2-13b/ \
    --tokenizer_path tokenizer.model \
    --max_seq_len 128 --max_batch_size 4

# 70B chat — 8 GPUs (typically a single A100/H100 8-way node)
torchrun --nproc_per_node 8 example_chat_completion.py \
    --ckpt_dir llama-2-70b-chat/ \
    --tokenizer_path tokenizer.model \
    --max_seq_len 512 --max_batch_size 6
```

> [!IMPORTANT]
> `--nproc_per_node` MUST equal the number of `consolidated.*.pth` files
> in the checkpoint directory. Mismatches surface as
> `AssertionError: Loading a checkpoint for MP=N but world size is M`
> (see GitHub issues [#806](https://github.com/meta-llama/llama/issues/806)
> and [#736](https://github.com/meta-llama/llama/issues/736)).

### 4.4 Embedding `Llama` in your own Python program

The example scripts are tiny wrappers — you can call the same API
directly:

```python
from llama import Llama

generator = Llama.build(
    ckpt_dir="llama-2-7b/",
    tokenizer_path="tokenizer.model",
    max_seq_len=512,
    max_batch_size=4,
)

results = generator.text_completion(
    ["The three laws of robotics are"],
    temperature=0.6,
    top_p=0.9,
    max_gen_len=64,
)
print(results[0]["generation"])
```

```text
The three laws of robotics are
> 1. A robot may not injure a human being or, through inaction, allow a human being to come to harm.
2. A robot must obey the orders given it by human beings ...
```

> [!NOTE]
> Even when you call `Llama.build` from your own script, it must be
> launched under `torchrun` (or an equivalent that sets `WORLD_SIZE`,
> `RANK`, `MASTER_ADDR`, `MASTER_PORT`). The first thing `build` does is
> `torch.distributed.init_process_group("nccl")`.

### 4.5 The `generate` primitive

Both `text_completion` and `chat_completion` are thin shells around
`Llama.generate`, which operates on token IDs:

```python
out_tokens, out_logprobs = generator.generate(
    prompt_tokens=[[1, 306, 4658, 278, 6593, 310, 2834, 338]],  # token IDs
    max_gen_len=32,
    temperature=0.6,
    top_p=0.9,
    logprobs=False,
    echo=False,
)
```

`generate` returns a tuple: `(out_tokens, out_logprobs_or_None)`. When
`logprobs=True`, the second element is per-token log probabilities; when
`logprobs=False`, it is `None`. Decode `out_tokens[i]` back to text with
`generator.tokenizer.decode(out_tokens[i])`.

---

## 5. Common Tasks

### 5.1 Download a specific model variant

`download.sh` prompts for two values: the presigned URL emailed by Meta,
and a comma-separated list of model names. The accepted names are:

```text
7B,13B,70B,7B-chat,13B-chat,70B-chat
```

Pressing Enter at the second prompt downloads all six. The script writes
each variant to a folder named after its `MODEL_PATH`:

| Input | Folder | Shards |
|-------|--------|------:|
| `7B` | `llama-2-7b/` | 1 (`consolidated.00.pth`) |
| `7B-chat` | `llama-2-7b-chat/` | 1 |
| `13B` | `llama-2-13b/` | 2 (`consolidated.00.pth`, `consolidated.01.pth`) |
| `13B-chat` | `llama-2-13b-chat/` | 2 |
| `70B` | `llama-2-70b/` | 8 (`consolidated.00.pth` ... `consolidated.07.pth`) |
| `70B-chat` | `llama-2-70b-chat/` | 8 |

The script also fetches `LICENSE`, `USE_POLICY.md`, `tokenizer.model`, and
`tokenizer_checklist.chk` into the working directory and verifies the
tokenizer checksum.

> [!CAUTION]
> The Meta presigned URL is **single-tenant and time-limited**. Treat the
> emailed URL like a secret — anyone with it can download the weights
> against your acceptance. Do not paste it into shared chat or commit
> it to a repository.

### 5.2 Run a single text completion

```bash
torchrun --nproc_per_node 1 example_text_completion.py \
    --ckpt_dir llama-2-7b/ \
    --tokenizer_path tokenizer.model
```

The four built-in prompts run unchanged. To customize, edit the `prompts`
list inside `example_text_completion.py` — there is no flag for prompt
text. For programmatic prompts, see Section 5.6.

### 5.3 Run a chat session with a system prompt

```bash
torchrun --nproc_per_node 1 example_chat_completion.py \
    --ckpt_dir llama-2-7b-chat/ \
    --tokenizer_path tokenizer.model
```

The script demonstrates four system-prompt patterns:

```python
# Haiku style
[{"role": "system", "content": "Always answer with Haiku"},
 {"role": "user",   "content": "I am going to Paris, what should I see?"}]

# Emoji style
[{"role": "system", "content": "Always answer with emojis"},
 {"role": "user",   "content": "How to go from Beijing to NY?"}]
```

The Llama 2 chat template prepends `<<SYS>>\n{system}\n<</SYS>>\n\n` to the
first user message and wraps each `(user, assistant)` pair as
`[INST] {user} [/INST] {assistant}`. The final user turn closes with
`[INST] {user} [/INST]`.

### 5.4 Build a custom prompt loop

```python
from llama import Llama, Dialog

generator = Llama.build(
    ckpt_dir="llama-2-7b-chat/",
    tokenizer_path="tokenizer.model",
    max_seq_len=2048,
    max_batch_size=1,
)

dialog: Dialog = [
    {"role": "system", "content": "You are a concise senior engineer."},
    {"role": "user",   "content": "Explain the role of RMSNorm in a transformer block."},
]
result = generator.chat_completion(
    [dialog],
    temperature=0.0,        # greedy
    top_p=0.9,
    max_gen_len=256,
)[0]
print(result["generation"]["content"])
```

> [!TIP]
> Set `temperature=0.0` for evaluation runs you want to be reproducible.
> The `Llama.build` `seed` parameter (default `1`) is also honored across
> all model-parallel ranks — pass the same value if you want bit-identical
> outputs across runs.

### 5.5 Run multi-GPU inference via `torchrun`

`torchrun` is the supported launcher. The two flags that matter are
`--nproc_per_node` (number of model-parallel ranks on this host) and,
for multi-host runs, `--nnodes` / `--node_rank` / `--rdzv_endpoint`.

```bash
# Single host, 8 GPUs (70B)
torchrun --nproc_per_node 8 example_chat_completion.py \
    --ckpt_dir llama-2-70b-chat/ \
    --tokenizer_path tokenizer.model

# Two hosts, 4 GPUs each (also 70B, MP=8)
# Host 0
torchrun --nproc_per_node 4 --nnodes 2 --node_rank 0 \
    --rdzv_endpoint <host0>:29500 \
    example_chat_completion.py \
    --ckpt_dir llama-2-70b-chat/ --tokenizer_path tokenizer.model
# Host 1
torchrun --nproc_per_node 4 --nnodes 2 --node_rank 1 \
    --rdzv_endpoint <host0>:29500 \
    example_chat_completion.py \
    --ckpt_dir llama-2-70b-chat/ --tokenizer_path tokenizer.model
```

### 5.6 Programmatic batch completion

```python
from llama import Llama

generator = Llama.build(
    ckpt_dir="llama-2-7b/",
    tokenizer_path="tokenizer.model",
    max_seq_len=256,
    max_batch_size=8,
)

prompts = [
    "Translate to French: I love the smell of fresh bread.",
    "Translate to French: The cat is on the mat.",
]
results = generator.text_completion(
    prompts, temperature=0.0, top_p=0.9, max_gen_len=64,
)
for prompt, result in zip(prompts, results):
    print(f"{prompt}\n  -> {result['generation']}")
```

> [!NOTE]
> The KV cache is sized to `(max_batch_size, max_seq_len, n_kv_heads,
> head_dim)` at `Llama.build` time. Pre-size for the largest batch you
> intend to use — calling with a batch larger than `max_batch_size` raises
> `AssertionError`.

### 5.7 Migrate to `llama-models` / `llama-stack`

For new work — Llama 3, Llama 3.1, Llama 3.2, Llama 3.3, code Llama,
Llama Guard, vision, and tool-using variants — the supported entry points
are:

- **[llama-models](https://github.com/meta-llama/llama-models)** —
  current model definitions, including a `tiktoken`-based tokenizer for
  the Llama 3 family. Replaces `llama/model.py` and `llama/tokenizer.py`
  in this repo.
- **[llama-stack](https://github.com/meta-llama/llama-stack)** —
  production inference, evaluation, agents, safety, and the
  `llama-stack-client` CLI. Replaces `example_*.py` and the
  `Llama.build` workflow.

The architecture in this repo maps cleanly to the Llama 3 reference
(same RMSNorm, same RoPE, same SwiGLU FFN, same grouped-query attention),
so reading this code remains a good prerequisite for understanding the
modern stack.

---

## 6. Troubleshooting

> [!WARNING]
> Most failures here happen at `Llama.build` time, before any tokens are
> generated. If the loaded-in-N-seconds line never prints, the model
> never finished loading — check NCCL, checkpoint shard count, and
> tokenizer paths in that order.

### Error: `AssertionError: Loading a checkpoint for MP=N but world size is M`

**Cause:** The number of `consolidated.*.pth` shards in `--ckpt_dir`
does not match the launcher's `WORLD_SIZE` (set by
`torchrun --nproc_per_node`). 7B has 1 shard, 13B has 2, 70B has 8.

**Solution:**

1. Count the shards: `ls llama-2-13b/consolidated.*.pth`.
2. Match `--nproc_per_node` to that count.
3. If you want to run a 13B checkpoint on a single GPU, reshard first;
   this code does not handle it for you.

**Prevention:** Stick to the model-parallel sizes documented in
Section 2.3. See GitHub issues
[#736](https://github.com/meta-llama/llama/issues/736) and
[#806](https://github.com/meta-llama/llama/issues/806).

### Error: `RuntimeError: probability tensor contains either 'inf', 'nan' or element < 0`

**Cause:** Numerical instability in the softmax-then-multinomial sampling
path inside `sample_top_p`. Most often triggered by `temperature` set very
close to but not exactly `0`, by half-precision overflow on long prompts,
or by a corrupted checkpoint.

**Solution:**

1. Try `temperature=0.0` (which switches to deterministic `argmax` and
   bypasses the multinomial draw entirely).
2. Reduce `max_gen_len` and `max_seq_len`.
3. Re-verify the checkpoint with `md5sum -c checklist.chk`.

**Prevention:** Avoid extremely small non-zero temperatures
(e.g. `1e-6`) — use `0.0` for greedy or `>= 0.1` for sampled. Tracked
in [#380](https://github.com/meta-llama/llama/issues/380).

### Error: `RuntimeError: unmatched '}' in format string`

**Cause:** A path-handling bug seen on Windows and some macOS shells
when `torchrun` interpolates rendezvous strings containing `{` / `}`.

**Solution:**

1. Use WSL2 (Windows Subsystem for Linux) instead of native Windows.
2. Run inside a clean Linux container.

**Prevention:** This codebase is intended for Linux + CUDA. Tracked in
[#347](https://github.com/meta-llama/llama/issues/347). Native Windows is
not supported.

### Error: `RuntimeError: Distributed package doesn't have NCCL built in`

**Cause:** PyTorch was installed without NCCL support. This affects
CPU-only PyTorch builds and some macOS installs.

**Solution:**

1. Reinstall PyTorch with the CUDA wheel:
   `pip install torch --index-url https://download.pytorch.org/whl/cu121`
   (or the CUDA major/minor version matching your driver).
2. Verify: `python -c "import torch; print(torch.distributed.is_nccl_available())"`.

**Prevention:** Always install the CUDA-enabled wheel on Linux + NVIDIA.
Tracked in [#112](https://github.com/meta-llama/llama/issues/112).

### Error: `torch.distributed.elastic.multiprocessing.api: failed (exitcode: -9)`

**Cause:** Exit code `-9` is `SIGKILL` from the Linux OOM killer — your
process exhausted host RAM (not VRAM) while loading the checkpoint. The
70B checkpoint shards alone are >130 GB on disk and need to be paged
through host memory at load time.

**Solution:**

1. Check `dmesg | tail` for the OOM record.
2. Add swap or run on a host with more RAM. For 70B, target 256 GB+ RAM.
3. Close other processes; this is a one-shot peak during `torch.load`.

**Prevention:** Plan for **host RAM ≥ checkpoint size on disk**, not just
VRAM. Tracked in [#928](https://github.com/meta-llama/llama/issues/928)
and [#960](https://github.com/meta-llama/llama/issues/960).

### Error: `no checkpoint files found in <ckpt_dir>`

**Cause:** `Path(ckpt_dir).glob("*.pth")` returned an empty list.
Either the directory is wrong, the download failed silently, or the
`.pth` files have a non-default name.

**Solution:**

1. Confirm the path: `ls <ckpt_dir>/consolidated.*.pth`.
2. Re-run `./download.sh` — the helper resumes via `wget --continue`.
3. Verify with `md5sum -c <ckpt_dir>/checklist.chk`.

**Prevention:** Always verify checksums after download.

### Error: `Last message must be from user, got assistant`

**Cause:** A `chat_completion` dialog ended with an assistant turn instead
of a user turn. The model is asked to **continue** an unfinished user
prompt — a fully-resolved dialog is not a valid input.

**Solution:** Append a final `{"role": "user", "content": "..."}` message
to your dialog before calling `chat_completion`.

**Prevention:** Treat dialogs like a chat input box: the user always
gets the last word before you press Send.

### Error: `model only supports 'system', 'user' and 'assistant' roles, starting with 'system', then 'user' and alternating (u/a/u/a/u...)`

**Cause:** Roles were not in the required `[system?, user, assistant,
user, assistant, ..., user]` pattern.

**Solution:** Reshape the dialog. Drop any non-`{system,user,assistant}`
roles, ensure `system` (if present) is first, and alternate user /
assistant strictly.

**Prevention:** Build dialogs with a helper that enforces the contract.

### Surprise output: `"Error: special tags are not allowed as part of the prompt."`

**Cause:** A user message contained one of the chat templating tokens:
`[INST]`, `[/INST]`, `<<SYS>>`, or `<</SYS>>`. This is by design —
`chat_completion` flags such inputs as `unsafe` and overwrites the
assistant content with `UNSAFE_ERROR`.

**Solution:** Sanitize user input. If you must accept arbitrary text
(e.g. quoting an instruction), escape or strip those tags before
passing the dialog to `chat_completion`.

**Prevention:** Filter `[INST]`, `[/INST]`, `<<SYS>>`, `<</SYS>>` from any
user-controlled string.

### Slow first run, fast subsequent runs

**Cause:** The first call to `Llama.build` reads each `consolidated.*.pth`
shard fully into host memory before mapping to GPU. On a cold filesystem
this is I/O bound (often 30–60 s for 13B, several minutes for 70B).
Subsequent runs benefit from page cache.

**Solution:** None — this is expected. For repeated runs in research
loops, keep one Python process alive and reuse the `generator` object
instead of relaunching `torchrun` per call.

**Prevention:** Build a long-lived inference server around `Llama.build`.

---

## 7. FAQ

**Q: Is this the right repository for Llama 3, 3.1, 3.2, or 3.3?**

No. This repo only loads Llama 2 era checkpoints (7B, 13B, 70B + their
`-chat` variants). For Llama 3 and later, use the
[`llama-models`](https://github.com/meta-llama/llama-models) repository
for model definitions and
[`llama-stack`](https://github.com/meta-llama/llama-stack) for inference,
evaluation, and agentic features.

**Q: Where do I get the model weights?**

You request them from
[ai.meta.com/resources/models-and-libraries/llama-downloads](https://ai.meta.com/resources/models-and-libraries/llama-downloads/),
accept the Llama 2 Community License, and Meta emails you a presigned
URL. You feed that URL to `./download.sh`.

**Q: Can I redistribute the weights?**

The Llama 2 Community License governs redistribution. In practice,
redistribute responsibly: do not bundle weights into a public mirror, do
not commit them to a public repo, and do not share the presigned URL.
Read `LICENSE` and `USE_POLICY.md` (downloaded by `download.sh`) before
redistributing in any form.

**Q: Can I run this on CPU only?**

No. `Llama.build` calls `torch.cuda.set_device(local_rank)` and
`torch.set_default_tensor_type(torch.cuda.HalfTensor)`, and the
`Attention` and `FeedForward` modules allocate caches with `.cuda()`.
A CPU-only PyTorch install will fail at build time. Use a CUDA-enabled
install on NVIDIA hardware.

**Q: Does it work on Apple Silicon (MPS)?**

Not as written. The build path is hardcoded to CUDA, and `fairscale`
model-parallel layers expect NCCL. Community forks have ported parts of
this code to MPS, but the upstream reference assumes NVIDIA GPUs.

**Q: Why must I launch with `torchrun`?**

`Llama.build` initializes a torch distributed process group and then
calls `fairscale.nn.model_parallel.initialize_model_parallel`.
Both require the environment variables (`WORLD_SIZE`, `RANK`,
`LOCAL_RANK`, `MASTER_ADDR`, `MASTER_PORT`) that `torchrun` sets up.
You can replicate it manually, but `torchrun` is the documented path.

**Q: What tokenizer does this use?**

SentencePiece, via `sentencepiece.SentencePieceProcessor`. The model
file is `tokenizer.model` (downloaded by `download.sh`). The `Tokenizer`
class is a thin wrapper exposing `encode(s, bos, eos)` and `decode(t)`.

> [!NOTE]
> The Llama 3 family switched to a `tiktoken`-based BPE tokenizer with
> a 128K vocabulary. Code in this repo will **not** load a Llama 3
> tokenizer. If you need that tokenizer, use `llama-models` or
> `llama-stack`.

**Q: How do I get deterministic / reproducible outputs?**

Use `temperature=0.0` (greedy) and pass the same `seed` to `Llama.build`
across all ranks. The `Llama.build` docstring is explicit: "seed must
be the same in all processes". Sampling with `temperature > 0` is
non-deterministic by design (it draws from `torch.multinomial`).

**Q: Can I fine-tune with this code?**

No. The model is decorated with `@torch.inference_mode()` at both the
`Llama.generate` and `Transformer.forward` level. There is no training
loop, no optimizer, no loss-aware data loader. For fine-tuning, use
`llama-recipes`, `llama-stack`, or the Hugging Face `transformers` port.

**Q: What are `[INST]`, `[/INST]`, `<<SYS>>`, `<</SYS>>`?**

The Llama 2 chat-template tokens. `[INST] {user} [/INST]` wraps a user
turn, and `<<SYS>>\n{system}\n<</SYS>>\n\n` prepends the system prompt
to the first user message. `chat_completion` builds these for you — you
should not include them in `content` strings yourself.

**Q: Can I increase the context window past `max_seq_len=2048`?**

The Llama 2 base models were trained with 4096-token context (see the
comment in `Transformer.__init__` that pre-computes `freqs_cis` for
`max_seq_len * 2`). You can pass `--max_seq_len 4096` at launch and the
RoPE table will be sized correctly. Pushing beyond that requires
positional-encoding extension techniques (NTK-aware RoPE, YaRN, etc.)
that this reference does NOT implement.

**Q: Where is the architecture documented in detail?**

See the **Architecture Overview** and **Developer Internals** sections
of the Administrator & Developer Guide.

---

_Generated April 2026 from meta-llama/llama at 689c7f2. AI-generated documentation. The library is checked by sampling (three repositories so far); this guide was not individually reviewed. Verify against the project before relying on it. Unofficial; not affiliated with or reviewed by the project._