Llama 2 Reference Implementation — User Guide

The meta-llama/llama repository is Meta's original reference implementation for the Llama 2 family of large language models (7B, 13B, 70B, and their -chat variants). It contains the model architecture in pure PyTorch, a SentencePiece tokenizer wrapper, model-parallel weight loading via fairscale, a download.sh script for fetching weights after license acceptance, and two example scripts (example_text_completion.py, example_chat_completion.py) that exercise the public Llama class.

Important

This repository is the Llama 2 reference release and is now in maintenance mode. Meta's active development has moved to the llama-models repository for model definitions and to llama-stack for the production inference toolchain. New projects targeting Llama 3, Llama 3.1, Llama 3.2, or Llama 3.3 should use those instead. The code in this repo only loads Llama 2 era checkpoints (7B / 13B / 70B + chat variants) and should be treated as a learning artifact and a faithful minimal implementation of the Llama 2 architecture.

1. Quick Start

The fastest path from a fresh clone to a running text-completion call is five steps: clone, install, request a download URL from Meta, run the download script, and launch one of the example scripts under torchrun.

# 1. Clone
git clone https://github.com/meta-llama/llama.git
cd llama

# 2. Install in editable mode (pulls torch, fairscale, fire, sentencepiece)
pip install -e .

# 3. Request a license-bound download URL from Meta:
#    https://ai.meta.com/resources/models-and-libraries/llama-downloads/
#    You will receive an email containing a presigned URL.

# 4. Run the download helper (it will prompt for the URL and the model size)
./download.sh

# 5. Run a one-line text completion against the downloaded 7B base model
torchrun --nproc_per_node 1 example_text_completion.py \
    --ckpt_dir llama-2-7b/ \
    --tokenizer_path tokenizer.model \
    --max_seq_len 128 --max_batch_size 4

Loaded in 12.42 seconds
I believe the meaning of life is
> to be happy, to find joy in the small things, and to live each day to the fullest.

==================================

Tip

The 7B base model is the lowest-friction starting point — it loads on a single GPU (model-parallel size 1) and finishes a 64-token completion in a few seconds on consumer-grade Ampere/Ada hardware. Save the chat variants and 13B / 70B for after you have the toolchain working.

What you get

• A faithful PyTorch implementation of the Llama 2 architecture: pre-norm, RMSNorm, rotary positional embeddings, grouped-query attention with KV caching, SwiGLU feed-forward.
• A model-parallel build path layered on top of fairscale so the same code runs on 1 GPU (7B), 2 GPUs (13B), or 8 GPUs (70B) by changing --nproc_per_node.
• A minimal public surface: one class (Llama), three methods (build, text_completion, chat_completion).
• Two ready-to-run examples that double as integration tests for your install.

Prerequisites checklist

• [x] Python 3.8 or newer (3.10 works well; tested by community on 3.11)
• [x] CUDA-capable NVIDIA GPU(s) with at least 16 GB VRAM for 7B
• [x] PyTorch with CUDA + NCCL support (the default CUDA wheels)
• [x] wget and md5sum (or md5 on macOS arm64) available in PATH
• [x] A presigned URL from Meta's license-acceptance flow

Note

The download script verifies every checkpoint and the tokenizer with MD5 checksums (md5sum -c checklist.chk on Linux / md5 checklist.chk on macOS arm64). A failed checksum means the download was truncated — rerun ./download.sh; wget --continue resumes partial transfers.

2. Installation

2.1 Minimum-friction install (recommended)

git clone https://github.com/meta-llama/llama.git
cd llama
pip install -e .

The setup.py declares the package name llama at version 0.0.1 and reads runtime requirements from requirements.txt:

torch
fairscale
fire
sentencepiece

Important

The requirements.txt deliberately does not pin versions. The repo was validated against torch >= 2.0 and fairscale >= 0.4.13. If your distributed launch hangs or NCCL fails to initialize, the most common root cause is a torch / fairscale combination that disagrees about the model-parallel API. A clean conda environment usually resolves it.

2.2 Conda environment (clean baseline)

conda create -n llama python=3.10 -y
conda activate llama
git clone https://github.com/meta-llama/llama.git
cd llama
pip install -e .

2.3 GPU / driver requirements

The model-parallel size is fixed by the number of consolidated.NN.pth shards that ship with each checkpoint: 1 shard for 7B, 2 for 13B, 8 for 70B. Llama.build asserts that model_parallel_size == len(checkpoints) — if you launch with the wrong --nproc_per_node, you will see the error documented in Troubleshooting below.

2.4 Verifying the install

python -c "from llama import Llama, Dialog; from llama.model import ModelArgs, Transformer; from llama.tokenizer import Tokenizer; print('ok')"

ok

Warning

Importing llama only verifies that the Python package is on the path — it does NOT initialize CUDA or load weights. The first real check is running an example script under torchrun. If pip install -e . succeeded but import torch; torch.cuda.is_available() returns False, fix your CUDA-enabled PyTorch install before downloading any weights.

3. Configuration

Llama's reference implementation is a research-grade library: there is no config file, no environment-variable layer above what torch.distributed already reads, and no plugin system. All configuration is runtime parameters passed at launch.

3.1 Distributed runtime variables

These are read from the environment by Llama.build and are normally populated for you by torchrun:

Note

torchrun --nproc_per_node N sets WORLD_SIZE=N and LOCAL_RANK per spawned process automatically. You should not have to set them by hand unless you are launching the example script outside torchrun.

3.2 Build parameters (Llama.build)

3.3 Sampling parameters (per call)

Tip

Setting temperature=0 switches the sampler from nucleus sampling (sample_top_p) to greedy argmax. Use this for deterministic-style evaluation runs where you want repeatable outputs.

For the full structural configuration of the model itself (ModelArgs.dim, n_layers, n_heads, multiple_of, ffn_dim_multiplier, norm_eps), see the Configuration Reference in the Administrator & Developer Guide.

4. Core Usage

The two example scripts are the canonical entry points and they show every moving part of the public surface.

4.1 example_text_completion.py

A four-prompt batch run that demonstrates the base (non-chat) text completion API.

Signature (from main in example_text_completion.py):

def main(
    ckpt_dir: str,
    tokenizer_path: str,
    temperature: float = 0.6,
    top_p: float = 0.9,
    max_seq_len: int = 128,
    max_gen_len: int = 64,
    max_batch_size: int = 4,
):

Launch (single GPU, 7B base):

torchrun --nproc_per_node 1 example_text_completion.py \
    --ckpt_dir llama-2-7b/ \
    --tokenizer_path tokenizer.model \
    --max_seq_len 128 \
    --max_gen_len 64 \
    --max_batch_size 4

The script feeds four prompts through generator.text_completion(...) — including a few-shot English-to-French translation prompt — and prints the generation for each.

Note

The fire.Fire(main) wrapper at the bottom of each example means every argument of main becomes a --flag on the command line. There are no hidden options — what you see in the function signature is the entire CLI for the example.

4.2 example_chat_completion.py

A six-dialog batch that exercises the chat templating logic, including system-prompted dialogs, multi-turn dialogs, and the unsafe-tag guardrail.

Signature (from main in example_chat_completion.py):

def main(
    ckpt_dir: str,
    tokenizer_path: str,
    temperature: float = 0.6,
    top_p: float = 0.9,
    max_seq_len: int = 512,
    max_batch_size: int = 8,
    max_gen_len: Optional[int] = None,
):

Launch (single GPU, 7B chat):

torchrun --nproc_per_node 1 example_chat_completion.py \
    --ckpt_dir llama-2-7b-chat/ \
    --tokenizer_path tokenizer.model \
    --max_seq_len 512 \
    --max_batch_size 8

The dialogs include a [/INST]-injection probe to demonstrate the UNSAFE_ERROR path. When special tags ([INST], [/INST], <<SYS>>, <</SYS>>) appear inside user content, the model still generates an output, but chat_completion replaces the assistant content with the literal string "Error: special tags are not allowed as part of the prompt." before returning.

Warning

Chat completion enforces a strict role alternation: the dialog must start with system (optional) or user, must alternate user / assistant / user ..., and the last message must be from user. Violations raise AssertionError with the message "Last message must be from user, got <role>" or "model only supports 'system', 'user' and 'assistant' roles...".

4.3 Multi-GPU launch (13B and 70B)

# 13B base — 2 GPUs
torchrun --nproc_per_node 2 example_text_completion.py \
    --ckpt_dir llama-2-13b/ \
    --tokenizer_path tokenizer.model \
    --max_seq_len 128 --max_batch_size 4

# 70B chat — 8 GPUs (typically a single A100/H100 8-way node)
torchrun --nproc_per_node 8 example_chat_completion.py \
    --ckpt_dir llama-2-70b-chat/ \
    --tokenizer_path tokenizer.model \
    --max_seq_len 512 --max_batch_size 6

Important

--nproc_per_node MUST equal the number of consolidated.*.pth files in the checkpoint directory. Mismatches surface as AssertionError: Loading a checkpoint for MP=N but world size is M (see GitHub issues #806 and #736).

4.4 Embedding Llama in your own Python program

The example scripts are tiny wrappers — you can call the same API directly:

from llama import Llama

generator = Llama.build(
    ckpt_dir="llama-2-7b/",
    tokenizer_path="tokenizer.model",
    max_seq_len=512,
    max_batch_size=4,
)

results = generator.text_completion(
    ["The three laws of robotics are"],
    temperature=0.6,
    top_p=0.9,
    max_gen_len=64,
)
print(results[0]["generation"])

The three laws of robotics are
> 1. A robot may not injure a human being or, through inaction, allow a human being to come to harm.
2. A robot must obey the orders given it by human beings ...

Note

Even when you call Llama.build from your own script, it must be launched under torchrun (or an equivalent that sets WORLD_SIZE, RANK, MASTER_ADDR, MASTER_PORT). The first thing build does is torch.distributed.init_process_group("nccl").

4.5 The generate primitive

Both text_completion and chat_completion are thin shells around Llama.generate, which operates on token IDs:

out_tokens, out_logprobs = generator.generate(
    prompt_tokens=[[1, 306, 4658, 278, 6593, 310, 2834, 338]],  # token IDs
    max_gen_len=32,
    temperature=0.6,
    top_p=0.9,
    logprobs=False,
    echo=False,
)

generate returns a tuple: (out_tokens, out_logprobs_or_None). When logprobs=True, the second element is per-token log probabilities; when logprobs=False, it is None. Decode out_tokens[i] back to text with generator.tokenizer.decode(out_tokens[i]).

5. Common Tasks

5.1 Download a specific model variant

download.sh prompts for two values: the presigned URL emailed by Meta, and a comma-separated list of model names. The accepted names are:

7B,13B,70B,7B-chat,13B-chat,70B-chat

Pressing Enter at the second prompt downloads all six. The script writes each variant to a folder named after its MODEL_PATH:

The script also fetches LICENSE, USE_POLICY.md, tokenizer.model, and tokenizer_checklist.chk into the working directory and verifies the tokenizer checksum.

Caution

The Meta presigned URL is single-tenant and time-limited. Treat the emailed URL like a secret — anyone with it can download the weights against your acceptance. Do not paste it into shared chat or commit it to a repository.

5.2 Run a single text completion

torchrun --nproc_per_node 1 example_text_completion.py \
    --ckpt_dir llama-2-7b/ \
    --tokenizer_path tokenizer.model

The four built-in prompts run unchanged. To customize, edit the prompts list inside example_text_completion.py — there is no flag for prompt text. For programmatic prompts, see Section 5.6.

5.3 Run a chat session with a system prompt

torchrun --nproc_per_node 1 example_chat_completion.py \
    --ckpt_dir llama-2-7b-chat/ \
    --tokenizer_path tokenizer.model

The script demonstrates four system-prompt patterns:

# Haiku style
[{"role": "system", "content": "Always answer with Haiku"},
 {"role": "user",   "content": "I am going to Paris, what should I see?"}]

# Emoji style
[{"role": "system", "content": "Always answer with emojis"},
 {"role": "user",   "content": "How to go from Beijing to NY?"}]

The Llama 2 chat template prepends <<SYS>>\n{system}\n<</SYS>>\n\n to the first user message and wraps each (user, assistant) pair as [INST] {user} [/INST] {assistant}. The final user turn closes with [INST] {user} [/INST].

5.4 Build a custom prompt loop

from llama import Llama, Dialog

generator = Llama.build(
    ckpt_dir="llama-2-7b-chat/",
    tokenizer_path="tokenizer.model",
    max_seq_len=2048,
    max_batch_size=1,
)

dialog: Dialog = [
    {"role": "system", "content": "You are a concise senior engineer."},
    {"role": "user",   "content": "Explain the role of RMSNorm in a transformer block."},
]
result = generator.chat_completion(
    [dialog],
    temperature=0.0,        # greedy
    top_p=0.9,
    max_gen_len=256,
)[0]
print(result["generation"]["content"])

Tip

Set temperature=0.0 for evaluation runs you want to be reproducible. The Llama.build seed parameter (default 1) is also honored across all model-parallel ranks — pass the same value if you want bit-identical outputs across runs.

5.5 Run multi-GPU inference via torchrun

torchrun is the supported launcher. The two flags that matter are --nproc_per_node (number of model-parallel ranks on this host) and, for multi-host runs, --nnodes / --node_rank / --rdzv_endpoint.

# Single host, 8 GPUs (70B)
torchrun --nproc_per_node 8 example_chat_completion.py \
    --ckpt_dir llama-2-70b-chat/ \
    --tokenizer_path tokenizer.model

# Two hosts, 4 GPUs each (also 70B, MP=8)
# Host 0
torchrun --nproc_per_node 4 --nnodes 2 --node_rank 0 \
    --rdzv_endpoint <host0>:29500 \
    example_chat_completion.py \
    --ckpt_dir llama-2-70b-chat/ --tokenizer_path tokenizer.model
# Host 1
torchrun --nproc_per_node 4 --nnodes 2 --node_rank 1 \
    --rdzv_endpoint <host0>:29500 \
    example_chat_completion.py \
    --ckpt_dir llama-2-70b-chat/ --tokenizer_path tokenizer.model

5.6 Programmatic batch completion

from llama import Llama

generator = Llama.build(
    ckpt_dir="llama-2-7b/",
    tokenizer_path="tokenizer.model",
    max_seq_len=256,
    max_batch_size=8,
)

prompts = [
    "Translate to French: I love the smell of fresh bread.",
    "Translate to French: The cat is on the mat.",
]
results = generator.text_completion(
    prompts, temperature=0.0, top_p=0.9, max_gen_len=64,
)
for prompt, result in zip(prompts, results):
    print(f"{prompt}\n  -> {result['generation']}")

Note

The KV cache is sized to (max_batch_size, max_seq_len, n_kv_heads, head_dim) at Llama.build time. Pre-size for the largest batch you intend to use — calling with a batch larger than max_batch_size raises AssertionError.

5.7 Migrate to llama-models / llama-stack

For new work — Llama 3, Llama 3.1, Llama 3.2, Llama 3.3, code Llama, Llama Guard, vision, and tool-using variants — the supported entry points are:

• llama-models — current model definitions, including a tiktoken-based tokenizer for the Llama 3 family. Replaces llama/model.py and llama/tokenizer.py in this repo.
• llama-stack — production inference, evaluation, agents, safety, and the llama-stack-client CLI. Replaces example_*.py and the Llama.build workflow.

The architecture in this repo maps cleanly to the Llama 3 reference (same RMSNorm, same RoPE, same SwiGLU FFN, same grouped-query attention), so reading this code remains a good prerequisite for understanding the modern stack.

6. Troubleshooting

Warning

Most failures here happen at Llama.build time, before any tokens are generated. If the loaded-in-N-seconds line never prints, the model never finished loading — check NCCL, checkpoint shard count, and tokenizer paths in that order.

Error: AssertionError: Loading a checkpoint for MP=N but world size is M

Cause: The number of consolidated.*.pth shards in --ckpt_dir does not match the launcher's WORLD_SIZE (set by torchrun --nproc_per_node). 7B has 1 shard, 13B has 2, 70B has 8.

Solution:

1. Count the shards: ls llama-2-13b/consolidated.*.pth.
2. Match --nproc_per_node to that count.
3. If you want to run a 13B checkpoint on a single GPU, reshard first; this code does not handle it for you.

Prevention: Stick to the model-parallel sizes documented in Section 2.3. See GitHub issues #736 and #806.

Error: RuntimeError: probability tensor contains either 'inf', 'nan' or element < 0

Cause: Numerical instability in the softmax-then-multinomial sampling path inside sample_top_p. Most often triggered by temperature set very close to but not exactly 0, by half-precision overflow on long prompts, or by a corrupted checkpoint.

Solution:

1. Try temperature=0.0 (which switches to deterministic argmax and bypasses the multinomial draw entirely).
2. Reduce max_gen_len and max_seq_len.
3. Re-verify the checkpoint with md5sum -c checklist.chk.

Prevention: Avoid extremely small non-zero temperatures (e.g. 1e-6) — use 0.0 for greedy or >= 0.1 for sampled. Tracked in #380.

Error: RuntimeError: unmatched '}' in format string

Cause: A path-handling bug seen on Windows and some macOS shells when torchrun interpolates rendezvous strings containing { / }.

Solution:

1. Use WSL2 (Windows Subsystem for Linux) instead of native Windows.
2. Run inside a clean Linux container.

Prevention: This codebase is intended for Linux + CUDA. Tracked in #347. Native Windows is not supported.

Error: RuntimeError: Distributed package doesn't have NCCL built in

Cause: PyTorch was installed without NCCL support. This affects CPU-only PyTorch builds and some macOS installs.

Solution:

1. Reinstall PyTorch with the CUDA wheel: pip install torch --index-url https://download.pytorch.org/whl/cu121 (or the CUDA major/minor version matching your driver).
2. Verify: python -c "import torch; print(torch.distributed.is_nccl_available())".

Prevention: Always install the CUDA-enabled wheel on Linux + NVIDIA. Tracked in #112.

Error: torch.distributed.elastic.multiprocessing.api: failed (exitcode: -9)

Cause: Exit code -9 is SIGKILL from the Linux OOM killer — your process exhausted host RAM (not VRAM) while loading the checkpoint. The 70B checkpoint shards alone are >130 GB on disk and need to be paged through host memory at load time.

Solution:

1. Check dmesg | tail for the OOM record.
2. Add swap or run on a host with more RAM. For 70B, target 256 GB+ RAM.
3. Close other processes; this is a one-shot peak during torch.load.

Prevention: Plan for host RAM ≥ checkpoint size on disk, not just VRAM. Tracked in #928 and #960.

Error: no checkpoint files found in <ckpt_dir>

Cause: Path(ckpt_dir).glob("*.pth") returned an empty list. Either the directory is wrong, the download failed silently, or the .pth files have a non-default name.

Solution:

1. Confirm the path: ls <ckpt_dir>/consolidated.*.pth.
2. Re-run ./download.sh — the helper resumes via wget --continue.
3. Verify with md5sum -c <ckpt_dir>/checklist.chk.

Prevention: Always verify checksums after download.

Error: Last message must be from user, got assistant

Cause: A chat_completion dialog ended with an assistant turn instead of a user turn. The model is asked to continue an unfinished user prompt — a fully-resolved dialog is not a valid input.

Solution: Append a final {"role": "user", "content": "..."} message to your dialog before calling chat_completion.

Prevention: Treat dialogs like a chat input box: the user always gets the last word before you press Send.

Error: model only supports 'system', 'user' and 'assistant' roles, starting with 'system', then 'user' and alternating (u/a/u/a/u...)

Cause: Roles were not in the required [system?, user, assistant, user, assistant, ..., user] pattern.

Solution: Reshape the dialog. Drop any non-{system,user,assistant} roles, ensure system (if present) is first, and alternate user / assistant strictly.

Prevention: Build dialogs with a helper that enforces the contract.

Surprise output: "Error: special tags are not allowed as part of the prompt."

Cause: A user message contained one of the chat templating tokens: [INST], [/INST], <<SYS>>, or <</SYS>>. This is by design — chat_completion flags such inputs as unsafe and overwrites the assistant content with UNSAFE_ERROR.

Solution: Sanitize user input. If you must accept arbitrary text (e.g. quoting an instruction), escape or strip those tags before passing the dialog to chat_completion.

Prevention: Filter [INST], [/INST], <<SYS>>, <</SYS>> from any user-controlled string.

Slow first run, fast subsequent runs

Cause: The first call to Llama.build reads each consolidated.*.pth shard fully into host memory before mapping to GPU. On a cold filesystem this is I/O bound (often 30–60 s for 13B, several minutes for 70B). Subsequent runs benefit from page cache.

Solution: None — this is expected. For repeated runs in research loops, keep one Python process alive and reuse the generator object instead of relaunching torchrun per call.

Prevention: Build a long-lived inference server around Llama.build.

7. FAQ

Q: Is this the right repository for Llama 3, 3.1, 3.2, or 3.3?

No. This repo only loads Llama 2 era checkpoints (7B, 13B, 70B + their -chat variants). For Llama 3 and later, use the llama-models repository for model definitions and llama-stack for inference, evaluation, and agentic features.

Q: Where do I get the model weights?

You request them from ai.meta.com/resources/models-and-libraries/llama-downloads, accept the Llama 2 Community License, and Meta emails you a presigned URL. You feed that URL to ./download.sh.

Q: Can I redistribute the weights?

The Llama 2 Community License governs redistribution. In practice, redistribute responsibly: do not bundle weights into a public mirror, do not commit them to a public repo, and do not share the presigned URL. Read LICENSE and USE_POLICY.md (downloaded by download.sh) before redistributing in any form.

Q: Can I run this on CPU only?

No. Llama.build calls torch.cuda.set_device(local_rank) and torch.set_default_tensor_type(torch.cuda.HalfTensor), and the Attention and FeedForward modules allocate caches with .cuda(). A CPU-only PyTorch install will fail at build time. Use a CUDA-enabled install on NVIDIA hardware.

Q: Does it work on Apple Silicon (MPS)?

Not as written. The build path is hardcoded to CUDA, and fairscale model-parallel layers expect NCCL. Community forks have ported parts of this code to MPS, but the upstream reference assumes NVIDIA GPUs.

Q: Why must I launch with torchrun?

Llama.build initializes a torch distributed process group and then calls fairscale.nn.model_parallel.initialize_model_parallel. Both require the environment variables (WORLD_SIZE, RANK, LOCAL_RANK, MASTER_ADDR, MASTER_PORT) that torchrun sets up. You can replicate it manually, but torchrun is the documented path.

Q: What tokenizer does this use?

SentencePiece, via sentencepiece.SentencePieceProcessor. The model file is tokenizer.model (downloaded by download.sh). The Tokenizer class is a thin wrapper exposing encode(s, bos, eos) and decode(t).

Note

The Llama 3 family switched to a tiktoken-based BPE tokenizer with a 128K vocabulary. Code in this repo will not load a Llama 3 tokenizer. If you need that tokenizer, use llama-models or llama-stack.

Q: How do I get deterministic / reproducible outputs?

Use temperature=0.0 (greedy) and pass the same seed to Llama.build across all ranks. The Llama.build docstring is explicit: "seed must be the same in all processes". Sampling with temperature > 0 is non-deterministic by design (it draws from torch.multinomial).

Q: Can I fine-tune with this code?

No. The model is decorated with @torch.inference_mode() at both the Llama.generate and Transformer.forward level. There is no training loop, no optimizer, no loss-aware data loader. For fine-tuning, use llama-recipes, llama-stack, or the Hugging Face transformers port.

Q: What are [INST], [/INST], <<SYS>>, <</SYS>>?

The Llama 2 chat-template tokens. [INST] {user} [/INST] wraps a user turn, and <<SYS>>\n{system}\n<</SYS>>\n\n prepends the system prompt to the first user message. chat_completion builds these for you — you should not include them in content strings yourself.

Q: Can I increase the context window past max_seq_len=2048?

The Llama 2 base models were trained with 4096-token context (see the comment in Transformer.__init__ that pre-computes freqs_cis for max_seq_len * 2). You can pass --max_seq_len 4096 at launch and the RoPE table will be sized correctly. Pushing beyond that requires positional-encoding extension techniques (NTK-aware RoPE, YaRN, etc.) that this reference does NOT implement.

Q: Where is the architecture documented in detail?

See the Architecture Overview and Developer Internals sections of the Administrator & Developer Guide.