# DeepSeek-V3 User Guide

The `deepseek-ai/DeepSeek-V3` repository is the **reference inference implementation** for DeepSeek's V3-class large language models. It is a deliberately small Python codebase — a single-file Transformer (`model.py`), a generation loop (`generate.py`), a Hugging Face checkpoint converter (`convert.py`), an FP8-to-BF16 weight caster (`fp8_cast_bf16.py`), and a Triton kernel module (`kernel.py`) — together with four JSON model configurations covering the 16B, 236B, 671B, and V3.1 variants.

This guide documents only what is in the repository at commit `9b4e978`. It does not assume capabilities published in papers or blog posts unless those capabilities are present in the code.

## 1. Quick Start

DeepSeek-V3's inference repo is a **research-grade reference**. It loads a converted checkpoint, runs a multi-rank distributed forward pass with tensor parallelism, and decodes tokens through a small interactive or batch loop.

### 1.1 What's in the box

- **`inference/model.py`** — the Transformer: `ModelArgs`, `Transformer`, `Block`, `MLA` (Multi-Head Latent Attention), `MoE` (Mixture-of-Experts), `Gate`, `MLP`, `Expert`, parallel embedding, parallel/RMS-norm primitives.
- **`inference/generate.py`** — the run script. Loads the model, builds an `AutoTokenizer` from the checkpoint directory, supports `--interactive` and `--input-file` modes.
- **`inference/convert.py`** — converts a Hugging Face safetensors checkpoint into the repo's sharded layout (`model{rank}-mp{mp}.safetensors`).
- **`inference/fp8_cast_bf16.py`** — dequantizes an FP8 checkpoint into BF16 safetensors for hardware that lacks FP8 support.
- **`inference/kernel.py`** — Triton kernels for `act_quant`, `weight_dequant`, and `fp8_gemm` (block-scaled FP8 GEMM, autotuned).
- **`inference/configs/*.json`** — four model configs: `config_16B.json`, `config_236B.json`, `config_671B.json`, `config_v3.1.json`.

### 1.2 Prerequisites (single line)

CUDA-capable NVIDIA GPU(s), Python 3.10+, PyTorch 2.4.1, Triton 3.0.0, Transformers 4.46.3, Safetensors 0.4.5, plus the converted DeepSeek weights.

### 1.3 Minimal install + minimal run

```bash
git clone https://github.com/deepseek-ai/DeepSeek-V3.git
cd DeepSeek-V3
pip install -r inference/requirements.txt
```

```bash
# Single-GPU smoke test against an already-converted checkpoint
cd inference
torchrun --nnodes 1 --nproc-per-node 1 generate.py \
    --ckpt-path /path/to/converted-weights \
    --config configs/config_16B.json \
    --interactive
```

You will land at a `>>> ` prompt. Type a message and press Enter; the model decodes a completion of up to `--max-new-tokens` (default `200`) at `--temperature 0.2`.

> [!TIP]
> Use `/clear` at the prompt to reset the chat history (the script keeps a running `messages` list for multi-turn chat). Use `/exit` to quit cleanly.

### 1.4 What to try next

- **Convert a Hugging Face checkpoint** before running anything — see Installation §2.4.
- **Cast FP8 weights to BF16** if your GPUs do not support FP8 — see §5.
- **Run the same prompt across multiple GPUs** with tensor parallelism — see §5.

## 2. Installation

### 2.1 System requirements

| Requirement | Value | Notes |
|---|---|---|
| GPU | NVIDIA, CUDA-capable | The generation loop hard-codes `device="cuda"`; CPU-only is not supported. |
| Python | 3.10+ recommended | PyTorch 2.4.1 + Triton 3.0.0 require a recent Python. |
| Disk | tens-of-GB to ~700 GB | Depends on which model you load (16B → 671B). FP8 671B weights are ~700 GB; BF16 cast roughly doubles that. |
| RAM | 32 GB+ | The converter caches up to 2 safetensors files in CUDA memory at a time during the FP8→BF16 cast. |
| Network | for tokenizer + weights | `AutoTokenizer.from_pretrained(ckpt_path)` reads tokenizer files from the checkpoint directory. |

### 2.2 Pinned dependencies

The repository pins exact versions in `inference/requirements.txt`:

```text
torch==2.4.1
triton==3.0.0
transformers==4.46.3
safetensors==0.4.5
```

> [!IMPORTANT]
> These versions are pinned for a reason: the FP8 GEMM kernel uses `torch.float8_e4m3fn` and Triton 3.0.0's autotune API. Newer Triton versions have changed kernel-launch internals, and newer PyTorch versions can shift FP8 dtype behavior. If you upgrade, expect to debug.

### 2.3 Install

```bash
pip install -r inference/requirements.txt
```

There is no `setup.py`, no `pyproject.toml`, and no console-script entry point. You execute the scripts directly with `torchrun` (or, for single-GPU, with `python` if you set the env vars manually).

### 2.4 Convert a Hugging Face checkpoint

DeepSeek publishes its weights in Hugging Face safetensors layout. The repo expects a sharded layout keyed by model-parallel rank. Convert once:

```bash
cd inference
python convert.py \
    --hf-ckpt-path /path/to/hf-checkpoint \
    --save-path /path/to/converted-output \
    --n-experts 256 \
    --model-parallel 8
```

The converter:

1. Walks every `*.safetensors` in `--hf-ckpt-path`.
2. Renames Hugging Face parameter names (`self_attn` → `attn`, `mlp` → `ffn`, `embed_tokens` → `embed`, etc.) using the `mapping` dict in `convert.py`.
3. Skips `model.layers.61` (the multi-token-prediction head, not used by this inference loop).
4. Shards each parameter across `--model-parallel` ranks: experts are partitioned by index, other weights are sharded along their tensor-parallel dimension (`dim=0` for column-parallel weights, `dim=1` for row-parallel `o_proj`/`down_proj`).
5. Writes one `model{i}-mp{mp}.safetensors` per rank into `--save-path`, plus copies any `*token*` files (tokenizer, tokenizer config) verbatim.

> [!IMPORTANT]
> `--n-experts` MUST be divisible by `--model-parallel`. The script asserts this and aborts otherwise. For the 671B / V3.1 configs (`n_routed_experts: 256`), valid `--model-parallel` values are 1, 2, 4, 8, 16, 32, 64, 128, 256.

### 2.5 Optional: cast FP8 weights to BF16

If your hardware does not support FP8, dequantize the checkpoint:

```bash
cd inference
python fp8_cast_bf16.py \
    --input-fp8-hf-path /path/to/fp8-checkpoint \
    --output-bf16-hf-path /path/to/bf16-checkpoint
```

The script:

- Reads `model.safetensors.index.json` from the input directory.
- Walks each safetensors shard, calls `weight_dequant` (Triton kernel) for any 1-byte-element weight, looking up the block-wise `*_scale_inv` tensor.
- Caches at most 2 source shards in CUDA memory at a time.
- Writes the resulting BF16 shards to `--output-bf16-hf-path` and emits a new `model.safetensors.index.json` with `_scale_inv` entries removed.

> [!WARNING]
> If a `_scale_inv` tensor is missing for an FP8 weight, the script prints `Warning: Missing scale_inv tensor for {name}, skipping conversion` and copies the FP8 weight through unchanged. The resulting checkpoint will then mix FP8 and BF16 tensors, which the model loader will not handle correctly. Verify your input checkpoint is complete before casting.

### 2.6 Verify the install

After installing dependencies, the fastest sanity check is to import the model module and instantiate the default `ModelArgs` (the 16B-shaped defaults — see `inference/model.py`):

```bash
cd inference
python -c "from model import ModelArgs, Transformer; import torch; \
torch.set_default_dtype(torch.bfloat16); torch.set_default_device('cuda'); \
args = ModelArgs(); m = Transformer(args); print('ok', sum(p.numel() for p in m.parameters()))"
```

This builds the model on GPU with random weights, useful only to confirm the imports and CUDA setup work. To actually generate text you need converted weights and `generate.py`.

## 3. Configuration

DeepSeek-V3 reads two kinds of configuration: the **model config JSON** (architecture) and **environment variables** for distributed launch.

### 3.1 The model config JSON

`generate.py` does:

```python
with open(config) as f:
    args = ModelArgs(**json.load(f))
```

Every key in the JSON becomes a field of `ModelArgs`. Keys not present in the JSON take their `ModelArgs` default. Four configs ship with the repo:

| Config file | dim | n_layers | n_heads | n_routed_experts | n_activated_experts | dtype | Notes |
|---|---:|---:|---:|---:|---:|---|---|
| `config_16B.json` | 2048 | 27 | 16 | 64 | 6 | (default `bf16`) | DeepSeek-V2-Lite shape; `mscale: 0.707`. |
| `config_236B.json` | 5120 | 60 | 128 | 160 | 6 | (default `bf16`) | DeepSeek-V2 shape; expert grouping (`n_expert_groups: 8`, `n_limited_groups: 3`). |
| `config_671B.json` | 7168 | 61 | 128 | 256 | 8 | `fp8` | DeepSeek-V3; `score_func: sigmoid`, `route_scale: 2.5`. |
| `config_v3.1.json` | 7168 | 61 | 128 | 256 | 8 | `fp8` | DeepSeek-V3.1; same as 671B plus `scale_fmt: ue8m0`. |

> [!NOTE]
> The 671B and V3.1 configs ship with `dtype: fp8` and bake in expert grouping. The V3.1 config additionally sets `scale_fmt: ue8m0`, which switches the activation-quantization kernel to UE8M0-formatted scales (a power-of-two scale). All four configs share the same `qk_nope_head_dim: 128`, `qk_rope_head_dim: 64`, `v_head_dim: 128`, and `kv_lora_rank: 512`.

### 3.2 Environment variables

`generate.py` reads three environment variables and passes them straight to PyTorch's distributed runtime:

| Variable | Read by | Default | Meaning |
|---|---|---|---|
| `WORLD_SIZE` | `os.getenv("WORLD_SIZE", "1")` | `1` | Total number of processes (= total GPUs across all nodes). |
| `RANK` | `os.getenv("RANK", "0")` | `0` | Global rank of this process. |
| `LOCAL_RANK` | `os.getenv("LOCAL_RANK", "0")` | `0` | Rank within the current node — used for `torch.cuda.set_device(local_rank)`. |

`torchrun` populates all three. You normally do not set them yourself.

### 3.3 CLI flags for `generate.py`

| Flag | Type | Default | Description |
|---|---|---|---|
| `--ckpt-path` | str | required | Directory containing the converted `model{rank}-mp{world_size}.safetensors` shards and tokenizer files. |
| `--config` | str | required | Path to one of `configs/config_*.json`. |
| `--input-file` | str | `""` | One prompt per line; runs in batch mode and prints completions. |
| `--interactive` | flag | `False` | Drops into a `>>> ` REPL. **At least one of `--input-file` / `--interactive` must be supplied** (asserted at startup). |
| `--max-new-tokens` | int | `200` | Maximum tokens to generate per prompt. |
| `--temperature` | float | `0.2` | Sampling temperature. `0` or negative falls back to greedy `argmax`. |

> [!TIP]
> The `sample()` function divides by `max(temperature, 1e-5)` and uses Gumbel-style argmax (`probs.div_(empty.exponential_(1)).argmax(-1)`) — equivalent to multinomial sampling but branchless. Top-k / top-p / repetition-penalty are not implemented in this repo.

For the full configuration reference (every `ModelArgs` field with type, default, and effect), see the Administrator & Developer Guide.

## 4. Core Usage

### 4.1 The generation loop, end to end

`generate.py` does the following on every rank:

1. Reads `WORLD_SIZE`, `RANK`, `LOCAL_RANK` from the environment.
2. If `WORLD_SIZE > 1`, calls `dist.init_process_group("nccl")`.
3. Silences `print` on every rank except 0.
4. `torch.cuda.set_device(local_rank)`, sets default dtype to `torch.bfloat16`, sets `torch.set_num_threads(8)`, seeds with `torch.manual_seed(965)`.
5. Loads the JSON config into `ModelArgs`.
6. Builds the `Transformer` on the current CUDA device.
7. `AutoTokenizer.from_pretrained(ckpt_path)` — the tokenizer is read directly from the checkpoint directory.
8. Runs a 2-token warm-up generation against the literal prompt `"DeepSeek"` (this triggers Triton autotuning and KV-cache allocation paths before the user sees a prompt).
9. `load_model(model, os.path.join(ckpt_path, f"model{rank}-mp{world_size}.safetensors"))` — each rank loads only its own shard.
10. Enters interactive or batch mode.

> [!NOTE]
> Step 9 happens **after** model construction, which means the model is built once with empty parameters and then weights are streamed into place via `safetensors.torch.load_model`. This is why `--ckpt-path` and `--config` must agree on the architecture.

### 4.2 Interactive chat

When `--interactive` is set:

```text
>>> Hello, who are you?
I am DeepSeek...
>>> /clear
>>> What's the capital of France?
Paris.
>>> /exit
```

Behavior:

- Rank 0 reads `input(">>> ")`. With multiple ranks, the prompt is broadcast to all ranks via `dist.broadcast_object_list`.
- The script maintains a `messages` list of `{"role": "user" | "assistant", "content": str}` dicts.
- Each turn, `tokenizer.apply_chat_template(messages, add_generation_prompt=True)` produces the input token IDs.
- `generate(...)` returns completion tokens; `tokenizer.decode(..., skip_special_tokens=True)` yields the printed reply.
- The reply is appended to `messages` so the next turn has full chat context.

> [!TIP]
> `/clear` resets `messages` to `[]` mid-session — useful when you want to start fresh without restarting the process and reloading hundreds of GB of weights.

### 4.3 Batch mode

When `--input-file path/to/prompts.txt` is set:

```bash
torchrun --nnodes 1 --nproc-per-node 8 generate.py \
    --ckpt-path /weights/v3-mp8 \
    --config configs/config_671B.json \
    --input-file prompts.txt \
    --max-new-tokens 512 \
    --temperature 0.7
```

Each line of the file is treated as a single user message and rendered through `apply_chat_template`. The script asserts:

```python
assert len(prompts) <= args.max_batch_size
```

`max_batch_size` defaults to `8` (see `ModelArgs`), so you cannot batch more than 8 prompts in a single invocation unless you raise `max_batch_size` in the JSON config.

> [!WARNING]
> `max_batch_size` does not just gate the assertion — it also sizes the KV cache buffers (`k_cache`, `v_cache`, `kv_cache`, `pe_cache`) that are allocated inside MLA at construction time. Raising it raises GPU memory consumption proportionally.

### 4.4 The decode loop in detail

`generate(model, prompt_tokens, max_new_tokens, eos_id, temperature)`:

1. Builds a `(batch, total_len)` token tensor pre-filled with `-1`, copies in each prompt.
2. Tracks a `prompt_mask = tokens != -1` so prompt positions are never overwritten.
3. Iterates `cur_pos` from `min(prompt_lens)` to `total_len`:
   - `logits = model.forward(tokens[:, prev_pos:cur_pos], prev_pos)` — forwards only the new slice (the rest is in KV cache).
   - If `temperature > 0`, samples; else takes `argmax`.
   - Where `prompt_mask[:, cur_pos]` is True, keeps the prompt token; otherwise writes the sampled token.
   - Marks rows finished if their newly emitted token equals `eos_id` and the position is past the prompt.
   - Breaks early once **all** rows have finished.
4. Truncates each completion at the first `eos_id` (if any).

> [!NOTE]
> The prompt-overwrite trick (filling with `-1`, masking with `prompt_mask`) lets the loop run the prefill phase as a single forward, even when prompts have different lengths within a batch.

## 5. Common Tasks

The recipes below are everything `generate.py`, `convert.py`, and `fp8_cast_bf16.py` actually support out of the box. There are no other CLI surfaces in the repo.

### 5.1 Recipe: load a model and run one completion (single GPU)

**Goal:** Quick prompt → completion, no chat state.

**Prerequisites:** Converted checkpoint at `/weights/v3-mp1` (i.e. `--model-parallel 1` was used during `convert.py`), tokenizer files present in the same directory.

**Steps:**

```bash
cd DeepSeek-V3/inference
echo "Write a haiku about transformers." > prompts.txt
torchrun --nnodes 1 --nproc-per-node 1 generate.py \
    --ckpt-path /weights/v3-mp1 \
    --config configs/config_16B.json \
    --input-file prompts.txt \
    --max-new-tokens 64 \
    --temperature 0.7
```

**Verify:** Output prints `Prompt:` then `Completion:` for each line.

**If it fails:** See Troubleshooting §6 — particularly CUDA OOM, the `max_seq_len` assertion, and the missing-`_scale_inv` warning.

### 5.2 Recipe: interactive multi-turn chat

**Goal:** Sit at the `>>> ` prompt and converse.

```bash
torchrun --nnodes 1 --nproc-per-node 1 generate.py \
    --ckpt-path /weights/v3-mp1 \
    --config configs/config_16B.json \
    --interactive
```

**Tips:**

- `/clear` to wipe context within the same process.
- `/exit` to quit.
- The chat template is whatever `tokenizer.apply_chat_template` was trained with — controlled by the tokenizer files inside `--ckpt-path`, not by this repo.

### 5.3 Recipe: multi-GPU tensor-parallel inference

**Goal:** Run the 671B / V3.1 model split across 8 GPUs on one node.

**Prerequisites:** A checkpoint converted with `--model-parallel 8` (so files `model0-mp8.safetensors` through `model7-mp8.safetensors` exist).

```bash
cd DeepSeek-V3/inference
torchrun --nnodes 1 --nproc-per-node 8 generate.py \
    --ckpt-path /weights/v3-mp8 \
    --config configs/config_671B.json \
    --interactive
```

`torchrun` sets `WORLD_SIZE=8`, assigns each process a `RANK` 0–7 and `LOCAL_RANK` matching its GPU. `dist.init_process_group("nccl")` brings up the NCCL communicator.

> [!NOTE]
> The model uses **tensor parallelism only**: column-parallel splits on `wq`, `wq_b`, `wkv_b`, `w1`, `w3`, and the LM `head`; row-parallel on `wo` and `w2`. There is no pipeline parallelism in this repo. Increasing `--nproc-per-node` strictly increases tensor-parallel degree.

### 5.4 Recipe: multi-node tensor-parallel inference

For multi-node launches, follow the standard `torchrun` recipe:

```bash
# On node 0 (master)
torchrun --nnodes 2 --nproc-per-node 8 \
    --node-rank 0 --master-addr <master-ip> --master-port 29500 \
    generate.py --ckpt-path /weights/v3-mp16 \
    --config configs/config_v3.1.json --interactive

# On node 1
torchrun --nnodes 2 --nproc-per-node 8 \
    --node-rank 1 --master-addr <master-ip> --master-port 29500 \
    generate.py --ckpt-path /weights/v3-mp16 \
    --config configs/config_v3.1.json --interactive
```

The checkpoint must have been converted with `--model-parallel 16` to match the 16-GPU world.

### 5.5 Recipe: cast an FP8 checkpoint to BF16

If your GPUs don't support FP8 (anything pre-Hopper, or any non-NVIDIA accelerator):

```bash
cd DeepSeek-V3/inference
python fp8_cast_bf16.py \
    --input-fp8-hf-path /weights/deepseek-v3-fp8 \
    --output-bf16-hf-path /weights/deepseek-v3-bf16
```

Then convert the BF16 directory with `convert.py` and run `generate.py` against `config_671B.json` after editing `dtype` away from `"fp8"` (or use a config without `dtype: fp8` — `config_16B.json` and `config_236B.json` ship with implicit `bf16`).

> [!CAUTION]
> The cast roughly doubles disk usage. The 671B model in FP8 is around 700 GB; the BF16 copy will be roughly 1.4 TB. Make sure the destination volume has space before kicking off — there is no resume-from-partial logic.

### 5.6 Recipe: full pipeline from Hugging Face download

End-to-end, from raw HF weights to a chat REPL on 8 GPUs:

```bash
# 1. Download the HF release (use huggingface-cli or git lfs; not part of this repo).
huggingface-cli download deepseek-ai/DeepSeek-V3 --local-dir /raw/deepseek-v3-hf

# 2. (Optional) Cast FP8 -> BF16 if your GPUs don't support FP8.
python fp8_cast_bf16.py \
    --input-fp8-hf-path /raw/deepseek-v3-hf \
    --output-bf16-hf-path /raw/deepseek-v3-bf16

# 3. Shard for tensor parallelism.
python convert.py \
    --hf-ckpt-path /raw/deepseek-v3-hf \
    --save-path /weights/v3-mp8 \
    --n-experts 256 \
    --model-parallel 8

# 4. Launch.
torchrun --nnodes 1 --nproc-per-node 8 generate.py \
    --ckpt-path /weights/v3-mp8 \
    --config configs/config_671B.json \
    --interactive
```

> [!TIP]
> Step 1 (the download) is not part of this repo. The Hugging Face CLI, `git lfs`, or `hf-transfer` are the usual choices.

### 5.7 Recipe: change the sampling temperature mid-experiment

Temperature is a CLI flag, not a runtime command. Restart the process with a new `--temperature`:

```bash
torchrun --nnodes 1 --nproc-per-node 1 generate.py \
    --ckpt-path /weights/v3-mp1 --config configs/config_16B.json \
    --interactive --temperature 0.0   # greedy
```

A non-positive value triggers the `else` branch in `generate()` and uses `logits.argmax(dim=-1)` directly.

### 5.8 Recipe: extend the maximum sequence length

`max_seq_len` defaults to `4096 * 4 = 16384` in `ModelArgs`. To change it without editing source, override in the config JSON:

```json
{
  "max_seq_len": 32768,
  "vocab_size": 102400,
  "dim": 2048,
  "...": "..."
}
```

The model uses YaRN-style RoPE scaling: when `max_seq_len > original_seq_len` (default `4096`), `precompute_freqs_cis` interpolates between scaled and unscaled frequencies using `rope_factor` (default `40`), `beta_fast` (`32`), and `beta_slow` (`1`). MLA also rescales `softmax_scale` by `(0.1 * mscale * log(rope_factor) + 1)^2` for extended contexts.

> [!WARNING]
> Larger `max_seq_len` enlarges KV-cache buffers in every attention layer. With `attn_impl = "absorb"` (the default), the cache is `(max_batch_size, max_seq_len, kv_lora_rank)` for `kv_cache` plus `(max_batch_size, max_seq_len, qk_rope_head_dim)` for `pe_cache` — manageable. With `attn_impl = "naive"` the caches scale with `n_local_heads * qk_head_dim` and `n_local_heads * v_head_dim`, much larger.

## 6. Troubleshooting

The repository surfaces only a handful of explicit error messages, mostly via Python `assert`. Below are the ones you'll actually hit.

### Error: `Number of experts must be divisible by model parallelism`

**Cause:** Raised by `convert.py` when `--n-experts % --model-parallel != 0`.

**Solution:**

1. Confirm `--n-experts` matches the config: `64` for 16B, `160` for 236B, `256` for 671B and V3.1.
2. Use a `--model-parallel` that divides it evenly. For 256 experts: 1, 2, 4, 8, 16, 32, 64, 128, 256.

**Prevention:** Pick `--model-parallel` to equal your target `--nproc-per-node × --nnodes` and verify divisibility before launching.

### Error: `Vocabulary size must be divisible by world size (world_size=N)`

**Cause:** Raised by `ParallelEmbedding.__init__` (and the column-parallel head). The shipping configs use `vocab_size: 102400` (16B/236B) or `129280` (671B/V3.1).

**Solution:** Pick a `world_size` that divides the vocab. `129280 = 2^7 × 5 × 202`, so 1, 2, 4, 5, 8, 10, 16, 20, 32, 40, 64, 80, 128, 160 all divide cleanly. `102400 = 2^12 × 25`, similarly flexible.

**Prevention:** Stick to powers of two for `--nproc-per-node`; both shipping vocabs cooperate.

### Error: `Output features must be divisible by world size` / `Input features must be divisible by world size`

**Cause:** `ColumnParallelLinear` / `RowParallelLinear` assertions when a Linear's output (or input) features aren't divisible by `world_size`. Most shipping configs use `n_heads` and `dim` values designed to divide cleanly across 1/2/4/8/16 GPUs.

**Solution:** Use a `world_size` that divides the relevant feature counts. For `config_671B.json`: `dim=7168` and `n_heads=128` — both divisible by 1/2/4/8/16/32/64.

**Prevention:** Don't pick exotic GPU counts (e.g. 3, 6, 12) unless the config has been hand-tuned for them.

### Error: `Prompt length exceeds model maximum sequence length (max_seq_len=N)`

**Cause:** Raised in `generate()` when any prompt is longer than `model.max_seq_len`.

**Solution:** Either truncate the prompt before tokenizing, or raise `max_seq_len` in the JSON config and re-launch.

**Prevention:** Length-check prompts in your harness before passing to `generate.py`.

### Error: `Number of prompts exceeds maximum batch size (N)`

**Cause:** In `--input-file` mode, the script asserts the line count is `<= args.max_batch_size` (default `8`).

**Solution:** Split the input file into chunks of ≤8 prompts, or raise `max_batch_size` in the JSON config (and pay the KV-cache memory cost).

**Prevention:** `wc -l prompts.txt` before launching.

### Error: `Either input-file or interactive mode must be specified`

**Cause:** `generate.py` asserts that at least one of `--input-file`/`--interactive` is provided.

**Solution:** Add `--interactive` for chat or `--input-file path` for batch.

**Prevention:** Bake one of the two flags into your launch script.

### Error: `Last dimension size must be divisible by block_size (block_size=128)`

**Cause:** Raised by `act_quant` when the input tensor's last dimension isn't divisible by 128. In normal operation this fires only if a config has been edited to a `dim` not divisible by 128.

**Solution:** Keep `dim` divisible by 128. All shipping configs comply (`2048`, `5120`, `7168`).

**Prevention:** Don't hand-edit `dim`.

### Error: `Input tensor must be contiguous` / `Input tensors must have 2 dimensions`

**Cause:** Triton kernels in `kernel.py` require contiguous, 2D tensors.

**Solution:** This indicates a bug in calling code; the shipping `model.py` always passes contiguous 2D tensors to `weight_dequant`/`fp8_gemm`/`act_quant`.

**Prevention:** If you've forked `model.py`, call `.contiguous()` on tensors before passing into the kernels.

### Warning: `Missing scale_inv tensor for {name}, skipping conversion`

**Cause:** `fp8_cast_bf16.py` couldn't find a `_scale_inv` companion for an FP8 weight.

**Solution:** Re-download the FP8 checkpoint; some shards are likely incomplete. Confirm `model.safetensors.index.json`'s `weight_map` references all expected `_scale_inv` entries.

**Prevention:** Verify the source checkpoint's integrity (Hugging Face provides per-file SHA256 in `*.safetensors`'s sibling files).

### CUDA out-of-memory at startup

**Cause:** The 671B / V3.1 model in BF16 is enormous; even FP8 needs roughly 1 byte per parameter for weights plus KV cache. The model is built on `cuda` immediately and KV caches are allocated for `max_batch_size * max_seq_len * n_layers`.

**Solution:**

1. Use `dtype: fp8` (the 671B / V3.1 configs default to this).
2. Reduce `max_batch_size` and/or `max_seq_len` in the JSON config.
3. Increase `--nproc-per-node × --nnodes` so each rank holds a smaller shard.

**Prevention:** Plan for ~80 GB / GPU when running 671B FP8 across 8 H100s; size accordingly.

### Mid-conversation, the model responds in the wrong language

**Cause:** Reported in [#1226](https://github.com/deepseek-ai/DeepSeek-V3/issues/1226) — the model occasionally replies in Chinese when conversation is in English.

**Solution:** Add an explicit system message to the chat template (e.g., `{"role": "system", "content": "Reply in English."}` prepended to `messages`). This repo doesn't auto-inject system prompts; you'd add one in your driver.

**Prevention:** Always set a system prompt for monolingual deployments.

### Model claims to be ChatGPT / another model

**Cause:** Reported in [#1224](https://github.com/deepseek-ai/DeepSeek-V3/issues/1224) — identity contamination from training data.

**Solution:** Treat as a known limitation of the released weights. Use a system prompt to set identity if it matters for your application.

**Prevention:** Don't rely on the model's self-identification; identity is a deployment concern, not a model guarantee.

### Repeated / duplicated tokens in output

**Cause:** Reported in [#1123](https://github.com/deepseek-ai/DeepSeek-V3/issues/1123). With this repo's sampler (no repetition penalty, no top-p, no top-k), low temperatures can drive into repetition loops.

**Solution:**

1. Raise `--temperature` toward `0.7`.
2. Lower `--max-new-tokens` so loops are short-lived.
3. If you need a repetition penalty, fork `sample()` in `generate.py` — it's six lines.

**Prevention:** Avoid `--temperature 0.0` for long-form generation.

## 7. FAQ

**Q: Does this repo include training code?**

No. It is inference-only. There is no optimizer, no dataloader, no training loop. `Transformer.forward` is wrapped in `@torch.inference_mode()`.

**Q: Where do I get the weights?**

From DeepSeek's Hugging Face releases. The repo does not bundle them and `repo_docs.cbf` ships empty README content for this commit. See the public Hugging Face page for the current set of released checkpoints.

**Q: Can I run this on a single 24 GB consumer GPU?**

Only the 16B config has any chance, and only with quantization beyond what this repo provides. The shipping configs build the entire model on `cuda` at construction time. For 16B in BF16 you need about 32 GB just for weights.

**Q: Does it support FlashAttention / PagedAttention / SDPA fused kernels?**

No. The attention path uses plain `torch.einsum` for QK and AV products, with `softmax(dim=-1, dtype=torch.float32)` in between. There is no SDPA / flash / paged backend.

**Q: Does it support speculative decoding?**

No. The decode loop is single-step: one forward per generated token (after prefill).

**Q: Does it support a server / OpenAI-compatible API?**

No. There is no HTTP server, no `/v1/chat/completions` endpoint, no streaming protocol — just a stdin REPL and a batch file mode. For serving, route to vLLM, SGLang, or TGI; this repo is not a server.

**Q: What is "MLA" and why is it not standard multi-head attention?**

Multi-Head **Latent** Attention. Instead of caching `(B, T, H, D)` keys and values, MLA caches a low-rank latent `kv_cache` of shape `(B, T, kv_lora_rank)` plus a small rotary `pe_cache` of shape `(B, T, qk_rope_head_dim)`. At decode time, the model reconstructs per-head K/V from the latent on the fly — dramatically reducing KV-cache memory at long context.

**Q: What is "absorb" vs "naive" attention?**

Two implementations of MLA's forward pass that share the same math but trade memory for compute. "absorb" (default; `attn_impl = "absorb"` module-level constant in `model.py`) keeps only the latent + rope caches and folds the up-projection (`wkv_b`) into the QK and AV einsums. "naive" materializes K and V like a normal attention layer and caches them — useful for debugging, painful for memory.

**Q: Can I switch between "absorb" and "naive" without re-launching?**

Not via a CLI flag. `attn_impl` is a module-level constant in `model.py`. Edit the source or set it before instantiating the model.

**Q: Does it support LoRA / adapters at inference?**

The model uses LoRA-shaped projections internally (`q_lora_rank`, `kv_lora_rank` parameters in `MLA`), but these are part of the **base architecture**, not user-attachable LoRA adapters. There is no `peft` / `bitsandbytes` integration.

**Q: What's MoE expert routing actually doing?**

`Gate.forward` scores tokens against `n_routed_experts` weights, optionally biases (671B/V3.1 only — `bias` is created when `dim == 7168`), and selects the top `n_activated_experts` (8 for 671B/V3.1). When `n_expert_groups > 1`, it first picks the top `n_limited_groups` groups, masks out the rest, then picks individual experts within them. This is "limited routing" — it caps cross-group communication.

**Q: Why does `Gate` create a bias only when `dim == 7168`?**

In `model.py`: `self.bias = nn.Parameter(...) if self.dim == 7168 else None`. The 671B and V3.1 configs use `dim: 7168`; their gate has a learned bias (used in conjunction with `score_func: sigmoid`). The 16B and 236B configs use `dim: 2048` / `5120` and don't get the bias. It's hard-coded; if you fork to a new `dim` and need a bias-shaped gate, edit that line.

**Q: How do I get logprobs for sampled tokens?**

Not exposed. `generate()` returns only token IDs. To get logprobs, fork `generate()` to also collect per-step `logits.softmax(-1).log()[next_token]`.

**Q: Why does the script print `print(args)` at startup?**

Diagnostic. It dumps the resolved `ModelArgs` to stdout (rank 0 only) so you can sanity-check what config was actually loaded. Helpful when you have many JSON variants.

**Q: Why a manual seed of `965` in `main()`?**

Reproducibility for the 2-token warm-up generation. Once you start typing prompts, the seed effectively no longer matters because each call to `sample()` consumes its own random tensor.

**Q: What is `scale_fmt: ue8m0`?**

UE8M0 — an unsigned 8-bit format with 8 exponent bits and 0 mantissa bits, i.e., a pure power-of-two scale. When `scale_fmt: "ue8m0"` is set (V3.1 config only), `act_quant_kernel` rounds the per-block scale up to the nearest power of two. This trades a tiny amount of quantization accuracy for cheaper hardware decoding of the scale.

**Q: Where is the chat template defined?**

In the tokenizer that ships with the checkpoint, not in this repo. `tokenizer.apply_chat_template(...)` reads the template from `tokenizer_config.json` inside `--ckpt-path`.

**Q: Can I export to ONNX / TensorRT / GGUF?**

Not from this repo. The export ecosystem (HF → GGUF, HF → TensorRT-LLM) plugs into the original Hugging Face checkpoint, not the post-`convert.py` sharded layout.

**Q: Does the repo support quantization beyond FP8?**

No. The only on-the-fly quantization is the FP8 path (`act_quant` quantizes activations to `float8_e4m3fn` per 128-element block when `dtype: fp8`). INT4 / INT8 / GPTQ / AWQ are not implemented here.

**Q: How do I file a bug?**

Open an issue at <https://github.com/deepseek-ai/DeepSeek-V3/issues>. Include your config JSON, exact `torchrun` command, full traceback, and `nvidia-smi` output.

---

_Generated April 2026 from deepseek-ai/DeepSeek-V3 at 9b4e978. AI-generated documentation. The library is checked by sampling (three repositories so far); this guide was not individually reviewed. Verify against the project before relying on it. Unofficial; not affiliated with or reviewed by the project._