DeepSeek-V3 User Guide

The deepseek-ai/DeepSeek-V3 repository is the reference inference implementation for DeepSeek's V3-class large language models. It is a deliberately small Python codebase — a single-file Transformer (model.py), a generation loop (generate.py), a Hugging Face checkpoint converter (convert.py), an FP8-to-BF16 weight caster (fp8_cast_bf16.py), and a Triton kernel module (kernel.py) — together with four JSON model configurations covering the 16B, 236B, 671B, and V3.1 variants.

This guide documents only what is in the repository at commit 9b4e978. It does not assume capabilities published in papers or blog posts unless those capabilities are present in the code.

1. Quick Start

DeepSeek-V3's inference repo is a research-grade reference. It loads a converted checkpoint, runs a multi-rank distributed forward pass with tensor parallelism, and decodes tokens through a small interactive or batch loop.

1.1 What's in the box

• inference/model.py — the Transformer: ModelArgs, Transformer, Block, MLA (Multi-Head Latent Attention), MoE (Mixture-of-Experts), Gate, MLP, Expert, parallel embedding, parallel/RMS-norm primitives.
• inference/generate.py — the run script. Loads the model, builds an AutoTokenizer from the checkpoint directory, supports --interactive and --input-file modes.
• inference/convert.py — converts a Hugging Face safetensors checkpoint into the repo's sharded layout (model{rank}-mp{mp}.safetensors).
• inference/fp8_cast_bf16.py — dequantizes an FP8 checkpoint into BF16 safetensors for hardware that lacks FP8 support.
• inference/kernel.py — Triton kernels for act_quant, weight_dequant, and fp8_gemm (block-scaled FP8 GEMM, autotuned).
• inference/configs/*.json — four model configs: config_16B.json, config_236B.json, config_671B.json, config_v3.1.json.

1.2 Prerequisites (single line)

CUDA-capable NVIDIA GPU(s), Python 3.10+, PyTorch 2.4.1, Triton 3.0.0, Transformers 4.46.3, Safetensors 0.4.5, plus the converted DeepSeek weights.

1.3 Minimal install + minimal run

git clone https://github.com/deepseek-ai/DeepSeek-V3.git
cd DeepSeek-V3
pip install -r inference/requirements.txt

# Single-GPU smoke test against an already-converted checkpoint
cd inference
torchrun --nnodes 1 --nproc-per-node 1 generate.py \
    --ckpt-path /path/to/converted-weights \
    --config configs/config_16B.json \
    --interactive

You will land at a >>> prompt. Type a message and press Enter; the model decodes a completion of up to --max-new-tokens (default 200) at --temperature 0.2.

Tip

Use /clear at the prompt to reset the chat history (the script keeps a running messages list for multi-turn chat). Use /exit to quit cleanly.

1.4 What to try next

• Convert a Hugging Face checkpoint before running anything — see Installation §2.4.
• Cast FP8 weights to BF16 if your GPUs do not support FP8 — see §5.
• Run the same prompt across multiple GPUs with tensor parallelism — see §5.

2. Installation

2.1 System requirements

2.2 Pinned dependencies

The repository pins exact versions in inference/requirements.txt:

torch==2.4.1
triton==3.0.0
transformers==4.46.3
safetensors==0.4.5

Important

These versions are pinned for a reason: the FP8 GEMM kernel uses torch.float8_e4m3fn and Triton 3.0.0's autotune API. Newer Triton versions have changed kernel-launch internals, and newer PyTorch versions can shift FP8 dtype behavior. If you upgrade, expect to debug.

2.3 Install

pip install -r inference/requirements.txt

There is no setup.py, no pyproject.toml, and no console-script entry point. You execute the scripts directly with torchrun (or, for single-GPU, with python if you set the env vars manually).

2.4 Convert a Hugging Face checkpoint

DeepSeek publishes its weights in Hugging Face safetensors layout. The repo expects a sharded layout keyed by model-parallel rank. Convert once:

cd inference
python convert.py \
    --hf-ckpt-path /path/to/hf-checkpoint \
    --save-path /path/to/converted-output \
    --n-experts 256 \
    --model-parallel 8

The converter:

1. Walks every *.safetensors in --hf-ckpt-path.
2. Renames Hugging Face parameter names (self_attn → attn, mlp → ffn, embed_tokens → embed, etc.) using the mapping dict in convert.py.
3. Skips model.layers.61 (the multi-token-prediction head, not used by this inference loop).
4. Shards each parameter across --model-parallel ranks: experts are partitioned by index, other weights are sharded along their tensor-parallel dimension (dim=0 for column-parallel weights, dim=1 for row-parallel o_proj/down_proj).
5. Writes one model{i}-mp{mp}.safetensors per rank into --save-path, plus copies any *token* files (tokenizer, tokenizer config) verbatim.

Important

--n-experts MUST be divisible by --model-parallel. The script asserts this and aborts otherwise. For the 671B / V3.1 configs (n_routed_experts: 256), valid --model-parallel values are 1, 2, 4, 8, 16, 32, 64, 128, 256.

2.5 Optional: cast FP8 weights to BF16

If your hardware does not support FP8, dequantize the checkpoint:

cd inference
python fp8_cast_bf16.py \
    --input-fp8-hf-path /path/to/fp8-checkpoint \
    --output-bf16-hf-path /path/to/bf16-checkpoint

The script:

• Reads model.safetensors.index.json from the input directory.
• Walks each safetensors shard, calls weight_dequant (Triton kernel) for any 1-byte-element weight, looking up the block-wise *_scale_inv tensor.
• Caches at most 2 source shards in CUDA memory at a time.
• Writes the resulting BF16 shards to --output-bf16-hf-path and emits a new model.safetensors.index.json with _scale_inv entries removed.

Warning

If a _scale_inv tensor is missing for an FP8 weight, the script prints Warning: Missing scale_inv tensor for {name}, skipping conversion and copies the FP8 weight through unchanged. The resulting checkpoint will then mix FP8 and BF16 tensors, which the model loader will not handle correctly. Verify your input checkpoint is complete before casting.

2.6 Verify the install

After installing dependencies, the fastest sanity check is to import the model module and instantiate the default ModelArgs (the 16B-shaped defaults — see inference/model.py):

cd inference
python -c "from model import ModelArgs, Transformer; import torch; \
torch.set_default_dtype(torch.bfloat16); torch.set_default_device('cuda'); \
args = ModelArgs(); m = Transformer(args); print('ok', sum(p.numel() for p in m.parameters()))"

This builds the model on GPU with random weights, useful only to confirm the imports and CUDA setup work. To actually generate text you need converted weights and generate.py.

3. Configuration

DeepSeek-V3 reads two kinds of configuration: the model config JSON (architecture) and environment variables for distributed launch.

3.1 The model config JSON

generate.py does:

with open(config) as f:
    args = ModelArgs(**json.load(f))

Every key in the JSON becomes a field of ModelArgs. Keys not present in the JSON take their ModelArgs default. Four configs ship with the repo:

Note

The 671B and V3.1 configs ship with dtype: fp8 and bake in expert grouping. The V3.1 config additionally sets scale_fmt: ue8m0, which switches the activation-quantization kernel to UE8M0-formatted scales (a power-of-two scale). All four configs share the same qk_nope_head_dim: 128, qk_rope_head_dim: 64, v_head_dim: 128, and kv_lora_rank: 512.

3.2 Environment variables

generate.py reads three environment variables and passes them straight to PyTorch's distributed runtime:

torchrun populates all three. You normally do not set them yourself.

3.3 CLI flags for generate.py

Tip

The sample() function divides by max(temperature, 1e-5) and uses Gumbel-style argmax (probs.div_(empty.exponential_(1)).argmax(-1)) — equivalent to multinomial sampling but branchless. Top-k / top-p / repetition-penalty are not implemented in this repo.

For the full configuration reference (every ModelArgs field with type, default, and effect), see the Administrator & Developer Guide.

4. Core Usage

4.1 The generation loop, end to end

generate.py does the following on every rank:

1. Reads WORLD_SIZE, RANK, LOCAL_RANK from the environment.
2. If WORLD_SIZE > 1, calls dist.init_process_group("nccl").
3. Silences print on every rank except 0.
4. torch.cuda.set_device(local_rank), sets default dtype to torch.bfloat16, sets torch.set_num_threads(8), seeds with torch.manual_seed(965).
5. Loads the JSON config into ModelArgs.
6. Builds the Transformer on the current CUDA device.
7. AutoTokenizer.from_pretrained(ckpt_path) — the tokenizer is read directly from the checkpoint directory.
8. Runs a 2-token warm-up generation against the literal prompt "DeepSeek" (this triggers Triton autotuning and KV-cache allocation paths before the user sees a prompt).
9. load_model(model, os.path.join(ckpt_path, f"model{rank}-mp{world_size}.safetensors")) — each rank loads only its own shard.
10. Enters interactive or batch mode.

Note

Step 9 happens after model construction, which means the model is built once with empty parameters and then weights are streamed into place via safetensors.torch.load_model. This is why --ckpt-path and --config must agree on the architecture.

4.2 Interactive chat

When --interactive is set:

>>> Hello, who are you?
I am DeepSeek...
>>> /clear
>>> What's the capital of France?
Paris.
>>> /exit

Behavior:

• Rank 0 reads input(">>> "). With multiple ranks, the prompt is broadcast to all ranks via dist.broadcast_object_list.
• The script maintains a messages list of {"role": "user" | "assistant", "content": str} dicts.
• Each turn, tokenizer.apply_chat_template(messages, add_generation_prompt=True) produces the input token IDs.
• generate(...) returns completion tokens; tokenizer.decode(..., skip_special_tokens=True) yields the printed reply.
• The reply is appended to messages so the next turn has full chat context.

Tip

/clear resets messages to [] mid-session — useful when you want to start fresh without restarting the process and reloading hundreds of GB of weights.

4.3 Batch mode

When --input-file path/to/prompts.txt is set:

torchrun --nnodes 1 --nproc-per-node 8 generate.py \
    --ckpt-path /weights/v3-mp8 \
    --config configs/config_671B.json \
    --input-file prompts.txt \
    --max-new-tokens 512 \
    --temperature 0.7

Each line of the file is treated as a single user message and rendered through apply_chat_template. The script asserts:

assert len(prompts) <= args.max_batch_size

max_batch_size defaults to 8 (see ModelArgs), so you cannot batch more than 8 prompts in a single invocation unless you raise max_batch_size in the JSON config.

Warning

max_batch_size does not just gate the assertion — it also sizes the KV cache buffers (k_cache, v_cache, kv_cache, pe_cache) that are allocated inside MLA at construction time. Raising it raises GPU memory consumption proportionally.

4.4 The decode loop in detail

generate(model, prompt_tokens, max_new_tokens, eos_id, temperature):

1. Builds a (batch, total_len) token tensor pre-filled with -1, copies in each prompt.
2. Tracks a prompt_mask = tokens != -1 so prompt positions are never overwritten.
3. Iterates cur_pos from min(prompt_lens) to total_len:
   • logits = model.forward(tokens[:, prev_pos:cur_pos], prev_pos) — forwards only the new slice (the rest is in KV cache).
   • If temperature > 0, samples; else takes argmax.
   • Where prompt_mask[:, cur_pos] is True, keeps the prompt token; otherwise writes the sampled token.
   • Marks rows finished if their newly emitted token equals eos_id and the position is past the prompt.
   • Breaks early once all rows have finished.
4. Truncates each completion at the first eos_id (if any).

Note

The prompt-overwrite trick (filling with -1, masking with prompt_mask) lets the loop run the prefill phase as a single forward, even when prompts have different lengths within a batch.

5. Common Tasks

The recipes below are everything generate.py, convert.py, and fp8_cast_bf16.py actually support out of the box. There are no other CLI surfaces in the repo.

5.1 Recipe: load a model and run one completion (single GPU)

Goal: Quick prompt → completion, no chat state.

Prerequisites: Converted checkpoint at /weights/v3-mp1 (i.e. --model-parallel 1 was used during convert.py), tokenizer files present in the same directory.

Steps:

cd DeepSeek-V3/inference
echo "Write a haiku about transformers." > prompts.txt
torchrun --nnodes 1 --nproc-per-node 1 generate.py \
    --ckpt-path /weights/v3-mp1 \
    --config configs/config_16B.json \
    --input-file prompts.txt \
    --max-new-tokens 64 \
    --temperature 0.7

Verify: Output prints Prompt: then Completion: for each line.

If it fails: See Troubleshooting §6 — particularly CUDA OOM, the max_seq_len assertion, and the missing-_scale_inv warning.

5.2 Recipe: interactive multi-turn chat

Goal: Sit at the >>> prompt and converse.

torchrun --nnodes 1 --nproc-per-node 1 generate.py \
    --ckpt-path /weights/v3-mp1 \
    --config configs/config_16B.json \
    --interactive

Tips:

• /clear to wipe context within the same process.
• /exit to quit.
• The chat template is whatever tokenizer.apply_chat_template was trained with — controlled by the tokenizer files inside --ckpt-path, not by this repo.

5.3 Recipe: multi-GPU tensor-parallel inference

Goal: Run the 671B / V3.1 model split across 8 GPUs on one node.

Prerequisites: A checkpoint converted with --model-parallel 8 (so files model0-mp8.safetensors through model7-mp8.safetensors exist).

cd DeepSeek-V3/inference
torchrun --nnodes 1 --nproc-per-node 8 generate.py \
    --ckpt-path /weights/v3-mp8 \
    --config configs/config_671B.json \
    --interactive

torchrun sets WORLD_SIZE=8, assigns each process a RANK 0–7 and LOCAL_RANK matching its GPU. dist.init_process_group("nccl") brings up the NCCL communicator.

Note

The model uses tensor parallelism only: column-parallel splits on wq, wq_b, wkv_b, w1, w3, and the LM head; row-parallel on wo and w2. There is no pipeline parallelism in this repo. Increasing --nproc-per-node strictly increases tensor-parallel degree.

5.4 Recipe: multi-node tensor-parallel inference

For multi-node launches, follow the standard torchrun recipe:

# On node 0 (master)
torchrun --nnodes 2 --nproc-per-node 8 \
    --node-rank 0 --master-addr <master-ip> --master-port 29500 \
    generate.py --ckpt-path /weights/v3-mp16 \
    --config configs/config_v3.1.json --interactive

# On node 1
torchrun --nnodes 2 --nproc-per-node 8 \
    --node-rank 1 --master-addr <master-ip> --master-port 29500 \
    generate.py --ckpt-path /weights/v3-mp16 \
    --config configs/config_v3.1.json --interactive

The checkpoint must have been converted with --model-parallel 16 to match the 16-GPU world.

5.5 Recipe: cast an FP8 checkpoint to BF16

If your GPUs don't support FP8 (anything pre-Hopper, or any non-NVIDIA accelerator):

cd DeepSeek-V3/inference
python fp8_cast_bf16.py \
    --input-fp8-hf-path /weights/deepseek-v3-fp8 \
    --output-bf16-hf-path /weights/deepseek-v3-bf16

Then convert the BF16 directory with convert.py and run generate.py against config_671B.json after editing dtype away from "fp8" (or use a config without dtype: fp8 — config_16B.json and config_236B.json ship with implicit bf16).

Caution

The cast roughly doubles disk usage. The 671B model in FP8 is around 700 GB; the BF16 copy will be roughly 1.4 TB. Make sure the destination volume has space before kicking off — there is no resume-from-partial logic.

5.6 Recipe: full pipeline from Hugging Face download

End-to-end, from raw HF weights to a chat REPL on 8 GPUs:

# 1. Download the HF release (use huggingface-cli or git lfs; not part of this repo).
huggingface-cli download deepseek-ai/DeepSeek-V3 --local-dir /raw/deepseek-v3-hf

# 2. (Optional) Cast FP8 -> BF16 if your GPUs don't support FP8.
python fp8_cast_bf16.py \
    --input-fp8-hf-path /raw/deepseek-v3-hf \
    --output-bf16-hf-path /raw/deepseek-v3-bf16

# 3. Shard for tensor parallelism.
python convert.py \
    --hf-ckpt-path /raw/deepseek-v3-hf \
    --save-path /weights/v3-mp8 \
    --n-experts 256 \
    --model-parallel 8

# 4. Launch.
torchrun --nnodes 1 --nproc-per-node 8 generate.py \
    --ckpt-path /weights/v3-mp8 \
    --config configs/config_671B.json \
    --interactive

Tip

Step 1 (the download) is not part of this repo. The Hugging Face CLI, git lfs, or hf-transfer are the usual choices.

5.7 Recipe: change the sampling temperature mid-experiment

Temperature is a CLI flag, not a runtime command. Restart the process with a new --temperature:

torchrun --nnodes 1 --nproc-per-node 1 generate.py \
    --ckpt-path /weights/v3-mp1 --config configs/config_16B.json \
    --interactive --temperature 0.0   # greedy

A non-positive value triggers the else branch in generate() and uses logits.argmax(dim=-1) directly.

5.8 Recipe: extend the maximum sequence length

max_seq_len defaults to 4096 * 4 = 16384 in ModelArgs. To change it without editing source, override in the config JSON:

{
  "max_seq_len": 32768,
  "vocab_size": 102400,
  "dim": 2048,
  "...": "..."
}

The model uses YaRN-style RoPE scaling: when max_seq_len > original_seq_len (default 4096), precompute_freqs_cis interpolates between scaled and unscaled frequencies using rope_factor (default 40), beta_fast (32), and beta_slow (1). MLA also rescales softmax_scale by (0.1 * mscale * log(rope_factor) + 1)^2 for extended contexts.

Warning

Larger max_seq_len enlarges KV-cache buffers in every attention layer. With attn_impl = "absorb" (the default), the cache is (max_batch_size, max_seq_len, kv_lora_rank) for kv_cache plus (max_batch_size, max_seq_len, qk_rope_head_dim) for pe_cache — manageable. With attn_impl = "naive" the caches scale with n_local_heads * qk_head_dim and n_local_heads * v_head_dim, much larger.

6. Troubleshooting

The repository surfaces only a handful of explicit error messages, mostly via Python assert. Below are the ones you'll actually hit.

Error: Number of experts must be divisible by model parallelism

Cause: Raised by convert.py when --n-experts % --model-parallel != 0.

Solution:

1. Confirm --n-experts matches the config: 64 for 16B, 160 for 236B, 256 for 671B and V3.1.
2. Use a --model-parallel that divides it evenly. For 256 experts: 1, 2, 4, 8, 16, 32, 64, 128, 256.

Prevention: Pick --model-parallel to equal your target --nproc-per-node × --nnodes and verify divisibility before launching.

Error: Vocabulary size must be divisible by world size (world_size=N)

Cause: Raised by ParallelEmbedding.__init__ (and the column-parallel head). The shipping configs use vocab_size: 102400 (16B/236B) or 129280 (671B/V3.1).

Solution: Pick a world_size that divides the vocab. 129280 = 2^7 × 5 × 202, so 1, 2, 4, 5, 8, 10, 16, 20, 32, 40, 64, 80, 128, 160 all divide cleanly. 102400 = 2^12 × 25, similarly flexible.

Prevention: Stick to powers of two for --nproc-per-node; both shipping vocabs cooperate.

Error: Output features must be divisible by world size / Input features must be divisible by world size

Cause: ColumnParallelLinear / RowParallelLinear assertions when a Linear's output (or input) features aren't divisible by world_size. Most shipping configs use n_heads and dim values designed to divide cleanly across 1/2/4/8/16 GPUs.

Solution: Use a world_size that divides the relevant feature counts. For config_671B.json: dim=7168 and n_heads=128 — both divisible by 1/2/4/8/16/32/64.

Prevention: Don't pick exotic GPU counts (e.g. 3, 6, 12) unless the config has been hand-tuned for them.

Error: Prompt length exceeds model maximum sequence length (max_seq_len=N)

Cause: Raised in generate() when any prompt is longer than model.max_seq_len.

Solution: Either truncate the prompt before tokenizing, or raise max_seq_len in the JSON config and re-launch.

Prevention: Length-check prompts in your harness before passing to generate.py.

Error: Number of prompts exceeds maximum batch size (N)

Cause: In --input-file mode, the script asserts the line count is <= args.max_batch_size (default 8).

Solution: Split the input file into chunks of ≤8 prompts, or raise max_batch_size in the JSON config (and pay the KV-cache memory cost).

Prevention: wc -l prompts.txt before launching.

Error: Either input-file or interactive mode must be specified

Cause: generate.py asserts that at least one of --input-file/--interactive is provided.

Solution: Add --interactive for chat or --input-file path for batch.

Prevention: Bake one of the two flags into your launch script.

Error: Last dimension size must be divisible by block_size (block_size=128)

Cause: Raised by act_quant when the input tensor's last dimension isn't divisible by 128. In normal operation this fires only if a config has been edited to a dim not divisible by 128.

Solution: Keep dim divisible by 128. All shipping configs comply (2048, 5120, 7168).

Prevention: Don't hand-edit dim.

Error: Input tensor must be contiguous / Input tensors must have 2 dimensions

Cause: Triton kernels in kernel.py require contiguous, 2D tensors.

Solution: This indicates a bug in calling code; the shipping model.py always passes contiguous 2D tensors to weight_dequant/fp8_gemm/act_quant.

Prevention: If you've forked model.py, call .contiguous() on tensors before passing into the kernels.

Warning: Missing scale_inv tensor for {name}, skipping conversion

Cause: fp8_cast_bf16.py couldn't find a _scale_inv companion for an FP8 weight.

Solution: Re-download the FP8 checkpoint; some shards are likely incomplete. Confirm model.safetensors.index.json's weight_map references all expected _scale_inv entries.

Prevention: Verify the source checkpoint's integrity (Hugging Face provides per-file SHA256 in *.safetensors's sibling files).

CUDA out-of-memory at startup

Cause: The 671B / V3.1 model in BF16 is enormous; even FP8 needs roughly 1 byte per parameter for weights plus KV cache. The model is built on cuda immediately and KV caches are allocated for max_batch_size * max_seq_len * n_layers.

Solution:

1. Use dtype: fp8 (the 671B / V3.1 configs default to this).
2. Reduce max_batch_size and/or max_seq_len in the JSON config.
3. Increase --nproc-per-node × --nnodes so each rank holds a smaller shard.

Prevention: Plan for ~80 GB / GPU when running 671B FP8 across 8 H100s; size accordingly.

Mid-conversation, the model responds in the wrong language

Cause: Reported in #1226 — the model occasionally replies in Chinese when conversation is in English.

Solution: Add an explicit system message to the chat template (e.g., {"role": "system", "content": "Reply in English."} prepended to messages). This repo doesn't auto-inject system prompts; you'd add one in your driver.

Prevention: Always set a system prompt for monolingual deployments.

Model claims to be ChatGPT / another model

Cause: Reported in #1224 — identity contamination from training data.

Solution: Treat as a known limitation of the released weights. Use a system prompt to set identity if it matters for your application.

Prevention: Don't rely on the model's self-identification; identity is a deployment concern, not a model guarantee.

Repeated / duplicated tokens in output

Cause: Reported in #1123. With this repo's sampler (no repetition penalty, no top-p, no top-k), low temperatures can drive into repetition loops.

Solution:

1. Raise --temperature toward 0.7.
2. Lower --max-new-tokens so loops are short-lived.
3. If you need a repetition penalty, fork sample() in generate.py — it's six lines.

Prevention: Avoid --temperature 0.0 for long-form generation.

7. FAQ

Q: Does this repo include training code?

No. It is inference-only. There is no optimizer, no dataloader, no training loop. Transformer.forward is wrapped in @torch.inference_mode().

Q: Where do I get the weights?

From DeepSeek's Hugging Face releases. The repo does not bundle them and repo_docs.cbf ships empty README content for this commit. See the public Hugging Face page for the current set of released checkpoints.

Q: Can I run this on a single 24 GB consumer GPU?

Only the 16B config has any chance, and only with quantization beyond what this repo provides. The shipping configs build the entire model on cuda at construction time. For 16B in BF16 you need about 32 GB just for weights.

Q: Does it support FlashAttention / PagedAttention / SDPA fused kernels?

No. The attention path uses plain torch.einsum for QK and AV products, with softmax(dim=-1, dtype=torch.float32) in between. There is no SDPA / flash / paged backend.

Q: Does it support speculative decoding?

No. The decode loop is single-step: one forward per generated token (after prefill).

Q: Does it support a server / OpenAI-compatible API?

No. There is no HTTP server, no /v1/chat/completions endpoint, no streaming protocol — just a stdin REPL and a batch file mode. For serving, route to vLLM, SGLang, or TGI; this repo is not a server.

Q: What is "MLA" and why is it not standard multi-head attention?

Multi-Head Latent Attention. Instead of caching (B, T, H, D) keys and values, MLA caches a low-rank latent kv_cache of shape (B, T, kv_lora_rank) plus a small rotary pe_cache of shape (B, T, qk_rope_head_dim). At decode time, the model reconstructs per-head K/V from the latent on the fly — dramatically reducing KV-cache memory at long context.

Q: What is "absorb" vs "naive" attention?

Two implementations of MLA's forward pass that share the same math but trade memory for compute. "absorb" (default; attn_impl = "absorb" module-level constant in model.py) keeps only the latent + rope caches and folds the up-projection (wkv_b) into the QK and AV einsums. "naive" materializes K and V like a normal attention layer and caches them — useful for debugging, painful for memory.

Q: Can I switch between "absorb" and "naive" without re-launching?

Not via a CLI flag. attn_impl is a module-level constant in model.py. Edit the source or set it before instantiating the model.

Q: Does it support LoRA / adapters at inference?

The model uses LoRA-shaped projections internally (q_lora_rank, kv_lora_rank parameters in MLA), but these are part of the base architecture, not user-attachable LoRA adapters. There is no peft / bitsandbytes integration.

Q: What's MoE expert routing actually doing?

Gate.forward scores tokens against n_routed_experts weights, optionally biases (671B/V3.1 only — bias is created when dim == 7168), and selects the top n_activated_experts (8 for 671B/V3.1). When n_expert_groups > 1, it first picks the top n_limited_groups groups, masks out the rest, then picks individual experts within them. This is "limited routing" — it caps cross-group communication.

Q: Why does Gate create a bias only when dim == 7168?

In model.py: self.bias = nn.Parameter(...) if self.dim == 7168 else None. The 671B and V3.1 configs use dim: 7168; their gate has a learned bias (used in conjunction with score_func: sigmoid). The 16B and 236B configs use dim: 2048 / 5120 and don't get the bias. It's hard-coded; if you fork to a new dim and need a bias-shaped gate, edit that line.

Q: How do I get logprobs for sampled tokens?

Not exposed. generate() returns only token IDs. To get logprobs, fork generate() to also collect per-step logits.softmax(-1).log()[next_token].

Q: Why does the script print print(args) at startup?

Diagnostic. It dumps the resolved ModelArgs to stdout (rank 0 only) so you can sanity-check what config was actually loaded. Helpful when you have many JSON variants.

Q: Why a manual seed of 965 in main()?

Reproducibility for the 2-token warm-up generation. Once you start typing prompts, the seed effectively no longer matters because each call to sample() consumes its own random tensor.

Q: What is scale_fmt: ue8m0?

UE8M0 — an unsigned 8-bit format with 8 exponent bits and 0 mantissa bits, i.e., a pure power-of-two scale. When scale_fmt: "ue8m0" is set (V3.1 config only), act_quant_kernel rounds the per-block scale up to the nearest power of two. This trades a tiny amount of quantization accuracy for cheaper hardware decoding of the scale.

Q: Where is the chat template defined?

In the tokenizer that ships with the checkpoint, not in this repo. tokenizer.apply_chat_template(...) reads the template from tokenizer_config.json inside --ckpt-path.

Q: Can I export to ONNX / TensorRT / GGUF?

Not from this repo. The export ecosystem (HF → GGUF, HF → TensorRT-LLM) plugs into the original Hugging Face checkpoint, not the post-convert.py sharded layout.

Q: Does the repo support quantization beyond FP8?

No. The only on-the-fly quantization is the FP8 path (act_quant quantizes activations to float8_e4m3fn per 128-element block when dtype: fp8). INT4 / INT8 / GPTQ / AWQ are not implemented here.

Q: How do I file a bug?

Open an issue at https://github.com/deepseek-ai/DeepSeek-V3/issues. Include your config JSON, exact torchrun command, full traceback, and nvidia-smi output.