DeepSeek-R1 Administrator & Developer Guide

1. Architecture Overview

DeepSeek-R1 is a reasoning language model series released by DeepSeek AI under the MIT license. The repository at deepseek-ai/DeepSeek-R1 (commit 0cf7856, dated 2025-04-09) is a model release artifact — it contains the research paper (DeepSeek_R1.pdf), a README.md, a benchmark image (figures/benchmark.jpg), and an MIT LICENSE file. There is no application source code, no inference runtime, no configuration files, and no dependency manifests in this repository.

The deployable system is assembled from components hosted elsewhere:

[DeepSeek API (cloud)]                       [Self-hosted deployment]
      │                                              │
      ▼                                    ┌─────────▼────────┐
 platform.deepseek.com                     │  Inference server │
 (OpenAI-compatible REST API)              │  (vLLM / TGI /   │
      │                                    │   Transformers)   │
      │                                    └─────────┬────────┘
      │                                              │
      ▼                                              ▼
 Your application                          Model weights (Hugging Face)
 (OpenAI SDK / HTTP)                       deepseek-ai/{variant}

The model family consists of:

Note

This repository contains only research artifacts — not runnable code. All operational components (inference runtimes, API server, model weights) are external to this repository. Deployments assemble these from Hugging Face, vLLM, and the DeepSeek API platform separately.

Data flow — API path

Client request (OpenAI SDK)
  → HTTPS POST https://api.deepseek.com/v1/chat/completions
  → DeepSeek inference cluster (full 671B MoE model)
  → JSON response with reasoning trace (<think>...</think>) + final answer

Data flow — self-hosted path

Client request (OpenAI SDK or HTTP)
  → vLLM / TGI serving endpoint (local or cloud VM)
  → Model weights loaded from disk (Hugging Face snapshot cache)
  → Streaming or batch response with reasoning trace + final answer

2. System Requirements

Hardware Requirements by Variant

Important

FP16 figures assume full-precision weights. INT4 figures use bitsandbytes or AWQ quantization. Quantized models trade a small quality loss for dramatically reduced VRAM requirements. For production inference at scale, INT4 with vLLM's tensor-parallel mode is the standard approach for the 70B variant.

Software Prerequisites

3. Configuration Reference

The GitHub repository (deepseek-ai/DeepSeek-R1) contains no configuration files. All configuration is managed at the inference framework level.

vLLM Server Configuration

vllm serve deepseek-ai/DeepSeek-R1-Distill-Qwen-7B \
  --host 0.0.0.0 \
  --port 8000 \
  --tensor-parallel-size 1 \
  --max-model-len 32768 \
  --dtype bfloat16

Tip

For multi-GPU deployments of the 70B distill variant, set --tensor-parallel-size 4 (for 4×A100 80GB) or --tensor-parallel-size 8 for H100 nodes. vLLM handles tensor parallelism transparently.

Hugging Face Transformers Configuration

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "deepseek-ai/DeepSeek-R1-Distill-Qwen-7B",
    device_map="auto",           # Automatically assign layers to GPU/CPU
    torch_dtype="auto",          # Use model's preferred dtype
    load_in_4bit=True,           # Optional: 4-bit quantization
    trust_remote_code=True       # Required for some model architectures
)
tokenizer = AutoTokenizer.from_pretrained(
    "deepseek-ai/DeepSeek-R1-Distill-Qwen-7B",
    trust_remote_code=True
)

Ollama Configuration

# Pull and serve
ollama pull deepseek-r1:7b
ollama serve   # Starts API server on localhost:11434

# Configure via environment variables
OLLAMA_HOST=0.0.0.0:11434 ollama serve
OLLAMA_NUM_GPU=1 ollama serve

API Call Parameters

Important

The tool_choice and logprobs parameters are not supported on the hosted DeepSeek API for the R1 model (GitHub issues #813, #815). Self-hosted vLLM deployments support logprobs. Tool calling support varies by deployment framework.

4. Deployment & Scaling

Deployment Option A — DeepSeek API (Zero Infrastructure)

The simplest production path. DeepSeek operates the full 671B model behind an OpenAI-compatible API.

from openai import OpenAI

client = OpenAI(
    api_key="sk-...",
    base_url="https://api.deepseek.com"
)

Advantages: No GPU infrastructure, no weight management, no model serving maintenance. Limitations: API parity gaps (no tool_choice, no logprobs), data leaves your environment, latency depends on DeepSeek's infrastructure.

Deployment Option B — vLLM (Recommended for Self-Hosted Production)

vLLM provides the highest throughput for serving distilled variants.

Single-node deployment (7B example):

pip install vllm
vllm serve deepseek-ai/DeepSeek-R1-Distill-Qwen-7B \
  --host 0.0.0.0 --port 8000 \
  --max-model-len 32768 \
  --dtype bfloat16

Multi-GPU deployment (70B example, 4×GPU):

vllm serve deepseek-ai/DeepSeek-R1-Distill-Llama-70B \
  --host 0.0.0.0 --port 8000 \
  --tensor-parallel-size 4 \
  --dtype bfloat16

The vLLM endpoint is OpenAI-compatible:

client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1")
response = client.chat.completions.create(
    model="deepseek-ai/DeepSeek-R1-Distill-Qwen-7B",
    messages=[{"role": "user", "content": "Test"}]
)

Deployment Option C — Text Generation Inference (TGI)

Hugging Face TGI is an alternative to vLLM with strong Docker integration:

docker run --gpus all \
  -v $HOME/.cache/huggingface:/data \
  -p 8080:80 \
  ghcr.io/huggingface/text-generation-inference:latest \
  --model-id deepseek-ai/DeepSeek-R1-Distill-Qwen-7B \
  --max-input-length 4096 \
  --max-total-tokens 8192

Deployment Option D — Ollama (Dev/Small Team)

ollama pull deepseek-r1:7b   # or 1.5b, 8b, 14b, 32b, 70b
OLLAMA_HOST=0.0.0.0:11434 ollama serve

Ollama's API is OpenAI-compatible when you set base_url="http://localhost:11434/v1".

Deployment Checklist

• [x] Choose the correct model variant for available hardware
• [x] Download weights from Hugging Face (or confirm Ollama pull succeeded)
• [x] Verify CUDA version matches PyTorch / vLLM requirements
• [ ] Configure serving endpoint host/port and firewall rules
• [ ] Set memory utilization limits to prevent OOM on concurrent requests
• [ ] Configure reverse proxy (nginx/Caddy) with TLS if internet-facing
• [ ] Implement API key authentication at the reverse proxy layer
• [ ] Set up log aggregation and alerting on OOM/crash events

Scaling Considerations

5. Operations

Starting and Stopping Services

vLLM:

# Start
vllm serve deepseek-ai/DeepSeek-R1-Distill-Qwen-7B --port 8000 &

# Stop
kill $(lsof -t -i:8000)

Ollama:

# Start daemon
ollama serve

# Stop (Linux systemd)
systemctl stop ollama

Note

vLLM does not include a built-in process manager. Run it under systemd, supervisord, or in a container to ensure automatic restart on crash.

Health Checks

vLLM:

curl http://localhost:8000/health

{}

An HTTP 200 response indicates the server is ready. Non-200 or connection refused means the model is still loading or has crashed.

Ollama:

curl http://localhost:11434/api/tags

{"models": [{"name": "deepseek-r1:7b", ...}]}

Monitoring

Key metrics to track for a self-hosted vLLM deployment:

# Quick GPU health check
nvidia-smi --query-gpu=name,memory.used,memory.total,temperature.gpu \
  --format=csv,noheader

Tip

vLLM exposes a Prometheus-compatible /metrics endpoint. Scrape it with Prometheus and visualize with Grafana for production observability. The most important metrics are vllm:num_requests_running, vllm:gpu_cache_usage_perc, and vllm:generation_tokens_total.

Model Weight Management

# Check Hugging Face cache location
python -c "from huggingface_hub import constants; print(constants.HF_HUB_CACHE)"

# List downloaded models
ls $(python -c "from huggingface_hub import constants; print(constants.HF_HUB_CACHE)")

# Download a specific variant
huggingface-cli download deepseek-ai/DeepSeek-R1-Distill-Qwen-7B

# Remove cached weights
huggingface-cli delete-cache

Note

Model weights are large binary files. The Hugging Face cache defaults to ~/.cache/huggingface/hub/. Bind-mount this to persistent storage in containerized deployments so weights survive container restarts.

6. Performance

Throughput Benchmarks (Approximate)

Warning

These figures are illustrative estimates based on hardware specifications. Actual throughput depends on prompt length, reasoning depth, and concurrent request count. Reasoning models typically generate more tokens than standard chat models due to the <think> block — benchmark with representative workloads, not toy prompts.

Optimizations

• Enable prefix caching (--enable-prefix-caching): reduces latency for repeated system prompts by reusing computed KV cache.
• Quantization (AWQ/GPTQ): reduces VRAM by ~50–75% with minimal quality loss.
• Continuous batching: vLLM handles this automatically — do not implement manual batching.
• Speculative decoding: available in newer vLLM versions; use a small draft model for faster generation.

7. Security

API Key Management

• Never embed API keys in source code. Store in environment variables or a secrets manager (Vault, AWS Secrets Manager, etc.).
• Rotate keys if exposed. Revoke at platform.deepseek.com.
• For self-hosted deployments, authenticate at the reverse proxy layer (nginx auth_request, Caddy forward-auth) — vLLM and Ollama do not implement API authentication by default.

Caution

Ollama and vLLM bind to all interfaces if --host 0.0.0.0 is set and have no built-in authentication. Never expose these ports directly to the internet. Always place a TLS-terminating reverse proxy with authentication in front.

Data Privacy

• API path: prompts and responses are processed by DeepSeek's infrastructure. Review DeepSeek's privacy policy before sending sensitive data via the API.
• Self-hosted path: all data stays within your infrastructure. Use network policies to restrict access to the inference endpoint.
• Model weights are static files — they do not exfiltrate data or phone home.

License Compliance

The full DeepSeek-R1 model weights and all distilled variants are released under the MIT license. Key obligations:

• Include the MIT license notice in any redistribution of the weights
• The license does not restrict commercial use, modification, or sublicensing
• Distilled variants derive from Qwen2.5 (Apache 2.0) and Llama-3 (Meta Community License) base models. Verify compliance with the applicable base model license for each variant.

Important

The Llama-3 base model (used by Distill-Llama-8B and Distill-Llama-70B) is subject to Meta's Community License Agreement, which imposes usage restrictions for services with more than 700 million monthly active users. Review this before large-scale commercial deployment of Llama-based distills.

Input Validation

When building applications on top of DeepSeek-R1:

• Sanitize user input before injecting into system prompts to prevent prompt injection
• Limit max_tokens per request to bound compute cost and response size
• Rate-limit requests per user at the application layer
• Log requests (excluding content if privacy-sensitive) for audit and abuse detection

8. Developer Internals

Repository Structure

deepseek-ai/DeepSeek-R1 @ 0cf7856
├── DeepSeek_R1.pdf          # Research paper (technical architecture, training methodology, results)
├── LICENSE                  # MIT License
├── README.md                # Model overview, benchmark table, Hugging Face links, usage examples
└── figures/
    └── benchmark.jpg        # Benchmark comparison chart referenced in README

The repository contains 4 files and 0 lines of source code. All implementation lives in:

• Hugging Face model repositories under the deepseek-ai organization
• The vLLM / Transformers inference frameworks
• DeepSeek's proprietary API infrastructure (closed-source)

Model Architecture — R1 Full (671B MoE)

Based on the research paper (DeepSeek_R1.pdf):

• Architecture type: Mixture-of-Experts (MoE) Transformer
• Active parameters per forward pass: A subset of the 671B total (MoE routing activates a fraction of expert layers per token)
• Training approach: Reinforcement Learning from Human Feedback (RLHF) with Group Relative Policy Optimization (GRPO), combined with supervised fine-tuning on chain-of-thought data
• Reasoning mechanism: Extended thinking via <think> tags produced as part of the autoregressive generation process — not a separate inference pass
• Context window: Up to 128K tokens

Distilled Model Architecture

The distilled models are dense transformers that do not use MoE routing. Knowledge distillation transfers R1's reasoning behavior into the smaller architectures:

Reasoning Training

The core training novelty documented in the paper is that the model learns multi-step reasoning through RL reward signals (correctness of final answer) rather than imitation of pre-labeled reasoning chains. The <think> block emerges as the model's learned scratchpad format during RL training.

Benchmark Performance

The figures/benchmark.jpg in the repository shows comparative benchmark results against other frontier models (GPT-4o, Claude series, Gemini) across mathematical reasoning, coding, and scientific problem-solving tasks. Consult the research paper for exact numbers and methodology.

AST Extraction Summary

cli_commands:   0   (no source code)
api_endpoints:  0   (no source code)
functions:      0   (no source code)
classes:        0   (no source code)
config_keys:    0   (no source code)

All zeros reflect the absence of runnable code in this repository, not a tool limitation.

Integration Points

9. Troubleshooting Reference

vLLM OOM on Startup

Cause: Model too large for available VRAM with current settings.

Solution:

1. Reduce --gpu-memory-utilization from 0.90 to 0.80
2. Enable quantization: --quantization awq (requires pre-quantized weights)
3. Add --max-model-len to cap sequence length and reduce KV cache allocation
4. Switch to a smaller model variant

Warning

Do not set --gpu-memory-utilization above 0.95 in production. The remaining headroom prevents OOM kills during peak KV cache usage under concurrent load.

Slow First Response (Cold Start)

Cause: Model weights are being loaded from disk and transferred to GPU VRAM.

Solution: Loading is a one-time cost per server start. Large models (70B) take 2–5 minutes to load on typical NVMe + PCIe bandwidth. Monitor with nvidia-smi — VRAM utilization climbing steadily indicates loading in progress.

Inconsistent Reasoning Quality

Cause: Temperature too high causing erratic chain-of-thought.

Solution: For deterministic math and code tasks, use temperature=0.5 to 0.7. For creative tasks, temperature=1.0 is appropriate. Avoid temperature=0 — it can cause infinite repetition in reasoning chains.

Tip

If the reasoning chain trails off into repetition before reaching a conclusion, reduce temperature and increase max_tokens. The model is more likely to complete a reasoning chain when there is token budget remaining and sampling is less random.

Hugging Face Download Stalls

Cause: Network interruption or hub rate limiting.

Solution:

1. Resume with huggingface-cli download {model_id} — it resumes from partial downloads
2. Use HF_HUB_ENABLE_HF_TRANSFER=1 environment variable for faster multi-part downloads
3. Set HF_ENDPOINT=https://hf-mirror.com for mirror access if the main hub is unreachable

Requests Fail After Long Idle Period

Cause: vLLM process has crashed and not restarted (no process manager).

Solution: Configure systemd or a supervisor to automatically restart vLLM on failure.

# /etc/systemd/system/vllm.service
[Unit]
Description=vLLM inference server
After=network.target

[Service]
Type=simple
ExecStart=/usr/local/bin/vllm serve deepseek-ai/DeepSeek-R1-Distill-Qwen-7B --host 0.0.0.0 --port 8000
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target

systemctl enable vllm && systemctl start vllm