llama.cpp — Administrator & Developer Guide

Note

This guide targets administrators deploying llama.cpp in production environments, developers building on the library, and contributors working on the codebase. For end-user installation and usage, see the companion User Guide.

Table of Contents

1. Architecture Overview
2. System Requirements
3. Configuration Reference
4. Deployment & Scaling
5. Operations
6. Performance
7. Security
8. Developer Internals
9. Integration Points

1. Architecture Overview

What llama.cpp Is

llama.cpp is a high-performance inference engine for large language models, written in C/C++ with minimal dependencies. Its primary goal is to run modern LLM architectures efficiently across the widest possible range of hardware — from laptop CPUs to multi-GPU server clusters to mobile devices — without requiring cloud infrastructure. The project is the canonical runtime for the GGUF model format and is the upstream source for many derivative projects (llama-python, ollama, LM Studio, Jan, and dozens more).

The library is structured as a clean C API with C++ internals. Applications link against libllama (or consume it via the OpenAI-compatible server), while the underlying compute is handled by GGML — a separate tensor-operation library embedded in the repository at ggml/. The separation is intentional: GGML is usable as a standalone library for other inference engines (Whisper.cpp, Stable Diffusion, etc.) and has its own backend abstraction that allows new hardware targets to be added without touching llama-level code.

What llama.cpp Does

At runtime, llama.cpp loads a GGUF model file into memory (optionally memory-mapping it for zero-copy access), constructs an inference context that owns the KV cache and execution graph, and runs the transformer forward pass for each token generation step. The core decode path (llama_decode()) tokenizes prompts, schedules batch computation across available backends, performs attention with optional flash attention acceleration, applies the sampler chain to select the next token, and returns logits or sampled tokens to the caller. The server wraps this in an HTTP interface; the CLI wraps it in an interactive terminal session.

Model support covers 100+ architectures (LLaMA, Mistral, Gemma, Phi, Qwen, DeepSeek, GPT-NeoX, Falcon, StarCoder, Command-R, BLOOM, and many more), implemented as separate files in src/models/. Quantization support spans 40+ types from full F32 down to sub-1-bit representations, all defined in the GGML quantization type registry.

Architecture Diagram

┌─────────────────────────────────────────────────────────────────┐
│                         Applications                             │
│   llama-cli   llama-server   llama-quantize   llama-bench        │
│   llama-perplexity  llama-imatrix  llama-tts  llama-mtmd         │
├─────────────────────────────────────────────────────────────────┤
│                        Common Layer                              │
│   arg.cpp   chat.cpp   sampling.cpp   download.cpp               │
│   log.cpp   ngram-cache.cpp   jinja/ (template engine)           │
├─────────────────────────────────────────────────────────────────┤
│                       llama Library                              │
│   llama.h (public C API)   llama-cpp.h (C++ RAII wrappers)       │
│   llama-model   llama-context   llama-sampling   llama-memory     │
│   llama-kv-cache   llama-graph   llama-adapter   llama-mmap       │
│   src/models/ (114 architecture implementations)                  │
├─────────────────────────────────────────────────────────────────┤
│                        GGML Engine                               │
│   ggml.h   ggml-backend (interface + registry)   ggml-alloc      │
│   gguf (reader/writer)   ggml-threading   ggml-quants             │
├─────────────────────────────────────────────────────────────────┤
│                     Hardware Backends                            │
│  CPU │ CUDA │ Metal │ Vulkan │ OpenCL │ SYCL │ CANN │ Hexagon   │
│  RPC │ HIP  │ MUSA  │ ZenDNN │ zDNN  │ VirtGPU │ BLAS │ OpenVINO│
│  WebGPU │ KleidiAI (ARM) │ llamafile SGEMM                       │
└─────────────────────────────────────────────────────────────────┘

Component Inventory

Technology Stack

Data Flow

Model File (GGUF on disk)
        │
        ▼
llama_model_load_from_file()
  ├── Parse GGUF header & metadata
  ├── Build tensor map
  ├── mmap or read tensors into memory
  └── Assign tensors to backend buffers (GPU/CPU)
        │
        ▼
llama_init_from_model()
  ├── Allocate KV cache (type_k / type_v, quantized optional)
  ├── Build computation graph skeleton
  └── Allocate scratch buffers per backend
        │
        ▼
Prompt Processing
  ├── llama_tokenize() → token IDs
  ├── llama_batch_init() → populate batch
  └── llama_decode() → fill KV cache
        │
        ▼
Token Generation Loop
  ├── llama_decode() → compute logits
  ├── llama_sampler_chain_sample() → next token
  ├── llama_token_to_piece() → detokenize
  └── Stream to caller / HTTP client

2. System Requirements

Hardware Requirements by Model Size

Note

VRAM figures assume Q4_K_M quantization. Full F16 weights require roughly 2× the parameter count in bytes (e.g., 7B F16 ≈ 14 GB). The KV cache adds additional memory proportional to n_ctx × n_layers × n_heads × head_dim × 2 × sizeof(type_k).

Memory Estimation Formulas

Model weights (approximate):

bytes ≈ parameters × bits_per_weight / 8

KV cache (per token, approximate):

kv_bytes_per_token = 2 × n_layers × n_kv_heads × head_dim × sizeof(kv_type)
total_kv = kv_bytes_per_token × n_ctx

For Llama 3 8B at F16 KV, 4096 context:

2 × 32 × 8 × 128 × 2 bytes = 131,072 bytes/token × 4096 = 512 MB

Quantized KV (Q8_0) halves this; Q4_0 quarters it at some quality cost.

Software Requirements

Backend-Specific Requirements

CUDA:

• NVIDIA driver 520+ (for CUDA 11.8); 545+ (for CUDA 12.x)
• cuBLAS is used automatically; no separate install needed with toolkit
• Compute capability 5.0+ (Maxwell); 7.5+ recommended for best performance

Metal (Apple Silicon / Intel Mac):

• macOS 12.3 (Monterey) minimum
• M1/M2/M3/M4 chips use unified memory — full model fits in shared GPU/CPU pool
• GGML_METAL_EMBED_LIBRARY=ON bundles .metallib into the binary for portable distribution

Vulkan:

• Windows: Install LunarG Vulkan SDK
• Linux: apt install libvulkan-dev vulkan-tools (Ubuntu) or equivalent
• Works on NVIDIA, AMD, Intel, and ARM Mali GPUs

Hexagon (Qualcomm Snapdragon):

• Qualcomm AI Engine Direct SDK
• Android NDK r26+
• Device must have Hexagon DSP (Snapdragon 8 Gen 1+)
• Set env vars: GGML_HEXAGON_NDEV, GGML_HEXAGON_NHVX to configure NPU units

CANN (Huawei Ascend):

• CANN 8.0+ toolkit installed on host
• Ascend NPU driver
• Tested on Atlas 300I/310/910 series

3. Configuration Reference

CMake Build Options (Complete)

Backend Selection

CPU Feature Flags

x86 SIMD

Intel AMX (Advanced Matrix Extensions)

RISC-V and LoongArch

Metal-Specific

Library Build

Example Build Commands

# CPU-only, release build
cmake -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j$(nproc)

# CUDA build
cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_CUDA=ON
cmake --build build --config Release -j$(nproc)

# Metal build (macOS, usually automatic)
cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_METAL=ON -DGGML_METAL_EMBED_LIBRARY=ON
cmake --build build --config Release

# Vulkan build
cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_VULKAN=ON
cmake --build build -j$(nproc)

# ROCm/HIP build (AMD GPU)
cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_HIP=ON -DAMDGPU_TARGETS="gfx1100"
cmake --build build -j$(nproc)

# Static library, no server or examples
cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_STATIC=ON \
    -DLLAMA_BUILD_SERVER=OFF -DLLAMA_BUILD_EXAMPLES=OFF
cmake --build build -j$(nproc)

# All CPU SIMD variants (fat binary for distribution)
cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_CPU_ALL_VARIANTS=ON
cmake --build build -j$(nproc)

Model Parameters (llama_model_params)

Populated by llama_model_default_params() and passed to llama_model_load_from_file().

Context Parameters (llama_context_params)

Populated by llama_context_default_params() and passed to llama_init_from_model().

Quantization Parameters (llama_model_quantize_params)

Sampling Parameters

Sampling controls how the next token is selected from the model's output distribution.

KV Cache Quantization Types

Important

Quantized KV cache types are EXPERIMENTAL. Quality impact varies by model and quantization level. Always validate perplexity when using quantized KV cache in production.

Environment Variables

General

Hexagon NPU (Qualcomm)

CUDA

4. Deployment & Scaling

Single-Machine CPU Deployment

CPU inference is suitable for smaller models (≤13B parameters) or when GPU hardware is unavailable. OpenMP threading provides parallel computation across all cores.

# Recommended CPU settings for generation
./llama-cli \
    --model /models/llama-3-8b-q4_k_m.gguf \
    --threads $(( $(nproc) / 2 )) \
    --threads-batch $(nproc) \
    --ctx-size 4096 \
    --batch-size 512 \
    -p "Your prompt here"

Tip

For CPU inference, setting n_threads to half the physical core count (not hyperthreads) often outperforms using all threads, since memory bandwidth becomes the bottleneck. Experiment with llama-bench to find the optimal value for your hardware.

Thread tuning guidance:

• --threads (generation): Set to physical core count, sometimes half performs better
• --threads-batch (prompt): Set to all logical cores for maximum throughput
• Disable hyperthreading in BIOS for pure inference workloads on server CPUs

Single-GPU Deployment

# Full GPU offload (all layers on GPU)
./llama-server \
    --model /models/llama-3-8b-q4_k_m.gguf \
    --n-gpu-layers -1 \
    --ctx-size 8192 \
    --batch-size 2048 \
    --ubatch-size 512 \
    --port 8080

# Partial GPU offload (model too large for VRAM)
# Offload 20 of 32 layers to GPU
./llama-server \
    --model /models/llama-3-70b-q4_k_m.gguf \
    --n-gpu-layers 20 \
    --threads 8 \
    --ctx-size 4096 \
    --port 8080

Multi-GPU Deployment

Three split modes are available:

LAYER split (default, recommended): Consecutive transformer layers are assigned to different GPUs. GPU 0 handles layers 0–N, GPU 1 handles N+1–M, etc.

./llama-server \
    --model /models/llama-3-70b-q4_k_m.gguf \
    --n-gpu-layers -1 \
    --split-mode layer \
    --tensor-split 0.5,0.5 \
    --ctx-size 8192 \
    --port 8080

ROW split: Tensor rows distributed across GPUs. Good for models where layer boundaries create load imbalance.

./llama-server \
    --model /models/llama-3-70b-q4_k_m.gguf \
    --n-gpu-layers -1 \
    --split-mode row \
    --port 8080

TENSOR split (experimental): Full tensor parallelism across GPUs. Higher communication overhead but potentially better VRAM utilization.

Uneven GPU configurations:

# 70B model on 3× GPUs with different VRAM (24GB, 24GB, 48GB)
./llama-server \
    --model /models/llama-3-70b-q4_k_m.gguf \
    --n-gpu-layers -1 \
    --split-mode layer \
    --tensor-split 0.25,0.25,0.5 \
    --main-gpu 2 \
    --port 8080

RPC Backend (Distributed Inference)

The RPC backend allows distributing model layers across multiple machines connected over a network. Each machine runs an llama-rpc-server process, and the primary machine connects to all of them.

On each worker machine:

# Start the RPC server (listens on all interfaces, port 50052)
./llama-rpc-server --host 0.0.0.0 --port 50052

# Or with specific memory limit
./llama-rpc-server --host 0.0.0.0 --port 50052 --mem 24000

On the primary machine:

./llama-server \
    --model /models/llama-3-70b-q4_k_m.gguf \
    --n-gpu-layers -1 \
    --rpc 192.168.1.10:50052,192.168.1.11:50052 \
    --port 8080

Warning

The RPC backend has no built-in authentication or encryption. Operate only on trusted private networks or VPNs. Never expose RPC ports to the public internet.

Docker Deployment

CPU-only:

FROM ubuntu:22.04

RUN apt-get update && apt-get install -y \
    build-essential cmake git python3 python3-pip \
    && rm -rf /var/lib/apt/lists/*

WORKDIR /app
COPY . .
RUN cmake -B build -DCMAKE_BUILD_TYPE=Release && \
    cmake --build build -j$(nproc)

EXPOSE 8080
CMD ["./build/bin/llama-server", "--model", "/models/model.gguf", \
     "--host", "0.0.0.0", "--port", "8080"]

CUDA (NVIDIA GPU passthrough):

FROM nvidia/cuda:12.2.0-devel-ubuntu22.04

RUN apt-get update && apt-get install -y \
    build-essential cmake git \
    && rm -rf /var/lib/apt/lists/*

WORKDIR /app
COPY . .
RUN cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_CUDA=ON && \
    cmake --build build -j$(nproc)

EXPOSE 8080
CMD ["./build/bin/llama-server", "--model", "/models/model.gguf", \
     "--host", "0.0.0.0", "--port", "8080", "--n-gpu-layers", "-1"]

Run with GPU access:

docker run --gpus all \
    -v /path/to/models:/models \
    -p 8080:8080 \
    llama-cuda

Docker Compose example:

version: '3.8'
services:
  llama-server:
    image: llama-cuda:latest
    runtime: nvidia
    environment:
      - NVIDIA_VISIBLE_DEVICES=all
    volumes:
      - /models:/models:ro
    ports:
      - "8080:8080"
    command: >
      ./build/bin/llama-server
      --model /models/llama-3-8b-q4_k_m.gguf
      --host 0.0.0.0
      --port 8080
      --n-gpu-layers -1
      --ctx-size 8192
      --parallel 4
      --cont-batching
    restart: unless-stopped
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8080/health"]
      interval: 30s
      timeout: 10s
      retries: 3

Server Deployment for Multiple Concurrent Users

./llama-server \
    --model /models/llama-3-8b-q4_k_m.gguf \
    --host 0.0.0.0 \
    --port 8080 \
    --n-gpu-layers -1 \
    --ctx-size 32768 \
    --parallel 8 \
    --cont-batching \
    --batch-size 2048 \
    --ubatch-size 512 \
    --type-k q8_0 \
    --type-v q8_0 \
    --api-key "your-secret-key" \
    --log-file /var/log/llama-server.log

Important

With --parallel N, the total KV cache size becomes N × n_ctx × kv_per_token. For 8 parallel slots at 32K context with F16 KV on Llama 3 8B, this can exceed 16 GB. Use --type-k q8_0 --type-v q8_0 or --n-ctx-cache-auto to stay within VRAM limits.

iOS/macOS Deployment (XCFramework)

llama.cpp provides a pre-built XCFramework for Apple platform integration.

# Build XCFramework from source
cd examples/llama.swiftui
./build-frameworks.sh

# Or download pre-built from GitHub releases
# Integrate in Xcode: File → Add Package Dependencies
# Use local path or GitHub URL

In Swift:

import LlamaFramework

let model = try LlamaModel(path: "/path/to/model.gguf")
let context = try LlamaContext(model: model, params: .default)
let result = try await context.complete(prompt: "Hello")

The examples/llama.swiftui/ directory contains a full SwiftUI app demonstrating model loading, chat UI, and background inference.

Android Deployment (NDK/JNI)

# Build for Android ARM64
cmake -B build-android \
    -DCMAKE_TOOLCHAIN_FILE=$ANDROID_NDK/build/cmake/android.toolchain.cmake \
    -DANDROID_ABI=arm64-v8a \
    -DANDROID_PLATFORM=android-26 \
    -DCMAKE_BUILD_TYPE=Release \
    -DGGML_OPENMP=OFF \
    -DGGML_NATIVE=OFF

cmake --build build-android -j$(nproc)

For Hexagon NPU acceleration on Snapdragon devices:

cmake -B build-android \
    -DCMAKE_TOOLCHAIN_FILE=$ANDROID_NDK/build/cmake/android.toolchain.cmake \
    -DANDROID_ABI=arm64-v8a \
    -DANDROID_PLATFORM=android-26 \
    -DGGML_HEXAGON=ON \
    -DGGML_NATIVE=OFF

cmake --build build-android -j$(nproc)

The examples/llama.android/ directory contains a complete Android Studio project with JNI bindings, model download UI, and chat interface.

CANN (Huawei Ascend) Deployment

# Source CANN environment
source /usr/local/Ascend/ascend-toolkit/set_env.sh

cmake -B build \
    -DCMAKE_BUILD_TYPE=Release \
    -DGGML_CANN=ON

cmake --build build -j$(nproc)

# Run with Ascend NPU
./build/bin/llama-server \
    --model /models/model.gguf \
    --n-gpu-layers -1 \
    --port 8080

WebGPU Deployment

WebGPU support enables browser-based inference (experimental). Build the WASM target:

# Requires Emscripten
emcmake cmake -B build-wasm \
    -DCMAKE_BUILD_TYPE=Release \
    -DGGML_BACKEND_DL=OFF

cmake --build build-wasm -j$(nproc)

5. Operations

Server Lifecycle Management

Starting the server:

# Foreground (development)
./llama-server --model /models/model.gguf --host 0.0.0.0 --port 8080

# Background with logging
nohup ./llama-server \
    --model /models/model.gguf \
    --host 0.0.0.0 \
    --port 8080 \
    --log-file /var/log/llama-server.log \
    2>&1 &

systemd unit file (/etc/systemd/system/llama-server.service):

[Unit]
Description=llama.cpp Inference Server
After=network.target

[Service]
Type=simple
User=llama
Group=llama
WorkingDirectory=/opt/llama
ExecStart=/opt/llama/llama-server \
    --model /models/llama-3-8b-q4_k_m.gguf \
    --host 0.0.0.0 \
    --port 8080 \
    --n-gpu-layers -1 \
    --parallel 4 \
    --cont-batching \
    --api-key "${LLAMA_API_KEY}"
EnvironmentFile=/etc/llama/env
Restart=always
RestartSec=5
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=multi-user.target

sudo systemctl enable llama-server
sudo systemctl start llama-server
sudo systemctl status llama-server

Health Checks

# Basic health check
curl http://localhost:8080/health

# Response when ready:
# {"status":"ok","slots_idle":4,"slots_processing":0}

# Response when loading:
# {"status":"loading model"}

# Response when all slots busy:
# {"status":"no slot available","slots_idle":0,"slots_processing":4}

The /health endpoint returns HTTP 200 when slots are available, HTTP 503 when all slots are busy or the model is loading.

Prometheus Metrics

Enable with --metrics flag:

./llama-server --model /models/model.gguf --metrics --port 8080

Metrics endpoint: GET http://localhost:8080/metrics

Key metrics exposed:

Prometheus scrape config:

scrape_configs:
  - job_name: 'llama-server'
    static_configs:
      - targets: ['localhost:8080']
    metrics_path: '/metrics'

Slot Management

Each slot represents one concurrent inference stream. The server manages slots automatically for --parallel N concurrent requests.

# View slot status
curl http://localhost:8080/slots

# Response shows each slot's state, current prompt, tokens generated, etc.

Slot states: idle, processing

LoRA Adapter Management

LoRA adapters can be loaded at startup or hot-swapped at runtime.

# Load LoRA at startup
./llama-server \
    --model /models/llama-3-8b.gguf \
    --lora /adapters/custom-adapter.gguf \
    --lora-scale 0.8 \
    --port 8080

# Multiple LoRA adapters
./llama-server \
    --model /models/llama-3-8b.gguf \
    --lora /adapters/adapter1.gguf \
    --lora /adapters/adapter2.gguf \
    --port 8080

Hot-swap via API:

# Update LoRA adapters at runtime
curl -X POST http://localhost:8080/lora-adapters \
    -H "Content-Type: application/json" \
    -d '{
        "adapters": [
            {"id": 0, "scale": 1.0}
        ]
    }'

# Remove all LoRA adapters
curl -X DELETE http://localhost:8080/lora-adapters

Memory Monitoring

# NVIDIA GPU memory
watch -n 1 nvidia-smi --query-gpu=memory.used,memory.free,memory.total \
    --format=csv,noheader

# Apple Silicon unified memory
sudo powermetrics --samplers memory_pressure -i 1000 -n 1

# Linux system memory
watch -n 1 free -h

# KV cache usage via API
curl http://localhost:8080/slots | python3 -c "
import json, sys
slots = json.load(sys.stdin)
for s in slots:
    print(f'Slot {s[\"id\"]}: {s[\"n_ctx\"]} tokens context, state={s[\"state\"]}')
"

Model Hot-Reloading

The server does not support in-process model hot-reload. To swap models:

# Graceful restart with new model
kill -SIGTERM $(pgrep llama-server)
# Wait for in-flight requests to complete
sleep 5
./llama-server --model /models/new-model.gguf --port 8080 &

For zero-downtime model swaps, use a load balancer (nginx, HAProxy) with two server instances and perform a blue-green swap.

Session State (GGSN Format)

The CLI supports saving and loading conversation state:

# Save session
./llama-cli --model /models/model.gguf \
    --prompt "..." \
    --prompt-cache /tmp/session.ggsn

# Resume session
./llama-cli --model /models/model.gguf \
    --prompt-cache /tmp/session.ggsn \
    --prompt-cache-ro

Session files use the GGSN format: a binary dump of the KV cache and sampler state.

Backup and Recovery

Model files: GGUF files are immutable after quantization. Back up originals (HuggingFace F16) before quantization; regenerating quantized files from originals is always possible.

Session caches: GGSN files are transient; back up only if long-running conversation state is critical.

LoRA adapters: Treat like model files — immutable binary artifacts to be version-controlled.

Configuration: Store startup scripts and configuration in version control. The server has no mutable runtime configuration state beyond what's in loaded model weights.

6. Performance

Benchmarking with llama-bench

llama-bench provides standardized performance measurement for comparing configurations.

# Standard benchmark (pp128 = prompt processing 128 tokens, tg128 = token generation 128 tokens)
./llama-bench \
    --model /models/llama-3-8b-q4_k_m.gguf \
    --n-gpu-layers -1 \
    -pp 128 -tg 128

# Multiple test sizes
./llama-bench \
    --model /models/llama-3-8b-q4_k_m.gguf \
    --n-gpu-layers -1 \
    -pp 128,512,2048 -tg 128,512

# JSON output for automated processing
./llama-bench \
    --model /models/llama-3-8b-q4_k_m.gguf \
    --n-gpu-layers -1 \
    --output json \
    > results.json

# Multi-GPU benchmark
./llama-bench \
    --model /models/llama-3-70b-q4_k_m.gguf \
    --n-gpu-layers -1 \
    --split-mode layer \
    --tensor-split 0.5,0.5 \
    -pp 512 -tg 128

Key metrics reported:

• t/s (prompt): Tokens per second during prompt processing (prefill)
• t/s (generation): Tokens per second during token generation (decode)
• ms/token: Latency per generated token

Perplexity Measurement

Perplexity quantifies model quality — lower is better. Use to validate quantization doesn't degrade model quality beyond acceptable thresholds.

# Measure perplexity on standard corpus
./llama-perplexity \
    --model /models/llama-3-8b-q4_k_m.gguf \
    --n-gpu-layers -1 \
    --file wikitext-2-raw/wiki.test.raw \
    --ctx-size 512 \
    --batch-size 512

# Compare quantization levels
for quant in q2_k q3_k_m q4_k_m q5_k_m q6_k q8_0; do
    echo "=== $quant ==="
    ./llama-perplexity \
        --model /models/llama-3-8b-${quant}.gguf \
        --n-gpu-layers -1 \
        --file wiki.test.raw \
        --ctx-size 512
done

Importance Matrix Generation

The imatrix improves quantization quality by weighting quantization error by activation importance. Always generate before quantizing below Q4.

# Generate importance matrix
./llama-imatrix \
    --model /models/llama-3-8b-f16.gguf \
    --n-gpu-layers -1 \
    --training-data groups_merged.txt \
    --output /models/llama-3-8b-imatrix.dat \
    --ctx-size 512 \
    --batch-size 512 \
    -ngl 99

# Quantize using importance matrix
./llama-quantize \
    --imatrix /models/llama-3-8b-imatrix.dat \
    /models/llama-3-8b-f16.gguf \
    /models/llama-3-8b-iq4_xs.gguf \
    IQ4_XS

KV Cache Optimization

Quantized KV cache reduces memory usage allowing larger context windows:

# Q8_0 KV cache: 50% memory savings, minimal quality loss
./llama-server \
    --model /models/model.gguf \
    --n-gpu-layers -1 \
    --type-k q8_0 --type-v q8_0 \
    --ctx-size 32768 \
    --port 8080

# Q4_0 KV cache: 75% savings, moderate quality impact
./llama-server \
    --model /models/model.gguf \
    --n-gpu-layers -1 \
    --type-k q4_0 --type-v q4_0 \
    --ctx-size 65536 \
    --port 8080

Caution

KV cache quantization below Q8_0 can visibly degrade coherence on long contexts. Always validate with perplexity measurements and qualitative testing before deploying Q4 KV cache in production.

Flash Attention

Flash attention reduces memory usage for long contexts by computing attention in tiles without materializing the full attention matrix.

# Enable flash attention (reduces VRAM for long contexts)
./llama-server \
    --model /models/model.gguf \
    --n-gpu-layers -1 \
    --flash-attn \
    --ctx-size 32768 \
    --port 8080

# Flash attention is AUTO by default on supported backends
# Explicitly disable if you suspect compatibility issues:
./llama-server --no-flash-attn ...

Flash attention is supported on CUDA, Metal, and some Vulkan configurations. CPU flash attention is available for completeness but provides no memory benefit.

Thread Tuning Guide

# Benchmark thread configurations
for t in 1 2 4 8 16; do
    echo -n "threads=$t: "
    ./llama-bench --model /models/model.gguf --threads $t -tg 128 \
        | grep "tg" | awk '{print $NF, "t/s"}'
done

Batch Size Optimization

For multi-user server mode with continuous batching, larger n_batch allows the server to group more user requests together, improving GPU utilization:

# High-throughput server
./llama-server \
    --batch-size 4096 \
    --ubatch-size 1024 \
    --parallel 8 \
    --cont-batching \
    ...

Speculative Decoding

Speculative decoding uses a small "draft" model to propose multiple tokens, which the large "target" model verifies in a single forward pass. Provides 2–4× speedup on generation-bound workloads.

./llama-server \
    --model /models/llama-3-70b-q4_k_m.gguf \
    --n-gpu-layers -1 \
    --draft-model /models/llama-3-8b-q4_k_m.gguf \
    --draft-n-gpu-layers -1 \
    --draft-max 8 \
    --port 8080

Tip

Speculative decoding works best when the draft and target models share a tokenizer and training distribution. Using a smaller model from the same model family (e.g., Llama 3 8B drafting for Llama 3 70B) gives the highest acceptance rates.

Lookahead Decoding

Lookahead decoding provides speculative acceleration without a draft model, using n-gram caching from the current context.

./llama-cli \
    --model /models/model.gguf \
    --n-gpu-layers -1 \
    --lookup-cache-static /path/to/ngram-cache.bin \
    -p "Your prompt"

GPU Offloading Strategy

Optimal layer distribution when model doesn't fully fit in VRAM:

1. First determine how many layers fit: run with -ngl 999 and watch for OOM
2. If OOM, reduce by 5 layers at a time
3. The output tensor (lm_head) and input embeddings are included in the layer count with -ngl -1

# Find maximum layers for your VRAM
./llama-bench \
    --model /models/model.gguf \
    --n-gpu-layers 20 -tg 32
# Increment until OOM, then back off by 2-3 layers for safety margin

For generation-latency-sensitive workloads (chatbots), prefer fewer GPU layers with full model in VRAM over partial offload. For throughput-sensitive workloads (batch processing), partial offload with high batch size may be acceptable.

Performance Tuning Checklist

• [ ] Use quantized model appropriate for available VRAM (Q4_K_M is the general-purpose recommendation)
• [ ] Set --n-gpu-layers -1 if model fits in VRAM
• [ ] Enable --flash-attn for contexts > 8K tokens
• [ ] Set --cont-batching for multi-user server deployments
• [ ] Tune --threads / --threads-batch with llama-bench
• [ ] Use --type-k q8_0 --type-v q8_0 if context window is VRAM-limited
• [ ] Generate imatrix before quantizing below Q4 quantization level
• [ ] Enable --mlock on servers with sufficient RAM to prevent swapping
• [ ] Set --parallel based on expected concurrent users and available VRAM
• [ ] For CPU inference: disable hyperthreading, enable performance governor
• [ ] Validate with llama-bench before and after configuration changes

7. Security

API Key Authentication

The server supports a single static API key for access control.

./llama-server \
    --model /models/model.gguf \
    --api-key "your-secret-key-here" \
    --port 8080

Clients must include the key in requests:

# Authorization header (Bearer)
curl http://localhost:8080/v1/completions \
    -H "Authorization: Bearer your-secret-key-here" \
    -H "Content-Type: application/json" \
    -d '{"prompt": "Hello", "max_tokens": 50}'

# X-API-Key header (alternative)
curl http://localhost:8080/v1/completions \
    -H "X-API-Key: your-secret-key-here" \
    -H "Content-Type: application/json" \
    -d '{"prompt": "Hello", "max_tokens": 50}'

Warning

The server does not support multiple API keys or per-key rate limiting natively. For production deployments requiring multi-tenant access control, place a reverse proxy (nginx, Caddy, Kong) in front of the server.

TLS/HTTPS

llama-server has no built-in TLS. Always use a reverse proxy for production.

nginx reverse proxy example:

server {
    listen 443 ssl;
    server_name llama.example.com;

    ssl_certificate /etc/letsencrypt/live/llama.example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/llama.example.com/privkey.pem;
    ssl_protocols TLSv1.2 TLSv1.3;

    location / {
        proxy_pass http://localhost:8080;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_read_timeout 300s;
        proxy_send_timeout 300s;
    }
}

Input Validation

The server validates:

• max_tokens and n_predict: capped at context window size
• Prompt length: rejected if longer than n_ctx
• JSON schema: malformed requests return 400

Administrators should additionally:

• Set --ctx-size to the minimum needed (limits resource use per request)
• Use --parallel to limit concurrent load
• Implement request rate limiting at the reverse proxy layer

Model File Integrity

# Enable tensor validation during model load
./llama-server \
    --model /models/model.gguf \
    --check-tensors \
    --port 8080

# Verify GGUF file is not corrupt before deployment
./llama-cli \
    --model /models/model.gguf \
    --check-tensors \
    --vocab-only \
    -p ""

Note

--check-tensors increases model load time significantly for large models (adds minutes for 70B models). Use for initial validation after download, not in routine startup.

Security Policy

The project maintains a SECURITY.md file at the repository root documenting the vulnerability disclosure process. Security issues should be reported privately to the maintainers rather than via public GitHub issues.

Key security considerations for operators:

• No prompt sanitization: the server passes prompts directly to the model; jailbreak attempts are a model-level concern, not a server-level one
• File system access: the server reads model files with the permissions of its process user; run as a dedicated non-root user
• GPU memory isolation: GPU memory is not zeroed between requests; on multi-user systems, one user's context is theoretically accessible to another through GPU memory forensics (not a practical concern for most deployments)
• Log files: avoid logging full prompt content if handling sensitive user data

MCP (Model Context Protocol) Security Considerations

The web UI supports connecting to MCP servers, which allows the LLM to execute tools and access external data sources. This significantly expands the attack surface:

Caution

MCP tool execution means the model can interact with external systems. In multi-user deployments, an adversarial user could craft prompts to abuse MCP tools. Only enable MCP with trusted MCP servers on trusted networks. Implement MCP server-side authorization and audit logging.

• Restrict MCP server access to specific tool names where possible
• Audit all MCP tool calls in server logs
• Never run MCP servers with elevated privileges
• Consider disabling MCP in the web UI for public-facing deployments

8. Developer Internals

Repository Structure

llama.cpp/
├── include/                    # Public API headers
│   ├── llama.h                 # Primary C API (stable public interface)
│   └── llama-cpp.h             # C++ RAII wrappers and smart pointers
├── src/                        # Core llama library implementation
│   ├── llama.cpp               # Main library implementation file
│   ├── llama-model.cpp/.h      # Model loading and architecture dispatch
│   ├── llama-context.cpp/.h    # Inference context management
│   ├── llama-sampling.cpp/.h   # Sampler chain implementation
│   ├── llama-memory.cpp/.h     # Memory management (KV cache, buffers)
│   ├── llama-kv-cache.cpp/.h   # KV cache implementation
│   ├── llama-graph.cpp/.h      # Computation graph construction
│   ├── llama-adapter.cpp/.h    # LoRA adapter support
│   ├── llama-mmap.cpp/.h       # Memory-mapped file I/O
│   ├── llama-batch.cpp/.h      # Batch management
│   ├── llama-vocab.cpp/.h      # Vocabulary and tokenization
│   ├── llama-chat.cpp/.h       # Chat template utilities
│   ├── llama-arch.cpp/.h       # Architecture definitions and registry
│   └── models/                 # Model architecture implementations (114 files)
│       ├── llama.cpp            # LLaMA / LLaMA 2 / LLaMA 3
│       ├── mistral.cpp          # Mistral / Mixtral
│       ├── gemma.cpp            # Google Gemma
│       ├── phi.cpp              # Microsoft Phi
│       ├── qwen.cpp             # Alibaba Qwen
│       ├── deepseek.cpp         # DeepSeek
│       ├── command-r.cpp        # Cohere Command R
│       ├── bloom.cpp            # BigScience BLOOM
│       ├── falcon.cpp           # Technology Innovation Institute Falcon
│       ├── starcoder.cpp        # BigCode StarCoder
│       └── ... (100+ more)
├── ggml/                       # GGML tensor library
│   ├── include/
│   │   ├── ggml.h              # Core tensor operations API
│   │   ├── ggml-backend.h      # Backend abstraction interface
│   │   ├── ggml-alloc.h        # Memory allocator
│   │   └── gguf.h              # GGUF format reader/writer
│   └── src/
│       ├── ggml.c              # Core implementation (quants, ops)
│       ├── ggml-backend.cpp    # Backend registry and routing
│       ├── ggml-alloc.c        # Allocator implementation
│       ├── gguf.cpp            # GGUF reader/writer implementation
│       ├── ggml-quants.c       # Quantization type implementations
│       ├── ggml-threading.cpp  # Thread pool
│       ├── ggml-cpu/           # CPU backend (SIMD kernels)
│       ├── ggml-cuda/          # CUDA backend
│       ├── ggml-metal/         # Metal backend (+ .metal shaders)
│       ├── ggml-vulkan/        # Vulkan backend (+ .comp shaders)
│       ├── ggml-opencl/        # OpenCL backend
│       ├── ggml-sycl/          # SYCL/oneAPI backend
│       ├── ggml-hip/           # AMD ROCm/HIP backend
│       ├── ggml-cann/          # Huawei CANN backend
│       ├── ggml-hexagon/       # Qualcomm Hexagon NPU backend
│       ├── ggml-musa/          # Moore Threads MUSA backend
│       ├── ggml-blas/          # BLAS backend
│       ├── ggml-rpc/           # RPC backend
│       └── ggml-kompute/       # Vulkan compute (Kompute)
├── common/                     # Shared utilities
│   ├── arg.cpp/.h              # Command-line argument parsing
│   ├── chat.cpp/.h             # Chat session management
│   ├── sampling.cpp/.h         # Sampler chain construction
│   ├── download.cpp/.h         # Model download from HuggingFace
│   ├── log.cpp/.h              # Logging subsystem
│   ├── ngram-cache.cpp/.h      # N-gram cache for lookahead decoding
│   ├── speculative.cpp/.h      # Speculative decoding utilities
│   └── jinja/                  # Jinja2-compatible template engine
│       └── jinja.hpp           # Single-header implementation
├── tools/                      # Executable tools
│   ├── cli/                    # llama-cli (interactive chat)
│   ├── server/                 # llama-server (HTTP API)
│   │   ├── server.cpp          # HTTP server implementation
│   │   ├── utils.hpp           # Server utilities
│   │   └── webui/             # SvelteKit web interface
│   │       ├── src/            # TypeScript/Svelte source
│   │       ├── tests/          # Vitest + Playwright tests
│   │       └── package.json
│   ├── quantize/               # llama-quantize
│   ├── llama-bench/            # Performance benchmarking
│   ├── perplexity/             # Perplexity measurement
│   ├── imatrix/                # Importance matrix
│   ├── tts/                    # Text-to-speech
│   ├── mtmd/                   # Multimodal (vision/audio)
│   ├── tokenize/               # Tokenization utility
│   ├── gguf-split/             # GGUF file split/merge
│   ├── export-lora/            # LoRA export
│   ├── cvector-generator/      # Control vector generation
│   ├── parser/                 # PEG-based grammar parser
│   ├── rpc/                    # RPC server
│   ├── batched-bench/          # Batched benchmarking
│   ├── completion/             # Non-interactive completion
│   └── results/                # Benchmark result processing
├── examples/                   # Integration examples
│   ├── llama.android/          # Android app (Kotlin/JNI)
│   ├── llama.swiftui/          # iOS/macOS app (Swift/SwiftUI)
│   └── ...
├── gguf-py/                    # Python GGUF library
│   └── gguf/
│       ├── gguf_reader.py      # GGUF file reading
│       ├── gguf_writer.py      # GGUF file writing
│       └── constants.py        # Type definitions, enum mappings
├── tests/                      # Test suite
│   ├── test-*.cpp              # C++ unit tests
│   └── peg-parser/             # PEG parser-specific tests
├── cmake/                      # CMake modules
│   ├── FindSIMD.cmake          # CPU feature detection
│   └── llama-config.cmake.in   # Package config template
├── grammars/                   # GBNF grammar files
│   ├── json.gbnf               # JSON output constraint
│   ├── json_arr.gbnf           # JSON array output
│   └── ...
├── docs/                       # Documentation
│   ├── build/                  # Build guides per backend
│   └── ops/                    # Supported ops CSV per backend
├── ci/
│   └── run.sh                  # CI runner script
├── convert_hf_to_gguf.py       # HuggingFace → GGUF converter
├── convert_hf_to_gguf_update.py # Update architecture support
├── .clang-format               # C/C++ formatter config
├── .clang-tidy                 # C/C++ linter config
├── .editorconfig               # Editor settings
├── .pre-commit-config.yaml     # Pre-commit hooks
├── CODEOWNERS                  # Review routing
├── CMakeLists.txt              # Root build file
└── SECURITY.md                 # Vulnerability disclosure policy

Key Internal APIs

Model Lifecycle

// Load a model from a GGUF file
struct llama_model * llama_model_load_from_file(
    const char * path_model,
    const struct llama_model_params params);

// Free model and all associated resources
void llama_model_free(struct llama_model * model);

// Create inference context from loaded model
struct llama_context * llama_init_from_model(
    struct llama_model * model,
    struct llama_context_params params);

// Free context (does not free the model)
void llama_free(struct llama_context * ctx);

Core Inference

// Run the transformer forward pass for a batch of tokens
// Returns 0 on success, negative on error
int32_t llama_decode(
    struct llama_context * ctx,
    struct llama_batch batch);

// For encoder-decoder models (T5, BART)
int32_t llama_encode(
    struct llama_context * ctx,
    struct llama_batch batch);

// Get logits for the last decoded token
float * llama_get_logits(struct llama_context * ctx);

// Get logits for a specific batch position
float * llama_get_logits_ith(struct llama_context * ctx, int32_t i);

// Get embeddings (when context has embeddings=true)
float * llama_get_embeddings(struct llama_context * ctx);
float * llama_get_embeddings_ith(struct llama_context * ctx, int32_t i);
float * llama_get_embeddings_seq(struct llama_context * ctx, llama_seq_id seq_id);

Batch Management

// Allocate a batch (n_tokens capacity, n_seq_max sequences)
struct llama_batch llama_batch_init(int32_t n_tokens, int32_t embd, int32_t n_seq_max);

// Free batch resources
void llama_batch_free(struct llama_batch batch);

// llama_batch structure:
struct llama_batch {
    int32_t      n_tokens;       // number of tokens in this batch
    llama_token * token;         // token IDs (NULL if using embeddings)
    float *      embd;           // embeddings input (NULL if using tokens)
    llama_pos *  pos;            // position for each token
    int32_t *    n_seq_id;       // number of sequences for each token
    llama_seq_id ** seq_id;      // sequence IDs for each token
    int8_t *     logits;         // 1 = compute logits for this position
};

Sampler Chain

// Initialize a sampler chain
struct llama_sampler * llama_sampler_chain_init(
    struct llama_sampler_chain_params params);

// Add a sampler to the chain (executed in order)
void llama_sampler_chain_add(
    struct llama_sampler * chain,
    struct llama_sampler * smpl);

// Sample next token from logits
llama_token llama_sampler_chain_sample(
    struct llama_sampler * chain,
    struct llama_context * ctx,
    int32_t idx);

// Built-in sampler constructors:
struct llama_sampler * llama_sampler_init_top_k(int32_t k);
struct llama_sampler * llama_sampler_init_top_p(float p, size_t min_keep);
struct llama_sampler * llama_sampler_init_min_p(float p, size_t min_keep);
struct llama_sampler * llama_sampler_init_typical(float p, size_t min_keep);
struct llama_sampler * llama_sampler_init_temp(float t);
struct llama_sampler * llama_sampler_init_dist(uint32_t seed);
struct llama_sampler * llama_sampler_init_mirostat(int32_t n_vocab, uint32_t seed,
                                                    float tau, float eta, int32_t m);
struct llama_sampler * llama_sampler_init_mirostat_v2(uint32_t seed, float tau, float eta);
struct llama_sampler * llama_sampler_init_grammar(const struct llama_vocab * vocab,
                                                   const char * grammar_str,
                                                   const char * grammar_root);
struct llama_sampler * llama_sampler_init_penalties(int32_t penalty_last_n,
                                                     float penalty_repeat,
                                                     float penalty_freq,
                                                     float penalty_present);
struct llama_sampler * llama_sampler_init_xtc(float p, float t,
                                               size_t min_keep, uint32_t seed);

// Free individual sampler
void llama_sampler_free(struct llama_sampler * smpl);

Memory Management

// Copy sequence state (for forking conversations)
bool llama_memory_seq_cp(
    struct llama_memory_state * ms,
    llama_seq_id seq_id_src,
    llama_seq_id seq_id_dst,
    llama_pos p0,
    llama_pos p1);

// Remove tokens from a sequence
bool llama_memory_seq_rm(
    struct llama_memory_state * ms,
    llama_seq_id seq_id,
    llama_pos p0,
    llama_pos p1);

// Get memory state from context
struct llama_memory_state * llama_get_memory(struct llama_context * ctx);

C++ Smart Pointers (llama-cpp.h)

#include "llama-cpp.h"

// RAII wrapper types
using llama_model_ptr   = std::unique_ptr<llama_model,   llama_model_deleter>;
using llama_context_ptr = std::unique_ptr<llama_context, llama_context_deleter>;
using llama_sampler_ptr = std::unique_ptr<llama_sampler, llama_sampler_deleter>;

// Usage
auto model   = llama_model_ptr(llama_model_load_from_file(path, params));
auto context = llama_context_ptr(llama_init_from_model(model.get(), ctx_params));
// Automatically freed when going out of scope

Tokenization

// Tokenize text to token IDs
int32_t llama_tokenize(
    const struct llama_vocab * vocab,
    const char * text,
    int32_t text_len,
    llama_token * tokens,
    int32_t n_tokens_max,
    bool add_special,
    bool parse_special);

// Convert token ID back to text piece
int32_t llama_token_to_piece(
    const struct llama_vocab * vocab,
    llama_token token,
    char * buf,
    int32_t length,
    int32_t lstrip,
    bool special);

// Get vocabulary from model
const struct llama_vocab * llama_model_get_vocab(const struct llama_model * model);

// Special tokens
llama_token llama_vocab_bos(const struct llama_vocab * vocab);
llama_token llama_vocab_eos(const struct llama_vocab * vocab);
llama_token llama_vocab_pad(const struct llama_vocab * vocab);

Model Architecture Support

The src/models/ directory contains 114 files implementing model architectures. Each file implements the forward pass computation graph for that architecture using the GGML graph builder. Architectures are registered in llama-arch.cpp via an architecture registry keyed by the general.architecture GGUF metadata field.

Architecture dispatch flow:

llama_model_load()
  → read "general.architecture" from GGUF metadata
  → look up architecture in registry (llama-arch.cpp)
  → call architecture-specific model builder
  → llama_model_build_graph() called at inference time
  → returns ggml_cgraph* for GGML to execute

Supported architecture families include (non-exhaustive):

GGUF Format Internals

GGUF (GGML Universal File) is the binary format used for all models in llama.cpp.

File structure:

┌─────────────────────────────────────┐
│ Magic: "GGUF" (4 bytes)             │
│ Version: uint32 (current = 3)       │
│ n_tensors: uint64                   │
│ n_kv: uint64                        │
├─────────────────────────────────────┤
│ Key-Value Metadata                  │
│  key: string                        │
│  type: gguf_metadata_value_type     │
│  value: (type-dependent)            │
│  ... × n_kv                        │
├─────────────────────────────────────┤
│ Tensor Descriptors                  │
│  name: string                       │
│  n_dims: uint32                     │
│  dims: uint64 × n_dims              │
│  type: ggml_type                    │
│  offset: uint64 (from data start)   │
│  ... × n_tensors                    │
├─────────────────────────────────────┤
│ Alignment padding                   │
├─────────────────────────────────────┤
│ Tensor Data                         │
│  (binary, aligned to               │
│   GGUF_DEFAULT_ALIGNMENT = 32 bytes)│
└─────────────────────────────────────┘

Standard GGUF metadata keys:

Python library usage:

from gguf import GGUFReader, GGUFWriter, GGMLQuantizationType

# Read a GGUF file
reader = GGUFReader("model.gguf")
for key, field in reader.fields.items():
    print(f"{key}: {field.data}")

for tensor in reader.tensors:
    print(f"{tensor.name}: shape={tensor.shape}, type={tensor.tensor_type}")

# Write a GGUF file
writer = GGUFWriter("output.gguf", arch="llama")
writer.add_name("My Model")
writer.add_context_length(4096)
writer.add_embedding_length(4096)
writer.add_tensor("token_embd.weight", tensor_data,
                  raw_dtype=GGMLQuantizationType.Q4_K)
writer.write_header_to_file()
writer.write_kv_data_to_file()
writer.write_tensors_to_file(progress=True)
writer.close()

Quantization type registry (GGMLQuantizationType enum, 40+ types):

IQ* types use importance-matrix-guided quantization and require an imatrix file to generate but can be loaded and run without it.

Chat Template System

The chat template system renders conversation history into the exact prompt format a model expects. Templates are stored in GGUF metadata under tokenizer.chat_template as Jinja2-format strings.

Template auto-detection flow:

llama_chat_apply_template()
  → if explicit template string: use it
  → else: read "tokenizer.chat_template" from model GGUF
  → if not found: detect from known model architectures
  → render messages array using Jinja2-compatible engine

Jinja2 engine in common/jinja/:

The engine supports the subset of Jinja2 used in chat templates:

• Variable substitution: {{ variable }}
• Control flow: {% if %}, {% for %}, {% set %}
• Filters: | lower, | upper, | trim, | join, | tojson
• Special variables: messages, tools, add_generation_prompt

Tool calling support:

Models supporting tool calling use the template system to serialize tool definitions and parse tool call responses. The tools variable in templates contains the JSON-serialized tool definitions.

Reasoning format support:

Models with chain-of-thought reasoning (DeepSeek R1, QwQ) use templates that include <think> block handling. The template engine correctly handles these extended formats.

Structured output (PEG parser):

tools/parser/ implements a PEG (Parsing Expression Grammar) based parser for constrained output generation. This is distinct from GBNF grammars — it provides higher-level structured parsing for JSON extraction, tool call parsing, and similar tasks.

Backend System Architecture

The GGML backend system provides hardware abstraction. Every hardware backend implements the same interface, allowing the scheduler to dispatch operations to the appropriate hardware.

Backend interface (simplified):

struct ggml_backend_i {
    const char * (*get_name)(ggml_backend_t backend);
    void         (*free)(ggml_backend_t backend);
    void         (*set_tensor_async)(ggml_backend_t backend,
                                     struct ggml_tensor * tensor,
                                     const void * data, size_t offset, size_t size);
    void         (*get_tensor_async)(ggml_backend_t backend,
                                     const struct ggml_tensor * tensor,
                                     void * data, size_t offset, size_t size);
    bool         (*cpy_tensor_async)(ggml_backend_t backend_src,
                                     ggml_backend_t backend_dst,
                                     const struct ggml_tensor * src,
                                     struct ggml_tensor * dst);
    void         (*synchronize)(ggml_backend_t backend);
    enum ggml_status (*graph_compute)(ggml_backend_t backend,
                                      struct ggml_cgraph * cgraph);
    bool         (*supports_op)(ggml_backend_dev_t dev,
                                const struct ggml_tensor * op);
    bool         (*supports_buft)(ggml_backend_dev_t dev,
                                  ggml_backend_buffer_type_t buft);
    bool         (*offload_op)(ggml_backend_dev_t dev,
                               const struct ggml_tensor * op);
};

Backend registration:

// Static registration (compile-time)
ggml_backend_t ggml_backend_cuda_init(int device);
ggml_backend_t ggml_backend_metal_init(void);
ggml_backend_t ggml_backend_vulkan_init(size_t dev_num);

// Dynamic loading (GGML_BACKEND_DL=ON)
void ggml_add_backend_library(const char * path);

Backend scheduler:

The ggml_backend_sched distributes computation graph nodes across available backends. Nodes are assigned based on:

1. Which backend owns the output buffer
2. Which backends support the operation
3. Whether peer-to-peer data transfer is available

Supported ops per backend:

Documented in docs/ops/ as CSV files listing which GGML operations each backend supports. This is the authoritative reference for backend capability limitations.

CPU feature detection:

The CPU backend uses CPUID instruction probing at startup to detect available SIMD features and selects optimal kernel variants for each operation. When GGML_CPU_ALL_VARIANTS=ON, multiple kernel variants are compiled in and selected at runtime.

Code Patterns and Conventions

Opaque pointer pattern:

Public headers declare structs without definitions:

// In llama.h (public):
struct llama_model;
struct llama_context;

// In src/llama.cpp (private):
struct llama_model { /* full definition */ };
struct llama_context { /* full definition */ };

This provides ABI stability — the struct layout can change without breaking client code compiled against old headers.

C API with C++ internals:

The public API is pure C (llama.h), making it linkable from any language. Internally, the implementation is C++17. The llama-cpp.h header provides idiomatic C++ RAII wrappers for applications that prefer them.

Atomic file operations:

Model loading uses atomic operations to prevent partial reads from a concurrent write. The GGUF reader validates the magic bytes and file structure before committing to loading.

Thread pool:

ggml-threading.cpp implements a work-stealing thread pool. Each backend can request compute work be distributed across the pool. The pool size is controlled by n_threads / n_threads_batch context parameters.

Testing Infrastructure

C++ Unit Tests

Located in tests/. Built when LLAMA_BUILD_TESTS=ON (default).

# Build and run all tests
cmake -B build -DLLAMA_BUILD_TESTS=ON
cmake --build build --target test
# or
cd build && ctest --output-on-failure

# Run specific test
./build/tests/test-tokenizer-0 /path/to/model.gguf
./build/tests/test-sampling
./build/tests/test-chat-template

Key test files:

• test-tokenizer-0.cpp: BPE tokenizer correctness
• test-tokenizer-1-bpe.cpp: Additional BPE tests
• test-sampling.cpp: Sampler chain behavior
• test-chat-template.cpp: Jinja template rendering
• test-gguf.cpp: GGUF read/write correctness
• test-backend-ops.cpp: Backend operation correctness (requires built backend)

PEG Parser Tests

cd tests/peg-parser
npm install
npm test

Server Integration Tests

Python test suite using aiohttp:

cd tools/server/tests
pip install -r requirements.txt

# Run tests (requires running server)
./run_tests.sh

# Or manually
python -m pytest test_completion.py -v
python -m pytest test_chat_completion.py -v
python -m pytest test_embedding.py -v

Web UI Tests

cd tools/server/webui
npm install

# Unit tests (Vitest)
npm run test

# E2E tests (Playwright)
npm run test:e2e

# Component tests (Storybook)
npm run storybook
npm run test:storybook

CI Pipeline

ci/run.sh orchestrates multi-backend test runs. CI validates:

• CPU build and test suite
• CUDA build (on CUDA-equipped runners)
• Metal build (on macOS runners)
• Vulkan build
• Python conversion scripts
• Server integration tests
• Tokenizer correctness

Contributing

Code style:

# Install pre-commit hooks
pip install pre-commit
pre-commit install

# Run manually
pre-commit run --all-files

# Format C/C++ code
clang-format -i src/*.cpp include/*.h
clang-tidy src/*.cpp -- -I include

Configuration files:

• .clang-format: Google style base with modifications
• .clang-tidy: Warning set for C++ best practices
• .editorconfig: 4-space indent, LF line endings, trim trailing whitespace

CODEOWNERS routes PR reviews to backend-specific maintainers:

• ggml/src/ggml-cuda/ → CUDA backend maintainers
• ggml/src/ggml-metal/ → Metal maintainers
• ggml/src/ggml-vulkan/ → Vulkan maintainers
• examples/llama.android/ → Android maintainers
• examples/llama.swiftui/ → iOS/macOS maintainers

Adding a new model architecture:

1. Add architecture name to llama_arch enum in src/llama-arch.h
2. Register architecture string mapping in src/llama-arch.cpp
3. Create src/models/{arch}.cpp implementing llm_build_{arch}() graph builder
4. Add model-specific metadata keys in src/llama-arch.h
5. Add conversion support in convert_hf_to_gguf.py (add model class)
6. Add test in tests/ for tokenizer and/or architecture

Adding a new backend:

1. Create ggml/src/ggml-{backend}/ directory
2. Implement ggml_backend_i interface
3. Add CMake option GGML_{BACKEND} in ggml/CMakeLists.txt
4. Register backend in ggml/src/ggml-backend-reg.cpp
5. Document supported ops in docs/ops/{backend}.csv
6. Add build documentation in docs/build/

9. Integration Points

C API Integration

The stable public C API is the recommended integration point for all external consumers.

#include "llama.h"

// Minimal working example
int main() {
    llama_backend_init();

    struct llama_model_params model_params = llama_model_default_params();
    model_params.n_gpu_layers = 99;

    struct llama_model * model =
        llama_model_load_from_file("model.gguf", model_params);

    struct llama_context_params ctx_params = llama_context_default_params();
    ctx_params.n_ctx = 4096;

    struct llama_context * ctx = llama_init_from_model(model, ctx_params);

    // Tokenize
    const char * prompt =
        "<|im_start|>user\nHello!<|im_end|>\n<|im_start|>assistant\n";
    int n_tokens = llama_tokenize(llama_model_get_vocab(model),
                                  prompt, strlen(prompt),
                                  NULL, 0, true, true);
    llama_token * tokens = malloc(n_tokens * sizeof(llama_token));
    llama_tokenize(llama_model_get_vocab(model),
                   prompt, strlen(prompt),
                   tokens, n_tokens, true, true);

    // Create batch and decode
    struct llama_batch batch = llama_batch_init(n_tokens, 0, 1);
    for (int i = 0; i < n_tokens; i++) {
        batch.token[i] = tokens[i];
        batch.pos[i] = i;
        batch.n_seq_id[i] = 1;
        batch.seq_id[i][0] = 0;
        batch.logits[i] = (i == n_tokens - 1);
    }
    batch.n_tokens = n_tokens;
    llama_decode(ctx, batch);

    // Sample
    struct llama_sampler * sampler = llama_sampler_chain_init(
        llama_sampler_chain_default_params());
    llama_sampler_chain_add(sampler, llama_sampler_init_temp(0.8));
    llama_sampler_chain_add(sampler, llama_sampler_init_dist(42));

    llama_token next = llama_sampler_chain_sample(sampler, ctx, n_tokens - 1);

    // Cleanup
    llama_sampler_free(sampler);
    llama_batch_free(batch);
    llama_free(ctx);
    llama_model_free(model);
    llama_backend_free();
    free(tokens);
    return 0;
}

Linking with CMake:

find_package(llama REQUIRED)
target_link_libraries(myapp PRIVATE llama)

Or with pkg-config:

gcc -o myapp myapp.c $(pkg-config --cflags --libs llama)

C++ RAII API

#include "llama-cpp.h"
#include "llama.h"

int main() {
    llama_backend_init();

    auto model_params = llama_model_default_params();
    model_params.n_gpu_layers = -1;

    auto model = llama_model_ptr(
        llama_model_load_from_file("model.gguf", model_params));

    auto ctx_params = llama_context_default_params();
    ctx_params.n_ctx = 4096;

    auto ctx = llama_context_ptr(
        llama_init_from_model(model.get(), ctx_params));

    auto sampler = llama_sampler_ptr(
        llama_sampler_chain_init(llama_sampler_chain_default_params()));
    llama_sampler_chain_add(sampler.get(), llama_sampler_init_temp(0.8f));
    llama_sampler_chain_add(sampler.get(), llama_sampler_init_dist(42));

    // model, ctx, sampler all freed automatically at scope exit
    llama_backend_free();
}

OpenAI-Compatible REST API

The server implements the OpenAI API specification for drop-in compatibility with existing OpenAI API clients.

Base URL: http://localhost:8080

Endpoints:

Chat completions example:

import openai

client = openai.OpenAI(
    base_url="http://localhost:8080/v1",
    api_key="your-api-key"
)

response = client.chat.completions.create(
    model="local-model",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "What is the capital of France?"}
    ],
    temperature=0.7,
    max_tokens=200,
    stream=True
)

for chunk in response:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)

llama.cpp-specific request extensions:

{
  "prompt": "...",
  "n_predict": 200,
  "temperature": 0.8,
  "top_k": 40,
  "top_p": 0.95,
  "min_p": 0.05,
  "repeat_penalty": 1.1,
  "repeat_last_n": 64,
  "grammar": "root ::= [a-z]+",
  "json_schema": {"type": "object", "properties": {}},
  "cache_prompt": true,
  "slot_id": -1,
  "mirostat": 0,
  "mirostat_tau": 5.0,
  "mirostat_eta": 0.1,
  "seed": -1,
  "ignore_eos": false,
  "stream": true
}

Python / gguf-py

# Install
pip install gguf

# Inspect a GGUF file
from gguf import GGUFReader

reader = GGUFReader("model.gguf", "r")
print(f"Architecture: {reader.fields['general.architecture'].data}")
print(f"Context length: {reader.fields['llama.context_length'].data}")
for tensor in reader.tensors:
    print(f"  {tensor.name}: {tensor.shape} ({tensor.tensor_type.name})")

# Convert a custom model to GGUF
from gguf import GGUFWriter, GGMLQuantizationType
import numpy as np

writer = GGUFWriter("output.gguf", arch="llama")
writer.add_name("My Custom Model")
writer.add_context_length(2048)
writer.add_embedding_length(4096)
writer.add_block_count(32)

weights = np.random.randn(4096, 4096).astype(np.float32)
writer.add_tensor("blk.0.attn_q.weight", weights)

writer.write_header_to_file()
writer.write_kv_data_to_file()
writer.write_tensors_to_file(progress=True)
writer.close()

Grammar / Constrained Output

GBNF (GGML BNF) grammar:

# Use built-in JSON grammar
./llama-cli --model model.gguf --grammar-file grammars/json.gbnf \
    -p "List 3 fruits as JSON"

# Custom grammar for specific output format
./llama-cli --model model.gguf \
    --grammar 'root ::= "Yes" | "No" | "Maybe"' \
    -p "Will it rain tomorrow?"

JSON Schema to grammar conversion:

curl http://localhost:8080/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "messages": [{"role": "user", "content": "Describe a person"}],
        "response_format": {
            "type": "json_schema",
            "json_schema": {
                "name": "person",
                "schema": {
                    "type": "object",
                    "properties": {
                        "name": {"type": "string"},
                        "age": {"type": "integer"},
                        "occupation": {"type": "string"}
                    },
                    "required": ["name", "age"]
                }
            }
        }
    }'

Embedding Extraction

# Start server in embedding mode
./llama-server \
    --model /models/nomic-embed-text-v1.5-q8_0.gguf \
    --port 8080 \
    --embeddings \
    --pooling mean

# Generate embeddings
curl http://localhost:8080/v1/embeddings \
    -H "Content-Type: application/json" \
    -d '{
        "input": ["Hello world", "How are you?"],
        "model": "local-embed",
        "encoding_format": "float"
    }'

For encoder-only models, use --pooling cls or --pooling mean as appropriate for the model's training.

CMake Package Integration

cmake_minimum_required(VERSION 3.14)
project(my_llama_app)

find_package(llama REQUIRED)

add_executable(my_app main.cpp)
target_link_libraries(my_app PRIVATE llama)
target_include_directories(my_app PRIVATE ${LLAMA_INCLUDE_DIRS})

cmake -B build \
    -Dllama_DIR=/path/to/llama.cpp/build/lib/cmake/llama
cmake --build build

pkg-config Integration

# After installing llama.cpp
pkg-config --cflags llama
pkg-config --libs llama

# Compile a single file
gcc -o myapp myapp.c \
    $(pkg-config --cflags llama) \
    $(pkg-config --libs llama)

XCFramework (Apple)

# Build XCFramework for distribution (iOS + macOS + simulators)
cd examples/llama.swiftui
./build-xcframework.sh

# Produces: llama.xcframework
# Contains:
#   ios-arm64/                       → Physical iPhone/iPad
#   ios-arm64_x86_64-simulator/      → Simulator
#   macos-arm64_x86_64/              → macOS (Apple Silicon + Intel)

Integrate in Xcode:

1. Drag llama.xcframework into your project
2. Set Embed & Sign for the framework
3. import llama in Swift files

Android JNI/NDK Integration

The examples/llama.android/ directory provides a complete Android integration:

// LlamaAndroid.kt (JNI wrapper)
class LlamaAndroid {
    companion object {
        init { System.loadLibrary("llama-android") }
    }

    external fun loadModel(modelPath: String, nGpuLayers: Int): Long
    external fun createContext(modelPtr: Long, nCtx: Int): Long
    external fun completion(contextPtr: Long, prompt: String): String
    external fun freeContext(contextPtr: Long)
    external fun freeModel(modelPtr: Long)
}

Build for multiple ABIs:

cmake -B build-arm64 \
    -DCMAKE_TOOLCHAIN_FILE=$NDK/build/cmake/android.toolchain.cmake \
    -DANDROID_ABI=arm64-v8a \
    -DANDROID_PLATFORM=android-26 \
    -DCMAKE_BUILD_TYPE=Release \
    -DGGML_NATIVE=OFF

cmake -B build-x86_64 \
    -DCMAKE_TOOLCHAIN_FILE=$NDK/build/cmake/android.toolchain.cmake \
    -DANDROID_ABI=x86_64 \
    -DANDROID_PLATFORM=android-26 \
    -DCMAKE_BUILD_TYPE=Release \
    -DGGML_NATIVE=OFF

MCP (Model Context Protocol)

The web UI supports MCP servers, enabling the LLM to call external tools:

{
  "servers": [
    {
      "name": "filesystem",
      "transport": "stdio",
      "command": "npx",
      "args": ["-y", "@modelcontextprotocol/server-filesystem", "/allowed/path"]
    },
    {
      "name": "web-search",
      "transport": "sse",
      "url": "http://localhost:3001/sse"
    }
  ]
}

The MCP client in the web UI translates tool calls from the LLM's function calling format into MCP protocol messages and returns results as tool response messages.

Note

MCP support requires a model with function calling capability (tool use). Not all models support this — check the model's documentation. Models must be loaded with a chat template that includes tool definition rendering.

Nix Flake

# Run with Nix (CPU)
nix run github:ggml-org/llama.cpp -- --model model.gguf -p "Hello"

# Build with CUDA in Nix
nix build github:ggml-org/llama.cpp#cuda

# Enter development shell
nix develop github:ggml-org/llama.cpp

Docker Hub / GHCR Images

Official Docker images are available at ghcr.io/ggml-org/llama.cpp:

# CPU
docker pull ghcr.io/ggml-org/llama.cpp:latest

# CUDA
docker pull ghcr.io/ggml-org/llama.cpp:cuda

# Run
docker run --gpus all \
    -v /models:/models \
    -p 8080:8080 \
    ghcr.io/ggml-org/llama.cpp:cuda \
    --server \
    --model /models/model.gguf \
    --n-gpu-layers -1 \
    --host 0.0.0.0 \
    --port 8080