# whisper.cpp Admin & Developer Guide

This guide is for engineers, architects, and operators who integrate, build, deploy, or extend `whisper.cpp`. It covers the system architecture, the complete build-option matrix across every acceleration backend, the C API, the HTTP server's operational characteristics, model conversion and quantization, security considerations, and the project's internal structure.

For installation basics and day-to-day CLI usage, see the companion **User Guide**.

---

## Table of Contents

1. [Executive Summary](#1-executive-summary)
2. [Architecture](#2-architecture)
3. [Repository Structure](#3-repository-structure)
4. [System Requirements](#4-system-requirements)
5. [Build System & Configuration Reference](#5-build-system--configuration-reference)
6. [Acceleration Backends](#6-acceleration-backends)
7. [The C API (`whisper.h`)](#7-the-c-api-whisperh)
8. [Models: Format, Conversion, Quantization](#8-models-format-conversion-quantization)
9. [Operating `whisper-server`](#9-operating-whisper-server)
10. [Language Bindings](#10-language-bindings)
11. [Security Analysis](#11-security-analysis)
12. [Code Quality & Conventions](#12-code-quality--conventions)
13. [Testing & CI](#13-testing--ci)
14. [Potential Improvements](#14-potential-improvements)
15. [Troubleshooting (Operational)](#15-troubleshooting-operational)
16. [Quick Reference](#16-quick-reference)

---

## 1. Executive Summary

`whisper.cpp` is a plain C/C++ port of OpenAI's Whisper ASR model, built on the [`ggml`](https://github.com/ggml-org/ggml) tensor library. Its design goals are portability, zero runtime dependencies, and broad hardware acceleration.

| Attribute | Value |
|-----------|-------|
| Language | C / C++17 |
| Core library | `libwhisper` (public header `include/whisper.h`) |
| Tensor backend | `ggml` (vendored under `ggml/`) |
| Stable release | v1.8.6 |
| License | MIT |
| Build system | CMake ≥ 3.28.0 |
| Runtime deps | None (CPU); optional GPU/accelerator runtimes |
| Model format | `ggml` single-file binary (`.bin`) |

**Key architectural properties:**

- The entire high-level model is implemented in just two files — `include/whisper.h` (the C API) and `src/whisper.cpp` (the implementation). Everything else is either the `ggml` backend or example programs.
- All heavy numerical work is delegated to `ggml`, which provides a pluggable backend abstraction. The **same model file** runs unchanged on CPU, Metal, CUDA, Vulkan, ROCm/HIP, SYCL, OpenVINO, Core ML, BLAS, CANN, MUSA, and more — the backend is selected at compile time.
- Memory is pre-allocated; the design advertises "zero memory allocations at runtime" during inference.
- The C API is thread-safe per `whisper_context` as long as a single context is not used by multiple threads concurrently.

---

## 2. Architecture

### 2.1 Inference pipeline

```
                       whisper.cpp inference pipeline
 ┌──────────┐   ┌───────────────┐   ┌───────────┐   ┌───────────┐   ┌────────┐
 │  Audio   │──▶│ Log-Mel       │──▶│ Encoder   │──▶│ Decoder   │──▶│  Text  │
 │ 16kHz PCM│   │ spectrogram   │   │ (Transf.) │   │ (Transf.) │   │segments│
 └──────────┘   └───────────────┘   └───────────┘   └───────────┘   └────────┘
       │              │                   │                │
   miniaudio /    whisper_pcm_to_mel  whisper_encode   whisper_decode
   ffmpeg         (n_mel = 80/128)    (audio ctx)      (autoregressive,
   decode                                               beam / greedy)
```

1. **Audio load** — example programs decode files with miniaudio (or FFmpeg if built in) into 16 kHz mono (or stereo) 32-bit float PCM.
2. **Mel spectrogram** — `whisper_pcm_to_mel()` converts PCM to a log-mel spectrogram using the model's embedded mel filters (`WHISPER_N_FFT=400`, `WHISPER_HOP_LENGTH=160`, 30-second `WHISPER_CHUNK_SIZE`).
3. **Encoder** — a Transformer encoder produces audio features. Optionally offloaded to Core ML (Apple Neural Engine) or OpenVINO.
4. **Decoder** — a Transformer decoder autoregressively emits text tokens, using greedy or beam-search sampling, conditioned on prior context, an optional initial prompt, and an optional GBNF grammar.
5. **Post-processing** — token streams are grouped into timestamped **segments**, optionally with token-level timestamps (DTW), confidence values, and speaker-turn markers.

### 2.2 Layered design

```
 ┌─────────────────────────────────────────────────────────────┐
 │  Example programs / apps                                      │
 │  whisper-cli · whisper-server · whisper-stream · command ·    │
 │  bench · quantize · talk-llama · mobile (objc/swiftui/android)│
 ├─────────────────────────────────────────────────────────────┤
 │  Public C API  —  include/whisper.h                           │
 │  whisper_init_* · whisper_full · whisper_vad_* · accessors    │
 ├─────────────────────────────────────────────────────────────┤
 │  Core implementation  —  src/whisper.cpp (+ coreml/openvino)  │
 ├─────────────────────────────────────────────────────────────┤
 │  ggml tensor library  —  ggml/ (compute graph + backends)     │
 │  CPU · Metal · CUDA · Vulkan · HIP · SYCL · BLAS · CANN ·      │
 │  MUSA · OpenVINO · WebGPU · OpenCL · RPC · zDNN · Hexagon      │
 └─────────────────────────────────────────────────────────────┘
```

### 2.3 The encoder offload paths

The encoder is the most compute-intensive part and has two optional alternative implementations:

- **Core ML** (`src/coreml/`) — on Apple Silicon, the encoder can run on the Neural Engine, reportedly >3× faster than CPU. Enabled with `WHISPER_COREML`. The first run compiles the model to a device-specific format and is slow; subsequent runs are fast.
- **OpenVINO** (`src/openvino/`) — on Intel CPUs/GPUs, the encoder can run via OpenVINO. Enabled with `WHISPER_OPENVINO`. Initialized at runtime via `whisper_ctx_init_openvino_encoder()`; the IR model path defaults next to the `ggml` model.

Both are no-ops if not compiled in, so code that calls them stays portable.

---

## 3. Repository Structure

```
whisper.cpp/
├── include/whisper.h          # Public C API (single header)
├── src/
│   ├── whisper.cpp            # Core model implementation
│   ├── coreml/                # Core ML encoder (Apple ANE)
│   └── openvino/              # OpenVINO encoder (Intel)
├── ggml/
│   ├── include/               # ggml public headers (ggml.h, ggml-backend.h, ...)
│   └── src/                   # ggml + every backend (ggml-cuda, ggml-metal,
│                              #   ggml-vulkan, ggml-sycl, ggml-cann, ggml-musa,
│                              #   ggml-hip, ggml-blas, ggml-opencl, ggml-rpc, ...)
├── examples/
│   ├── cli/                   # whisper-cli (main tool)
│   ├── server/                # whisper-server (HTTP API + vendored httplib)
│   ├── stream/                # whisper-stream (SDL2 live mic)
│   ├── command/               # whisper-command (SDL2 voice assistant)
│   ├── bench/                 # whisper-bench
│   ├── quantize/              # quantize tool
│   ├── talk-llama/            # voice chat with LLaMA
│   ├── vad-speech-segments/   # VAD segment extractor
│   ├── whisper.objc/ .swiftui/ .android/ .wasm/   # platform demos
│   └── ...                    # lsp, wchess, addon.node, python, nvim, etc.
├── bindings/
│   ├── go/  java/  ruby/  javascript/
├── models/                    # download/convert scripts, model docs
├── scripts/                   # bench.py, apple build scripts, helpers
├── tests/                     # earnings21 / librispeech WER harnesses
├── cmake/                     # toolchain files
├── CMakeLists.txt             # top-level build
└── build-xcframework.sh       # Apple XCFramework build
```

> [!NOTE]
> The high-level model logic is intentionally tiny (`whisper.h` + `whisper.cpp`). The bulk of the source tree is the `ggml` backend matrix and the example/binding ecosystem. When integrating, you only link against the core library and `ggml`.

---

## 4. System Requirements

### Build toolchain

| Component | Requirement |
|-----------|-------------|
| C/C++ compiler | C++17 — GCC, Clang, or MSVC (MinGW also supported) |
| CMake | ≥ 3.28.0 (the Apple XCFramework script enforces this) |
| Generator | Ninja or Make (Linux/macOS), MSVC/Ninja (Windows), Xcode (Apple) |

### Optional toolchains by backend

| Backend | Needs |
|---------|-------|
| CUDA | NVIDIA CUDA toolkit + driver |
| ROCm/HIP | AMD ROCm |
| Vulkan | Vulkan-capable driver + SDK |
| Metal / Core ML | Xcode + command-line tools (macOS) |
| OpenVINO | Intel OpenVINO toolkit (2024.6.0 recommended) |
| SYCL | Intel oneAPI (icx compiler) |
| MUSA | Moore Threads MUSA SDK rc4.2.0 |
| CANN | Huawei CANN toolkit |
| BLAS | OpenBLAS (or another BLAS) |
| SDL2 examples | libSDL2 dev package |
| FFmpeg decoding | libavcodec/libavformat/libavutil dev packages |
| Model conversion | Python 3.10/3.11, PyTorch, `openai-whisper`, `coremltools`/OpenVINO/`silero-vad` |

### Runtime memory/disk

See the model table in the User Guide: from `tiny` (75 MiB disk / ~273 MB RAM) to `large` (2.9 GiB / ~3.9 GB).

---

## 5. Build System & Configuration Reference

whisper.cpp uses CMake. The canonical flow:

```bash
cmake -B build [options]
cmake --build build -j --config Release
```

### 5.1 Top-level `whisper.cpp` (`WHISPER_*`) options

From the root `CMakeLists.txt`:

| Option | Default | Purpose |
|--------|---------|---------|
| `BUILD_SHARED_LIBS` | platform-dependent | Build shared vs. static libraries |
| `WHISPER_BUILD_TESTS` | ON (standalone) | Build the test suite |
| `WHISPER_BUILD_EXAMPLES` | ON (standalone) | Build the example programs |
| `WHISPER_BUILD_SERVER` | ON (standalone) | Build `whisper-server` |
| `WHISPER_SDL2` | OFF | Build SDL2 examples (`stream`, `command`) |
| `WHISPER_CURL` | OFF | Use libcurl to download models from a URL |
| `WHISPER_COREML` | OFF | Enable Core ML encoder (Apple) |
| `WHISPER_COREML_ALLOW_FALLBACK` | OFF | Allow non-Core ML CPU fallback |
| `WHISPER_OPENVINO` | OFF | Enable OpenVINO encoder (Intel) |
| `WHISPER_COMMON_FFMPEG` | OFF | FFmpeg audio decoding in examples |
| `WHISPER_USE_SYSTEM_GGML` | OFF | Link a system-installed `ggml` instead of the vendored copy |
| `WHISPER_ALL_WARNINGS` | ON | Enable all compiler warnings |
| `WHISPER_ALL_WARNINGS_3RD_PARTY` | OFF | Warnings in third-party code |
| `WHISPER_FATAL_WARNINGS` | OFF | Treat warnings as errors (`-Werror`) |
| `WHISPER_SANITIZE_THREAD` | OFF | Thread sanitizer |
| `WHISPER_SANITIZE_ADDRESS` | OFF | Address sanitizer |
| `WHISPER_SANITIZE_UNDEFINED` | OFF | Undefined-behavior sanitizer |

> [!IMPORTANT]
> **Deprecated `WHISPER_*` acceleration flags.** Older backend flags now live under the `GGML_*` namespace and are remapped at configure time (some with a fatal error). Use the new names:
>
> | Deprecated | Replacement |
> |------------|-------------|
> | `WHISPER_CUBLAS` | `GGML_CUDA` (now **fatal** — must update) |
> | `WHISPER_CUDA` | `GGML_CUDA` |
> | `WHISPER_METAL` | `GGML_METAL` |
> | `WHISPER_METAL_EMBED_LIBRARY` | `GGML_METAL_EMBED_LIBRARY` |
> | `WHISPER_KOMPUTE` | `GGML_KOMPUTE` |
> | `WHISPER_NATIVE` | `GGML_NATIVE` |
> | `WHISPER_OPENMP` | `GGML_OPENMP` |
> | `WHISPER_RPC` | `GGML_RPC` |
> | `WHISPER_SYCL` / `WHISPER_SYCL_F16` | `GGML_SYCL` / `GGML_SYCL_F16` |
> | `WHISPER_CCACHE` | `GGML_CCACHE` |
> | `WHISPER_FFMPEG` | `WHISPER_COMMON_FFMPEG` |

### 5.2 CMake presets

The repo ships `CMakePresets.json` with ready-made configurations. Highlights:

- Per-platform toolchains: `x64-linux-gcc-*`, `arm64-apple-clang-*`, `x64-windows-msvc-*`, `x64-windows-llvm-*`, `arm64-windows-llvm-*`.
- Backend presets: `x64-windows-sycl-*` (with optional `-f16`), `x64-windows-vulkan-*`.
- Build types are composed from hidden presets: `debug` (Debug), `release` (Release), `reldbg` (RelWithDebInfo), `static` (`GGML_STATIC=ON`).

Use one with:

```bash
cmake --preset x64-linux-gcc-release
cmake --build build-x64-linux-gcc-release -j
```

---

## 6. Acceleration Backends

All backends compile against the **same** `ggml` model file. Choose at configure time. At runtime, only `-ng/--no-gpu` and `-dev/--device` (and `WHISPER_ARG_DEVICE`) are adjustable.

| Backend | Configure flag | Notes |
|---------|----------------|-------|
| CPU (default) | (none) | AVX/AVX2/AVX512 on x86; ARM NEON + Apple Accelerate; VSX on POWER; per-arch code under `ggml/src/ggml-cpu/arch/` |
| Apple Metal | `-DGGML_METAL=ON` | Default on Apple Silicon; full GPU inference |
| Apple Core ML | `-DWHISPER_COREML=1` | Encoder on the Neural Engine; first run is slow (device compile) |
| NVIDIA CUDA | `-DGGML_CUDA=1` | cuBLAS + custom kernels; set `-DCMAKE_CUDA_ARCHITECTURES="86"` etc. for RTX 50-series |
| Vulkan | `-DGGML_VULKAN=1` | Cross-vendor GPU |
| AMD ROCm/HIP | `-DGGML_HIP=1 -DAMDGPU_TARGETS="gfx1201"` | Set arch (`rocminfo | grep gfx`) |
| Intel OpenVINO | `-DWHISPER_OPENVINO=1` | Encoder on Intel CPU/GPU; needs OpenVINO toolkit |
| Intel SYCL | `GGML_SYCL` (+ `GGML_SYCL_F16`) | Via oneAPI / `icx` |
| BLAS (CPU) | `-DGGML_BLAS=1` | OpenBLAS etc.; helps the encoder |
| Huawei Ascend | `-DGGML_CANN=1` | NPU via CANN |
| Moore Threads | `-DGGML_MUSA=1` | muBLAS + MUSA kernels; `-DMUSA_ARCHITECTURES="21"` |

Representative build commands:

```bash
# NVIDIA (RTX 50-series)
cmake -B build -DGGML_CUDA=1 -DCMAKE_CUDA_ARCHITECTURES="86"

# AMD RX 9070 XT
cmake -B build -DGGML_HIP=1 -DAMDGPU_TARGETS="gfx1201"

# Vulkan
cmake -B build -DGGML_VULKAN=1

# Apple Core ML
cmake -B build -DWHISPER_COREML=1

# CPU BLAS (also the path for POWER VSX)
cmake -B build -DGGML_BLAS=1
```

> [!TIP]
> Confirm the active backend at runtime: the `system_info:` line printed at startup shows feature flags like `AVX2`, `NEON`, `BLAS`, `COREML`, `OPENVINO`, etc.

The `ggml/src/` tree contains additional backends not all wired into whisper.cpp's documented build recipes (WebGPU, OpenCL, RPC, zDNN, ZenDNN, Hexagon, virtgpu). Treat those as experimental/advanced.

---

## 7. The C API (`whisper.h`)

`whisper.h` is a stable C interface (usable from C and C++). It is thread-safe per context, provided one `whisper_context` is not shared across threads concurrently.

### 7.1 Minimal usage

```c
#include "whisper.h"

struct whisper_context_params cparams = whisper_context_default_params();
struct whisper_context * ctx =
    whisper_init_from_file_with_params("/path/to/ggml-base.en.bin", cparams);

struct whisper_full_params wparams =
    whisper_full_default_params(WHISPER_SAMPLING_GREEDY);

if (whisper_full(ctx, wparams, pcmf32.data(), pcmf32.size()) != 0) {
    fprintf(stderr, "failed to process audio\n");
    return 7;
}

const int n = whisper_full_n_segments(ctx);
for (int i = 0; i < n; ++i) {
    printf("%s", whisper_full_get_segment_text(ctx, i));
}

whisper_free(ctx);
```

`pcmf32` is raw 32-bit float PCM at 16 kHz.

### 7.2 Lifecycle functions

| Function | Purpose |
|----------|---------|
| `whisper_init_from_file_with_params` | Load a model from a file path |
| `whisper_init_from_buffer_with_params` | Load a model from an in-memory buffer |
| `whisper_init_with_params` | Load via a custom `whisper_model_loader` callback |
| `*_no_state` variants | Allocate without internal state; caller manages `whisper_init_state()` |
| `whisper_init_state` | Allocate a `whisper_state` explicitly |
| `whisper_free`, `whisper_free_state` | Release resources |
| `whisper_free_params`, `whisper_free_context_params` | Free `*_by_ref` param structs |

> [!CAUTION]
> The legacy initializers without a `params` argument (`whisper_init_from_file`, `whisper_init`, etc.) are **deprecated** via `WHISPER_DEPRECATED`. Use the `_with_params` forms.

### 7.3 The core call: `whisper_full`

```c
int whisper_full(struct whisper_context * ctx,
                 struct whisper_full_params params,
                 const float * samples, int n_samples);
```

Runs the full pipeline (PCM → mel → encode → decode → text). Variants:

- `whisper_full_with_state` — run against an explicit state.
- `whisper_full_parallel` — split the audio across `n_processors` for speed (with some accuracy loss at chunk boundaries). This is what `whisper-server` uses.

Lower-level building blocks are also exposed: `whisper_pcm_to_mel`, `whisper_set_mel`, `whisper_encode`, `whisper_decode`, `whisper_tokenize`.

### 7.4 Key parameter structs

**`whisper_context_params`** — set once at load time: `use_gpu`, `flash_attn`, `gpu_device`, and the experimental DTW token-timestamp fields (`dtw_token_timestamps`, `dtw_aheads_preset`, `dtw_n_top`, `dtw_aheads`).

**`whisper_full_params`** — per-inference. Notable fields: `strategy` (greedy / beam search), `n_threads`, `n_max_text_ctx`, `offset_ms`, `duration_ms`, `translate`, `no_context`, `no_timestamps`, `single_segment`, `token_timestamps`, `max_len`, `split_on_word`, `max_tokens`, `audio_ctx`, `tdrz_enable`, `suppress_regex`, `initial_prompt` / `carry_initial_prompt` / `prompt_tokens`, `language` / `detect_language`, `suppress_blank` / `suppress_nst`, the decoding thresholds (`temperature`, `temperature_inc`, `entropy_thold`, `logprob_thold`, `no_speech_thold`, `max_initial_ts`, `length_penalty`), `greedy.best_of`, `beam_search.beam_size`, the GBNF grammar fields, and the VAD block (`vad`, `vad_model_path`, `vad_params`).

Obtain defaults with `whisper_full_default_params(strategy)` and `whisper_context_default_params()`.

### 7.5 Callbacks

`whisper_full_params` accepts function pointers for: new-segment notification (`whisper_new_segment_callback`), progress (`whisper_progress_callback`), encoder-begin (return `false` to abort), an abort callback (`ggml_abort_callback` — the server uses it to stop on client disconnect), and a logits filter (`whisper_logits_filter_callback`). Set logging globally with `whisper_log_set()`.

### 7.6 Result accessors

After `whisper_full`, read results: `whisper_full_n_segments`, `whisper_full_get_segment_text`, `whisper_full_get_segment_t0/t1`, `whisper_full_lang_id`, `whisper_full_n_tokens`, `whisper_full_get_token_text/_id/_data/_p`, `whisper_full_get_segment_speaker_turn_next` (tinydiarize), and `whisper_full_get_segment_no_speech_prob`.

### 7.7 Voice Activity Detection API

A self-contained VAD API mirrors the main flow: `whisper_vad_init_from_file_with_params`, `whisper_vad_detect_speech` (and `_no_reset` + `whisper_vad_reset_state` for streaming), `whisper_vad_segments_from_probs` / `_from_samples`, and the `whisper_vad_params` struct (`threshold`, `min_speech_duration_ms`, `min_silence_duration_ms`, `max_speech_duration_s`, `speech_pad_ms`, `samples_overlap`).

### 7.8 Language & model introspection

`whisper_lang_id`, `whisper_lang_str`, `whisper_lang_str_full`, `whisper_lang_max_id`, `whisper_lang_auto_detect`, plus a family of `whisper_model_n_*` accessors (vocab, audio/text ctx/state/head/layer, mels, ftype) and `whisper_print_system_info()`.

> [!NOTE]
> Constants: `WHISPER_SAMPLE_RATE` = 16000, `WHISPER_N_FFT` = 400, `WHISPER_HOP_LENGTH` = 160, `WHISPER_CHUNK_SIZE` = 30.

---

## 8. Models: Format, Conversion, Quantization

### 8.1 The `ggml` model format

A model is a single binary file packing everything the runtime needs: the `ggml` magic (`0x67676d6c`), hyperparameters (vocab size, audio/text context/state/head/layer counts, n_mels, ftype), the mel filter bank, the tokenizer vocabulary, and the weights (F16 by default, with small tensors kept F32). Download with `models/download-ggml-model.sh` or fetch from [huggingface.co/ggerganov/whisper.cpp](https://huggingface.co/ggerganov/whisper.cpp).

### 8.2 Converting from PyTorch / Hugging Face

The `models/` directory contains conversion scripts (Python; require PyTorch and OpenAI `whisper`):

| Script | Converts |
|--------|----------|
| `convert-pt-to-ggml.py` | OpenAI `.pt` checkpoint → `ggml` |
| `convert-h5-to-ggml.py` | Hugging Face model → `ggml` |
| `convert-whisper-to-coreml.py` | Whisper → Core ML encoder (ANE) |
| `convert-h5-to-coreml.py` | HF model → Core ML |
| `convert-whisper-to-openvino.py` | Whisper → OpenVINO IR encoder |
| `convert-silero-vad-to-ggml.py` | Silero-VAD → `ggml` VAD model |
| `ggml_to_pt.py` | `ggml` → PyTorch (reverse) |

Example:

```bash
python3 models/convert-pt-to-ggml.py ~/.cache/whisper/medium.pt ~/whisper/ ./models/whisper-medium
```

Valid model names enforced by the converters: `tiny`, `tiny.en`, `base`, `base.en`, `small`, `small.en`, `medium`, `medium.en`, `large-v1`, `large-v2`, `large-v3`, `large-v3-turbo` (an invalid name raises `ValueError("Invalid model name")`).

### 8.3 Core ML / OpenVINO encoder generation

```bash
# Core ML (Apple): pip install ane_transformers openai-whisper coremltools
./models/generate-coreml-model.sh base.en      # → models/ggml-base.en-encoder.mlmodelc

# OpenVINO (Intel): pip install -r models/requirements-openvino.txt
python models/convert-whisper-to-openvino.py --model base.en   # → ggml-base.en-encoder-openvino.xml/.bin
```

Build the matching backend (`-DWHISPER_COREML=1` or `-DWHISPER_OPENVINO=1`) and place the generated files alongside the `ggml` model so they are auto-discovered at runtime.

### 8.4 Quantization

```bash
./build/bin/quantize models/ggml-base.en.bin models/ggml-base.en-q5_0.bin q5_0
```

`quantize <in-f32.bin> <out.bin> <type>` validates the magic number, copies hyperparameters with an updated quantization version, and re-encodes weights with `ggml_common_quantize_0`. Supported types include `q4_0`, `q5_0`, `q5_1`, `q8_0` (the tool prints all `ggml` ftypes when run without arguments). Many quantized models are also published for direct download.

---

## 9. Operating `whisper-server`

`whisper-server` (built when `WHISPER_BUILD_SERVER` is on) is a single-binary HTTP service built on a vendored `cpp-httplib` and `nlohmann::json`.

### 9.1 Endpoints

| Method | Path | Purpose |
|--------|------|---------|
| `GET` | `/` (under `--request-path`) | Built-in HTML upload form (only if no `index.html` in `--public`) |
| `POST` | `/inference` (configurable via `--inference-path`) | Transcribe an uploaded file |
| `OPTIONS` | `/inference` | CORS preflight |
| `POST` | `/load` | Load a different model from a server-side path |
| `GET` | `/health` | Readiness: `{"status":"ok"}` (200) or `{"status":"loading model"}` (503) |

Request/response details (form fields, response formats) are in the User Guide §12.

### 9.2 Concurrency & lifecycle

- A single `whisper_context` is guarded by a `std::mutex` (`whisper_mutex`). **Requests are serialized** — the server handles one transcription at a time. Scale out with multiple instances behind a reverse proxy/load balancer.
- An `abort_callback` checks `req.is_connection_closed()`; if the client disconnects mid-inference, processing aborts and the server returns HTTP `499` (nginx convention).
- A server state flag (`SERVER_STATE_LOADING_MODEL` / `SERVER_STATE_READY`) drives `/health`. During `POST /load`, state flips to loading; if the new model fails to initialize, the process **exits** (`exit(1)`) rather than continuing without a model.
- Read/write timeouts default to 600 seconds.
- `SIGINT`/`SIGTERM` trigger a graceful shutdown; a second interrupt forces immediate exit.

### 9.3 Deployment recommendations

- **Bind address:** defaults to `127.0.0.1`. Use `--host 0.0.0.0` only behind a trusted proxy; do not expose the raw port to the internet.
- **Format conversion:** enable `--convert` (and install `ffmpeg`) to accept arbitrary audio formats; otherwise only miniaudio-decodable formats work. Set `--tmp-dir` to a writable scratch path.
- **Path prefixing:** `--request-path` lets you mount the server under a sub-path behind a proxy; `--inference-path` renames the transcription endpoint.
- **Static assets:** `--public` serves a directory at `/`; drop a custom `index.html` there to replace the built-in form.
- **Docker:** use `ghcr.io/ggml-org/whisper.cpp:main` (or `-cuda`/`-vulkan`/`-musa`), mounting a models volume, e.g. `whisper-server --host 0.0.0.0 -m /models/ggml-base.bin`.

> [!WARNING]
> `--convert` invokes `ffmpeg` via `std::system()` on a server-generated temp filename. Keep `ffmpeg` patched and `--tmp-dir` on a non-shared, least-privileged path. See also the related issues [#3774](https://github.com/ggml-org/whisper.cpp/issues/3774) / [#3819](https://github.com/ggml-org/whisper.cpp/issues/3819) about non-WAV handling when conversion is disabled.

---

## 10. Language Bindings

Official bindings live under `bindings/`:

| Binding | Location | Notes |
|---------|----------|-------|
| Go | `bindings/go` | `github.com/ggerganov/whisper.cpp/bindings/go`; uses `go-audio/wav`; Go 1.23 |
| Java | `bindings/java` | JNA-based (`net.java.dev.jna:jna`); published as `io.github.ggerganov:whispercpp` |
| Ruby | `bindings/ruby` | Native extension (`bindings/ruby/ext`) |
| JavaScript / WASM | `bindings/javascript` | npm package `whisper.cpp` |

The Java binding's Gradle build copies the native libraries (`libwhisper.{so,dylib}`, `whisper.dll`, `ggml*.dll`) into resources via the `copyLibs` task — your packaging must ship the matching native artifact for the target platform.

Many community bindings exist (Rust `whisper-rs`, Python `pywhispercpp`/`whispercpp`, .NET `whisper.net`, Swift `SwiftWhisper`, Unity, R). The Objective-C/Swift path is typically consumed via the prebuilt **XCFramework** (`build-xcframework.sh`).

---

## 11. Security Analysis

> [!CAUTION]
> whisper.cpp parses untrusted binary inputs (audio files and model files). Treat both as attack surface, especially in server deployments. Community-reported issues at this commit:
> - **[#3807](https://github.com/ggml-org/whisper.cpp/issues/3807)** — null-pointer dereference + assertion abort via zero-dimension model parameters in `whisper_model_load()`.
> - **[#3802](https://github.com/ggml-org/whisper.cpp/issues/3802)** — heap buffer overflow in the vendored `stb_vorbis` decoder via `setup_malloc` integer truncation.

**Input handling.** Audio is decoded by miniaudio (or `ggml`'s/example helpers; `stb_vorbis` for OGG). A malicious file can trigger the decoder bugs above. Only process trusted inputs, or sandbox the process.

**The server.**
- No authentication or authorization is built in — anyone who can reach the port can transcribe, and via `POST /load` can load **any model file path readable by the server process**. Put it behind an authenticating proxy and restrict the filesystem.
- `--convert` shells out to `ffmpeg` with `std::system()`. The command is built from a server-generated temp filename (not directly from the uploaded filename), which limits injection, but `ffmpeg` itself parses untrusted media — keep it updated and run least-privileged.
- Default bind is `127.0.0.1`; binding to `0.0.0.0` exposes the service.
- Requests are serialized behind a mutex, so a single slow/large upload blocks others — a denial-of-service consideration for public endpoints (mitigate with proxy timeouts, body-size limits, and the existing 600 s read/write timeouts).

**Secrets.** `HF_TOKEN` is read by the download script and sent as an `Authorization: Bearer` header to Hugging Face. The Java publishing path reads `MAVEN_USERNAME`/`MAVEN_PASSWORD`/`PGP_SECRET`/`PGP_PASSPHRASE` from the environment — relevant only to maintainers publishing artifacts.

**Hardening recommendations.**
- [ ] Run `whisper-server` behind a reverse proxy that enforces auth, TLS, rate limits, and request-size caps.
- [ ] Restrict the process's filesystem so `POST /load` cannot reach sensitive paths.
- [ ] Keep `ffmpeg` and the vendored decoders current; rebuild after upstream security fixes.
- [ ] Validate/limit upload size and duration at the proxy.
- [ ] Consider sanitizer builds (`WHISPER_SANITIZE_ADDRESS`/`_UNDEFINED`) when fuzzing or vetting inputs.

---

## 12. Code Quality & Conventions

- **C-style public API.** The interface is plain C in `whisper.h`, with `WHISPER_API` export macros (handling `__declspec(dllexport/dllimport)` on Windows and visibility on others) and `WHISPER_DEPRECATED` for graceful migration. This keeps FFI/binding surfaces simple.
- **Single-responsibility core.** Model logic is concentrated in `whisper.cpp`; numerical kernels are delegated to `ggml`. This separation lets backends evolve independently of the ASR logic.
- **Hand-rolled CLI parsing.** Each example program defines its own `whisper_params` struct (with inline defaults) and a manual `for (i < argc)` argument loop paired with a `whisper_print_usage()` block. There is no shared options library across examples, so the same flag (e.g. `-fa`) is re-declared per tool. This is a deliberate ggml-family idiom favoring zero dependencies.
- **Defaults as source of truth.** Default values live in the param structs and `whisper_full_default_params()`; the header explicitly warns that reordering `whisper_full_params` requires updating the defaults function.
- **Experimental features are flagged.** Token-level timestamps (DTW), tinydiarize (`[TDRZ]`), and speed-up techniques are marked `[EXPERIMENTAL]` in the header with caveats about quality.
- **Windows/Unicode care.** `whisper-cli` sets the console output code page to UTF-8 and merges multi-byte UTF-8 tokens (CJK) when emitting JSON (refs issue #1798).
- **Response-file support.** `whisper-cli` accepts `@file` to read args from a file — convenient for long invocations.

---

## 13. Testing & CI

- **WER harnesses.** `tests/earnings21/` and `tests/librispeech/` compute Word Error Rate against standard datasets. Both declare a minimal Python dependency set (`jiwer`, `regex`, `more-itertools`) in `requirements.txt`.
- **Binding tests.** The Java binding runs JUnit 5 tests with JNA on the built native library path; the Node addon uses Jest.
- **CI.** GitHub Actions runs the build matrix (badge in the README). A scheduled `close-issue` workflow marks issues stale after 30 days and closes them after 14 more, exempting labels like `bug`, `roadmap`, `help wanted`.
- **Sanitizers.** Thread/Address/Undefined sanitizers are available as `WHISPER_SANITIZE_*` build options for local verification.

> [!NOTE]
> The confidence/triage signal for this repo flagged limited test-coverage depth and many shallow (LOW-priority) folders — most of the tree is example apps and per-architecture `ggml` kernels rather than ASR-logic tests. Functional correctness is largely validated through the WER harnesses and downstream usage.

---

## 14. Potential Improvements

Derived from the code, comments, and community issues:

- **Decoder concurrency.** `whisper_decode` carries a `TODO: add support for multiple decoders`; beam search currently has limited parallelism. Server throughput is bounded by the single-context mutex — a context pool or worker model would help.
- **Server format coverage.** `verbose_json` notes `compression_ratio` and `no_speech_prob` are partially unimplemented (`compression_ratio` is TODO). The server also has a `TODO: add more output formats`.
- **Speaker diarization.** Real diarization is a top community request ([#64](https://github.com/ggml-org/whisper.cpp/issues/64)); current options (`--diarize` stereo energy, `--tinydiarize`) are heuristic/experimental.
- **Beam-search patience.** `whisper_full_params.beam_search.patience` is defined but `TODO: not implemented`.
- **DTW memory.** `whisper_context_params.dtw_mem_size` is marked `TODO: remove`.
- **Hardware edge cases.** Open issues report a CUDA-image `SIGILL` on AMD Zen 3 ([#3814](https://github.com/ggml-org/whisper.cpp/issues/3814)) and a Vulkan/flash-attention crash on RDNA4 ([#3806](https://github.com/ggml-org/whisper.cpp/issues/3806)) — backend stability across exotic hardware is an ongoing effort.
- **Packaging.** Requests for `pkg-config` files ([#781](https://github.com/ggml-org/whisper.cpp/issues/781)) and prebuilt Linux binaries ([#3852](https://github.com/ggml-org/whisper.cpp/issues/3852)) remain open.

---

## 15. Troubleshooting (Operational)

### Build can't find a backend toolkit

**Cause:** Backend enabled but its SDK/driver is missing or not on the path.
**Solution:** Install the toolkit (CUDA/ROCm/OpenVINO/etc.), source its environment (e.g. OpenVINO `setupvars.sh`), and reconfigure from a clean `build/`.

### Configure fails with a deprecated-flag error

**Cause:** Using an old `WHISPER_*` acceleration flag (e.g. `WHISPER_CUBLAS`, which is now fatal).
**Solution:** Switch to the `GGML_*` equivalent (see [§5.1](#51-top-level-whispercpp-whisper_-options)).

### CUDA image crashes with `SIGILL` after model load

**Cause:** CPU-feature mismatch / `LD_LIBRARY_PATH` shadowing host `libcuda` (issue [#3814](https://github.com/ggml-org/whisper.cpp/issues/3814)).
**Solution:** Avoid overriding `LD_LIBRARY_PATH` to host CUDA libs inside the container; use the image's bundled libraries.

### Server crashes with flash attention on some GPUs

**Cause:** Flash-attention path unstable on certain Vulkan drivers/GPUs (issue [#3806](https://github.com/ggml-org/whisper.cpp/issues/3806)).
**Solution:** Start the server/CLI with `-nfa`/`--no-flash-attn`.

### `/load` makes the server exit

**Cause:** The requested model failed to initialize; the server calls `exit(1)` rather than run model-less.
**Solution:** Validate the model path/file before calling `/load`; supervise the process (systemd/Docker restart) to recover.

### High latency under concurrent load

**Cause:** Single-context mutex serializes requests.
**Solution:** Run multiple server instances behind a load balancer; consider VAD and smaller/quantized models to cut per-request time.

### Core ML / OpenVINO first run is very slow

**Cause:** The device compiles the model to a device-specific blob on first use.
**Solution:** Expected; subsequent runs are cached and fast. Pre-warm during deployment.

---

## 16. Quick Reference

### Build cheat-sheet

```bash
cmake -B build                              # CPU
cmake -B build -DGGML_CUDA=1                # NVIDIA
cmake -B build -DGGML_VULKAN=1              # Vulkan
cmake -B build -DGGML_HIP=1 -DAMDGPU_TARGETS="gfx1201"   # AMD
cmake -B build -DWHISPER_COREML=1           # Apple ANE
cmake -B build -DWHISPER_OPENVINO=1         # Intel
cmake -B build -DWHISPER_SDL2=ON            # + stream/command
cmake -B build -DWHISPER_COMMON_FFMPEG=yes  # + broad audio formats
cmake --build build -j --config Release
```

### Server endpoints

| Method | Path | Returns |
|--------|------|---------|
| `POST` | `/inference` | Transcript (`json`/`verbose_json`/`text`/`srt`/`vtt`) |
| `POST` | `/load` | `Load was successful!` |
| `GET` | `/health` | `{"status":"ok"}` / `{"status":"loading model"}` |
| `GET` | `/` | HTML form |

### Core C API calls

```text
whisper_context_default_params()            → context params
whisper_init_from_file_with_params(path, cp)→ ctx
whisper_full_default_params(strategy)       → inference params
whisper_full(ctx, wp, samples, n)           → 0 on success
whisper_full_n_segments(ctx)                → segment count
whisper_full_get_segment_text(ctx, i)       → segment text
whisper_free(ctx)                           → cleanup
```

### Key constants

| Constant | Value |
|----------|-------|
| `WHISPER_SAMPLE_RATE` | 16000 |
| `WHISPER_N_FFT` | 400 |
| `WHISPER_HOP_LENGTH` | 160 |
| `WHISPER_CHUNK_SIZE` | 30 |
| `ggml` magic | `0x67676d6c` |

### Environment variables

| Variable | Used by | Purpose |
|----------|---------|---------|
| `WHISPER_ARG_DEVICE` | cli, server | Override GPU device ID at startup |
| `HF_TOKEN` | download script | Hugging Face auth for model download |

---

_Generated June 2026 from ggml-org/whisper.cpp at ad17783. AI-generated documentation. The library is checked by sampling (three repositories so far); this guide was not individually reviewed. Verify against the project before relying on it. Unofficial; not affiliated with or reviewed by the project._