# VibeVoice User Guide

VibeVoice is Microsoft's open-source family of speech models for **streaming text-to-speech (TTS)** and **long-form automatic speech recognition (ASR)** with speaker diarization. The codebase ships two production-ready model families, several inference scripts, two Gradio demos, a FastAPI WebSocket streaming server, a vLLM serving plugin, and a LoRA fine-tuning recipe — all built on Qwen2.5 backbones.

This guide is for end users running inference, transcribing audio, generating voice-over from scripts, or wiring VibeVoice into an application. For architecture, deployment, and developer internals, see the **VibeVoice Administrator & Developer Guide**.

---

## 1. Quick Start

### What is VibeVoice?

VibeVoice is a research-grade speech stack from Microsoft that delivers two capabilities from a single repository:

- **VibeVoice-Realtime-0.5B** — A 0.5B-parameter streaming TTS model that converts text into 24kHz speech with a chosen voice preset. Output streams chunk-by-chunk as it is synthesized, making it suitable for real-time read-aloud, podcast narration, and web playback.
- **VibeVoice-ASR** — A larger Qwen2.5-7B-based ASR model that transcribes long-form audio (up to ~61 minutes per request) to time-stamped, speaker-diarized text. It accepts MP3, WAV, FLAC, OGG, and 20+ other formats via FFmpeg.

### Key features

- Streaming TTS with WebSocket + browser playback at 24kHz, sub-second time-to-first-byte on CUDA + flash-attention-2.
- Multi-language voice presets shipped in the repository: English, German, French, Italian, Japanese, Korean, Polish, Portuguese, Spanish, Dutch, plus Indian English (`in-Samuel_man`).
- Long-form ASR with timestamped JSON output (`{"Start": ..., "End": ..., "Speaker": ..., "Content": ...}`), speaker diarization, and optional context hotwords for proper-noun accuracy.
- Hugging Face Transformers integration: every model loads with `from_pretrained()` and runs on CUDA, MPS (Apple Silicon), Intel XPU, or CPU with automatic attention-implementation selection.
- A vLLM general plugin (`vibevoice.vllm_plugin`) for high-throughput ASR serving.
- LoRA fine-tuning recipe for adapting the ASR model to new domains, hotword sets, or low-resource languages.

### Prerequisites

Python 3.10+, FFmpeg in `PATH`, PyTorch with the appropriate backend (CUDA 12.x, MPS, XPU, or CPU), and 16+ GB GPU memory recommended for the 7B ASR model.

### Hello-world: TTS from a text file

```bash
python demo/realtime_model_inference_from_file.py \
  --model_path microsoft/VibeVoice-Realtime-0.5B \
  --txt_path demo/text_examples/1p_vibevoice.txt \
  --speaker_name Carter \
  --output_dir ./outputs
```

```text
[startup] Found 23 voice presets
Using device: cuda, torch_dtype: torch.bfloat16, attn_implementation: flash_attention_2
Using voice preset for Carter: .../voices/streaming_model/en-Carter_man.pt
Generation time: 12.37 seconds
Generated audio duration: 38.42 seconds
RTF (Real Time Factor): 0.32x
Saved output to ./outputs/1p_vibevoice_generated.wav
```

> [!TIP]
> Real Time Factor (RTF) below `1.0` means generation is faster than playback. On a CUDA GPU with flash-attention-2 you should consistently see RTF in the 0.2 – 0.5 range, leaving headroom for streaming use.

### Hello-world: ASR from an audio file

```bash
python demo/vibevoice_asr_inference_from_file.py \
  --model_path microsoft/VibeVoice-ASR \
  --audio_files demo/asr_demo/demo1-chat.mp3 \
  --batch_size 1
```

The script prints per-segment results in the form:

```text
[0.00 - 5.34] Speaker 0: Hey everyone, welcome back to the show...
[5.34 - 9.81] Speaker 1: Thanks for having me, great to be here.
```

### What to try next

- Launch the **Gradio ASR demo** (`demo/vibevoice_asr_gradio_demo.py`) for a browser UI with audio upload, microphone recording, and per-segment playback.
- Launch the **streaming TTS web app** (`demo/vibevoice_realtime_demo.py`) for live WebSocket TTS playback in any browser.
- Add hotwords with `--context_info "Hotwords: Acme Corp, Dr. Tan"` to improve proper-noun transcription accuracy.
- Fine-tune the ASR model on your own data with `finetuning-asr/lora_finetune.py`.

---

## 2. Installation

### System requirements

| Resource | Minimum | Recommended |
|---|---|---|
| Python | 3.10 | 3.11+ |
| OS | Linux, macOS 13+, Windows 11 | Linux (best CUDA support) |
| GPU (TTS-Realtime-0.5B) | 8 GB VRAM | 12 GB CUDA + flash-attention-2 |
| GPU (ASR / 7B) | 16 GB VRAM | 24 GB CUDA + flash-attention-2 |
| CPU fallback | Yes (slow) | Not recommended for ASR |
| Apple Silicon | M1+ via MPS, float32 only | M2 Pro / M3 Max |
| Intel | XPU (Arc, Data Center GPU Max) | Linux + Intel Extension for PyTorch |
| FFmpeg | Required for any audio I/O | Latest stable |

> [!IMPORTANT]
> FFmpeg must be on `PATH` before launching any inference script. The audio loader shells out to `ffmpeg`/`ffprobe` for every input file. On Windows install via `winget install Gyan.FFmpeg` or place `ffmpeg.exe` next to your Python interpreter.

### Install from source

The repository is a regular Python package with `pyproject.toml`. Clone, create a virtual environment, and install editable.

```bash
git clone https://github.com/microsoft/VibeVoice.git
cd VibeVoice

python -m venv .venv
source .venv/bin/activate          # Windows: .venv\Scripts\activate

pip install --upgrade pip
pip install -e .
```

Add Hugging Face Transformers, PyTorch, PEFT (for LoRA), and audio dependencies. The exact PyTorch wheel depends on your platform — pick one of the rows below.

| Platform | Install command |
|---|---|
| **CUDA 12.1** (Linux/Windows) | `pip install torch --index-url https://download.pytorch.org/whl/cu121` |
| **CPU only** | `pip install torch` |
| **Apple Silicon (MPS)** | `pip install torch` (uses default wheel, MPS auto-detected) |
| **Intel XPU** | Follow the Intel Extension for PyTorch installation guide |

Then add the rest of the runtime stack:

```bash
pip install transformers peft accelerate diffusers
pip install soundfile pydub numpy
pip install fastapi uvicorn[standard] websockets
pip install gradio
pip install vllm                          # optional, for the vLLM plugin
pip install flash-attn --no-build-isolation   # optional, recommended on CUDA
pip install liger-kernel                  # optional, accelerates Qwen2 layers in the ASR Gradio demo
```

> [!TIP]
> `flash-attn` requires nvcc and a matching PyTorch version. If installation fails, omit it — the demos automatically fall back to PyTorch SDPA. SDPA quality is "lower than flash-attention-2" per the demo banners but is functionally complete.

### Verify the install

```bash
python -c "from vibevoice import VibeVoiceStreamingForConditionalGenerationInference; print('ok')"
```

```text
ok
```

For ASR:

```bash
python -c "from vibevoice.modular.modeling_vibevoice_asr import VibeVoiceASRForConditionalGeneration; print('ok')"
```

```text
ok
```

### Download voice presets

The 23 voice presets that ship in the repo (`demo/voices/streaming_model/*.pt`) are committed as Git LFS objects:

```bash
# After cloning, pull the .pt files
git lfs install
git lfs pull
```

For experimental voice packs (~9 language packs distributed via GitHub release attachments), run the downloader:

```bash
bash demo/download_experimental_voices.sh
```

```text
[INFO] Starting download of experimental voices...
[INFO] Target directory: .../demo/voices/streaming_model/experimental_voices
[INFO] Downloading experimental_voices_de.tar.gz ...
...
[SUCCESS] All experimental speakers installed successfully!
```

### Install troubleshooting

> [!WARNING]
> Several install paths fail silently and only surface as cryptic runtime errors. The entries below are pulled from real community reports.

| Symptom | Cause | Fix |
|---|---|---|
| `ImportError: soundfile is required to save audio files` | `pip install soundfile` was skipped. | `pip install soundfile` and re-run. |
| `flash_attn` build fails with `nvcc not found` | CUDA toolkit headers absent. | Install CUDA toolkit, or skip `flash-attn` and accept SDPA fallback. |
| `RuntimeError: MPS not available. Falling back to CPU.` | macOS < 13 or non-Apple-Silicon. | Use a Linux+CUDA host or accept CPU performance. |
| `MODEL_PATH not set in environment` when starting `demo/vibevoice_realtime_demo.py` | Launcher did not set `MODEL_PATH` before importing `web.app`. | Always launch via `vibevoice_realtime_demo.py`, not `uvicorn web.app:app` directly. |
| `Voices directory not found` at startup | Running from outside the repo root, or `demo/voices/streaming_model/` is missing. | `git lfs pull` from the repo root, then re-run. |
| `Token indices sequence length is longer than the specified maximum` | Script text exceeds the decoder context. | Shorten the script. The 1.5B-64K config supports 65,536 tokens; the 7B-32K supports 32,768 (issue #332). |
| Gradio ASR demo crashes when accessed via Docker | HTTP-only access to `/file=` endpoints (issue #339). | Use HTTPS, or expose the port natively without an HTTP-only proxy. |
| Constant crash on Windows 11 | Mixed CUDA-driver / Visual C++ runtime mismatch (issue #327). | Install latest NVIDIA driver, latest VC++ Redistributable, reinstall PyTorch wheel. |
| `Multiple voice presets match the speaker name` | Substring match to `--speaker_name` is ambiguous. | Use a more specific name, e.g. `Carter` instead of `er`. |
| `Audio duration … exceeds the configured limit` | Audio longer than `VIBEVOICE_MAX_AUDIO_DURATION` (default 3660s). | Set `VIBEVOICE_MAX_AUDIO_DURATION=7200` (or split the input). |

---

## 3. Configuration

VibeVoice is configured almost entirely through CLI flags and environment variables. Model architecture lives in JSON files distributed inside each Hugging Face checkpoint; users never edit those.

### Environment variables

| Variable | Default | Purpose |
|---|---|---|
| `MODEL_PATH` | (none, required) | Hugging Face repo ID or local directory for the streaming TTS web server. |
| `MODEL_DEVICE` | `cuda` | Device for the streaming TTS web server (`cuda`, `mps`, `cpu`). |
| `VOICE_PRESET` | `en-Carter_man` | Default voice for the streaming TTS server when no `voice` query param is supplied. |
| `VIBEVOICE_MAX_AUDIO_DURATION` | `3660` | Max input duration (seconds) accepted by the vLLM plugin's audio mapper before it raises. |
| `VIBEVOICE_FFMPEG_MAX_CONCURRENCY` | `0` (no limit) | Cap on simultaneous FFmpeg processes. Set to ~2× CPU cores under heavy concurrent load. |
| `VIBEVOICE_USE_MEAN` | unset | When set (truthy), the vLLM plugin uses mean pooling over speech tokens instead of the default last-token strategy (issue #256). |
| `OPTIMIZE_FOR_SPEED` | `0` | When set to `1`, the modular tokenizer drops layer-scale and a few normalization steps for faster but slightly lower-quality acoustic reconstruction. |
| `USERNAME` | `user` | Used on Windows by the ASR Gradio demo to set the `USER` env var that some upstream libraries expect. |
| `TEMP` | `.` | Used on Windows by the ASR Gradio demo as the parent directory for `TORCHINDUCTOR_CACHE_DIR`. |

> [!NOTE]
> `MODEL_PATH` and `MODEL_DEVICE` are read by `demo/web/app.py` at FastAPI startup. They are *also* set programmatically by `demo/vibevoice_realtime_demo.py`, so for normal use you launch through that wrapper and pass `--model_path` / `--device` instead of exporting variables manually.

### Common option combinations

| Goal | Recommended flags |
|---|---|
| Fastest TTS on CUDA | `--device cuda --cfg_scale 1.5` (uses flash-attention-2 + bfloat16) |
| TTS on Apple Silicon | `--device mps --cfg_scale 1.5` (forces float32 + SDPA) |
| Highest-quality ASR (greedy) | `--temperature 0.0 --num_beams 1 --do_sample false` |
| Diverse ASR sampling | `--temperature 0.7 --top_p 0.9 --do_sample true` |
| Beam-search ASR | `--num_beams 4 --do_sample false` |
| Long-document ASR | `--max_new_tokens 32768 --max_duration 3600` |

> [!TIP]
> CFG scale (Classifier-Free Guidance) is the single biggest dial for TTS expressiveness. The default `1.5` is balanced; `1.0` is more literal/calm; `2.0`+ exaggerates emphasis but can introduce artefacts. Tune one notch at a time.

For full configuration mastery — every CLI flag, every JSON config knob, every internal env var — see the **Admin Guide**.

---

## 4. Core Usage

VibeVoice ships eight primary user-facing entry points. Use the index below to jump to the one you need.

| Goal | Entry point |
|---|---|
| Generate speech from a script file | `demo/realtime_model_inference_from_file.py` |
| Run a streaming TTS WebSocket server with browser UI | `demo/vibevoice_realtime_demo.py` |
| Transcribe audio files in a one-shot batch | `demo/vibevoice_asr_inference_from_file.py` |
| Browser UI for ASR with playback per segment | `demo/vibevoice_asr_gradio_demo.py` |
| Programmatic streaming TTS in Python | `from vibevoice import VibeVoiceStreamingForConditionalGenerationInference` |
| Programmatic ASR in Python | `from vibevoice.modular.modeling_vibevoice_asr import VibeVoiceASRForConditionalGeneration` |
| LoRA fine-tune the ASR model | `finetuning-asr/lora_finetune.py` |
| Inference with a LoRA adapter | `finetuning-asr/inference_lora.py` |

### 4.1 Generate speech from a script file

```bash
python demo/realtime_model_inference_from_file.py \
  --model_path microsoft/VibeVoice-Realtime-0.5B \
  --txt_path demo/text_examples/1p_vibevoice.txt \
  --speaker_name Carter \
  --output_dir ./outputs \
  --device cuda \
  --cfg_scale 1.5
```

| Flag | Type | Default | Purpose |
|---|---|---|---|
| `--model_path` | str | `microsoft/VibeVoice-Realtime-0.5B` | HF repo ID or local path. |
| `--txt_path` | str | `demo/text_examples/1p_vibevoice.txt` | Path to the script. UTF-8, plain text. |
| `--speaker_name` | str | `Wayne` | Voice preset selector, case-insensitive substring match. |
| `--output_dir` | str | `./outputs` | Output directory; auto-created. |
| `--device` | str | auto-detect | `cuda`, `mps`, `cpu`. `mpx` is silently treated as `mps`. |
| `--cfg_scale` | float | `1.5` | Classifier-Free Guidance scale. |

The script writes one `.wav` per input file: `<txt_filename>_generated.wav` at 24 kHz mono. It also prints token statistics, audio duration, and Real Time Factor on completion.

> [!NOTE]
> The voice mapper does substring matching: passing `--speaker_name Carter` matches `en-Carter_man`. If two presets share a substring (e.g. passing `Spk0`), the script raises with the message *"Multiple voice presets match the speaker name"* — pass a more specific name.

### 4.2 Stream TTS to a browser

`demo/vibevoice_realtime_demo.py` boots a FastAPI app (`demo/web/app.py`) that serves three endpoints — an HTML player at `/`, a JSON config endpoint at `/config`, and a WebSocket streamer at `/stream`.

```bash
python demo/vibevoice_realtime_demo.py \
  --port 3000 \
  --model_path microsoft/VibeVoice-Realtime-0.5B \
  --device cuda
```

Then visit `http://localhost:3000` in any modern browser.

| Flag | Default | Purpose |
|---|---|---|
| `--port` | `3000` | TCP port for uvicorn. |
| `--model_path` | `microsoft/VibeVoice-Realtime-0.5B` | Sets `MODEL_PATH` env var before app import. |
| `--device` | `cuda` | Sets `MODEL_DEVICE` (`cpu`, `cuda`, `mpx`, `mps`). |
| `--reload` | off | Enables uvicorn auto-reload (developer use only). |

Direct WebSocket protocol (for embedding in your own app):

```text
ws://host:3000/stream?text=<URL-encoded>&cfg=<float>&steps=<int>&voice=<preset_key>
```

The server responds with binary frames of 16-bit signed little-endian PCM at 24 kHz mono, interleaved with JSON log frames carrying events like `backend_request_received`, `model_progress` (with `generated_sec`, `chunk_sec`), `backend_first_chunk_sent`, and `backend_stream_complete`. A second client connecting while a generation is in progress receives a `backend_busy` JSON frame and the WebSocket is closed with code 1013 (Service busy).

> [!IMPORTANT]
> Only one TTS request runs concurrently per server instance — the FastAPI app holds an `asyncio.Lock` for the lifetime of each generation. Run multiple uvicorn workers (and load-balance) if you need parallelism.

### 4.3 Transcribe audio files (CLI)

```bash
python demo/vibevoice_asr_inference_from_file.py \
  --model_path microsoft/VibeVoice-ASR \
  --audio_files podcast.mp3 interview.wav \
  --max_new_tokens 32768 \
  --batch_size 2
```

| Flag | Type | Default | Purpose |
|---|---|---|---|
| `--model_path` | str | (required) | Path or HF repo ID for the ASR checkpoint. |
| `--audio_files` | str+ | none | One or more audio paths. Repeatable. |
| `--audio_dir` | str | none | Process every audio file under this directory. |
| `--dataset` | str | none | Stream a HuggingFace dataset (e.g. `openslr/librispeech_asr`). |
| `--split` | str | `test` | Dataset split name. |
| `--max_duration` | float | `3600.0` | Max seconds per concatenated dataset audio. |
| `--batch_size` | int | `2` | Files per inference batch. |
| `--device` | str | auto | `cuda`, `cpu`, `mps`, `xpu`, or `auto`. |
| `--max_new_tokens` | int | `32768` | Generation cap. |
| `--temperature` | float | `0.0` | Sampling temperature; `0` selects greedy. |
| `--top_p` | float | `1.0` | Nucleus sampling threshold. |
| `--num_beams` | int | `1` | Beam-search width. `1` = greedy/sampling. |
| `--attn_implementation` | str | `auto` | `flash_attention_2`, `sdpa`, `eager`, `auto`. |

Output is printed per file — both raw text and JSON-parsed segments with timestamps and speaker IDs.

### 4.4 Run the Gradio ASR browser app

```bash
python demo/vibevoice_asr_gradio_demo.py \
  --model_path microsoft/VibeVoice-ASR \
  --host 0.0.0.0 \
  --port 7860
```

| Flag | Default | Purpose |
|---|---|---|
| `--model_path` | `microsoft/VibeVoice-ASR` | Checkpoint path. |
| `--attn_implementation` | `flash_attention_2` | Attention backend. |
| `--max_new_tokens` | `32768` | Generation cap shown in the UI as a slider. |
| `--host` | `0.0.0.0` | Bind interface. |
| `--port` | `7860` | Bind port. |
| `--share` | off | When set, Gradio creates a public tunnel. |

The UI provides:
- Audio upload + microphone recording.
- Optional **Context Info** textbox for hotwords ("John Smith", "Dr. Tan", "Acme Corp") that bias transcription.
- Sampling toggle with Temperature, Top-p, and Repetition Penalty sliders.
- Per-segment playback chips (each segment is clipped, optionally re-encoded as 32 kbps MP3 for compactness, and embedded inline as a data URI).
- A red **Stop** button that interrupts generation mid-stream via a custom `StoppingCriteria`.

> [!TIP]
> The Gradio demo applies Liger Kernel to Qwen2 (RoPE + RMSNorm + SwiGLU) when available — install `liger-kernel` for a meaningful speedup. If absent, the demo prints a warning and continues.

### 4.5 Use the Python API directly

#### Streaming TTS (synchronous chunks)

```python
import torch, copy
from vibevoice.modular.modeling_vibevoice_streaming_inference import (
    VibeVoiceStreamingForConditionalGenerationInference,
)
from vibevoice.processor.vibevoice_streaming_processor import VibeVoiceStreamingProcessor

processor = VibeVoiceStreamingProcessor.from_pretrained("microsoft/VibeVoice-Realtime-0.5B")
model = VibeVoiceStreamingForConditionalGenerationInference.from_pretrained(
    "microsoft/VibeVoice-Realtime-0.5B",
    torch_dtype=torch.bfloat16,
    device_map="cuda",
    attn_implementation="flash_attention_2",
).eval()
model.set_ddpm_inference_steps(num_steps=5)

prefilled = torch.load("demo/voices/streaming_model/en-Carter_man.pt",
                      map_location="cuda", weights_only=False)
inputs = processor.process_input_with_cached_prompt(
    text="Welcome to VibeVoice, a streaming TTS engine.",
    cached_prompt=prefilled,
    padding=True, return_tensors="pt", return_attention_mask=True,
)
inputs = {k: v.to("cuda") if torch.is_tensor(v) else v for k, v in inputs.items()}

outputs = model.generate(
    **inputs,
    max_new_tokens=None,
    cfg_scale=1.5,
    tokenizer=processor.tokenizer,
    generation_config={"do_sample": False},
    verbose=True,
    all_prefilled_outputs=copy.deepcopy(prefilled),
)
processor.save_audio(outputs.speech_outputs[0], output_path="hello.wav")
```

#### One-shot ASR

```python
import torch
from vibevoice.modular.modeling_vibevoice_asr import VibeVoiceASRForConditionalGeneration
from vibevoice.processor.vibevoice_asr_processor import VibeVoiceASRProcessor

processor = VibeVoiceASRProcessor.from_pretrained(
    "microsoft/VibeVoice-ASR",
    language_model_pretrained_name="Qwen/Qwen2.5-7B",
)
model = VibeVoiceASRForConditionalGeneration.from_pretrained(
    "microsoft/VibeVoice-ASR",
    dtype=torch.bfloat16,
    attn_implementation="flash_attention_2",
    trust_remote_code=True,
).to("cuda").eval()

inputs = processor(audio="podcast.mp3", return_tensors="pt", add_generation_prompt=True)
inputs = {k: v.to("cuda") if torch.is_tensor(v) else v for k, v in inputs.items()}

with torch.no_grad():
    out = model.generate(
        **inputs, max_new_tokens=32768, do_sample=False, num_beams=1,
        pad_token_id=processor.pad_id,
        eos_token_id=processor.tokenizer.eos_token_id,
    )

text = processor.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True)
segments = processor.post_process_transcription(text)
for seg in segments:
    print(f"[{seg['Start']:.2f} - {seg['End']:.2f}] Speaker {seg.get('Speaker', '?')}: {seg['Content']}")
```

### 4.6 LoRA fine-tune the ASR model

```bash
python finetuning-asr/lora_finetune.py \
  --model_path microsoft/VibeVoice-ASR \
  --data_dir ./my_dataset \
  --output_dir ./output \
  --num_train_epochs 3 \
  --per_device_train_batch_size 1 \
  --gradient_accumulation_steps 8 \
  --learning_rate 1e-4 \
  --bf16 true \
  --logging_steps 10 \
  --save_steps 100 \
  --lora_r 16 --lora_alpha 32 --lora_dropout 0.05
```

The script uses Hugging Face `Trainer` with right-padded batches via `VibeVoiceASRDataCollator`. Audio + JSON labels live side-by-side in `data_dir`:

```
my_dataset/
  0.mp3
  0.json
  1.flac
  1.json
  ...
```

JSON schema:

```json
{
  "audio_path": "0.mp3",
  "audio_duration": 351.73,
  "segments": [
    {"speaker": 0, "text": "Hey everyone...", "start": 0.0, "end": 38.68},
    {"speaker": 1, "text": "Thanks for having me.", "start": 38.68, "end": 42.10}
  ],
  "customized_context": ["Tea Brew", "Aiden Host"]
}
```

LoRA adapters target the Qwen2 attention/MLP projections (`q_proj`, `k_proj`, `v_proj`, `o_proj`, `gate_proj`, `up_proj`, `down_proj`); the acoustic and semantic tokenizers are frozen. After training, run inference via `inference_lora.py`:

```bash
python finetuning-asr/inference_lora.py \
  --base_model microsoft/VibeVoice-ASR \
  --lora_path ./output \
  --audio_file finetuning-asr/toy_dataset/0.mp3 \
  --context_info "Hotwords: Tea Brew, Aiden Host"
```

> [!WARNING]
> The default `lora_finetune.py` configuration is tuned for the toy dataset (`finetuning-asr/toy_dataset/`). For production runs you must override `learning_rate`, `num_train_epochs`, `per_device_train_batch_size`, `gradient_accumulation_steps`, `warmup_ratio`, and `weight_decay` via Hugging Face `TrainingArguments`. The script accepts every standard `TrainingArguments` flag.

---

## 5. Common Tasks

### Task 1 — Generate a single-speaker narration WAV

**Goal:** Turn a script of fewer than 64K tokens into a 24kHz WAV in a chosen voice.

**Steps:**

1. Place the script as a UTF-8 `.txt` file.
2. Pick a voice from `demo/voices/streaming_model/` — for English, `Carter`, `Davis`, `Emma`, `Frank`, `Grace`, `Mike`.
3. Run `python demo/realtime_model_inference_from_file.py --txt_path script.txt --speaker_name Emma --output_dir ./out`.
4. Listen to `./out/script_generated.wav`.

**If it fails:** check VRAM (the script halves it for `bfloat16` weights + KV cache) and re-run with a shorter script.

### Task 2 — Build a podcast transcript with speaker labels

```bash
python demo/vibevoice_asr_inference_from_file.py \
  --model_path microsoft/VibeVoice-ASR \
  --audio_files podcast.mp3 \
  --max_new_tokens 32768 \
  --num_beams 1 \
  --temperature 0.0
```

Pipe the JSON segments to a downstream tool (subtitle generator, summarizer):

```python
import json, subprocess
text = subprocess.check_output(["python", "demo/vibevoice_asr_inference_from_file.py",
                                "--model_path", "microsoft/VibeVoice-ASR",
                                "--audio_files", "podcast.mp3"], text=True)
# Pull the JSON block out of stdout (the script logs it on completion).
```

For programmatic use prefer the Python API in section 4.5 — it returns parsed segments directly.

### Task 3 — Improve transcription accuracy with hotwords

Industry jargon and proper nouns are the most common ASR failure mode. Pass a `--context_info` block with one term per line:

```bash
python demo/vibevoice_asr_gradio_demo.py --model_path microsoft/VibeVoice-ASR
# Then in the UI's "Context Info" textbox:
# Acme Corp
# Dr. Tan
# Continuous integration
# kubectl
```

The CLI demos accept the same idea via the model API but the Gradio UI is the easiest place to iterate.

### Task 4 — Use a non-default voice without renaming files

`--speaker_name` is a substring match on the preset's stem. To target `kr-Spk1_man` either pass `--speaker_name kr-Spk1_man` or any unambiguous substring like `kr-Spk1`. To force a default in the streaming server, set `VOICE_PRESET=de-Spk0_man` before launching `demo/vibevoice_realtime_demo.py`.

### Task 5 — Run on Apple Silicon

```bash
python demo/realtime_model_inference_from_file.py \
  --model_path microsoft/VibeVoice-Realtime-0.5B \
  --txt_path demo/text_examples/1p_vibevoice.txt \
  --speaker_name Emma \
  --device mps
```

The script automatically forces `torch.float32` on MPS (MPS does not support `bfloat16` reliably) and uses SDPA attention. Performance is significantly slower than CUDA + flash-attention-2; expect RTF > 1.0.

> [!WARNING]
> If you pass `--device mpx` the scripts treat it as `mps`. This is a deliberate compatibility shim for documentation typos — there is no `mpx` device. If MPS is unavailable, the scripts auto-fall-back to CPU and print a warning.

### Task 6 — Process a HuggingFace dataset

```bash
python demo/vibevoice_asr_inference_from_file.py \
  --model_path microsoft/VibeVoice-ASR \
  --dataset openslr/librispeech_asr \
  --split test.clean \
  --max_duration 600 \
  --batch_size 2
```

The script streams the dataset (no full download), concatenates samples up to `max_duration` per inference call, and yields one transcription block per concatenated audio.

### Task 7 — Embed streaming TTS in a Node.js or Python client

Open a WebSocket to `/stream` on the running TTS server and feed PCM bytes to a Web Audio context (browser) or a `sounddevice.RawOutputStream(samplerate=24000, dtype="int16", channels=1)` (Python). JSON log frames arrive on the same socket — distinguish them by checking whether the WebSocket message is text or binary.

```python
import asyncio, websockets, sounddevice as sd

async def play(text):
    uri = f"ws://localhost:3000/stream?text={text}&cfg=1.5&voice=en-Carter_man"
    async with websockets.connect(uri) as ws:
        with sd.RawOutputStream(samplerate=24000, dtype="int16", channels=1) as out:
            async for msg in ws:
                if isinstance(msg, bytes):
                    out.write(msg)
                # else: msg is a JSON log frame, ignore or parse
asyncio.run(play("Hello from VibeVoice"))
```

### Task 8 — Increase concurrent ASR throughput with vLLM

The `vllm_plugin/` package registers VibeVoice with vLLM through a `general_plugins` entry point. Launch a vLLM server pointing at `microsoft/VibeVoice-ASR` and your existing client code (curl, OpenAI SDK, etc.) gets multimodal audio input automatically. The plugin honors `VIBEVOICE_MAX_AUDIO_DURATION` and `VIBEVOICE_FFMPEG_MAX_CONCURRENCY` to keep memory and FFmpeg fork pressure under control. See the **Admin Guide → Operations** section for cluster sizing.

### Task 9 — Quickly test the toy dataset

```bash
python finetuning-asr/inference_lora.py \
  --base_model microsoft/VibeVoice-ASR \
  --lora_path microsoft/VibeVoice-ASR \
  --audio_file finetuning-asr/toy_dataset/0.mp3
```

This requires no real LoRA weights — passing the base model as `--lora_path` falls through PEFT's loader for a no-op adapter.

### Task 10 — Inspect or replace the noise scheduler

The streaming TTS server reconfigures its scheduler at startup to use `sde-dpmsolver++` with `squaredcos_cap_v2`. To reproduce in your own code:

```python
model.model.noise_scheduler = model.model.noise_scheduler.from_config(
    model.model.noise_scheduler.config,
    algorithm_type="sde-dpmsolver++",
    beta_schedule="squaredcos_cap_v2",
)
model.set_ddpm_inference_steps(num_steps=5)   # 5 is the default; raise to 10–20 for higher fidelity at the cost of latency
```

### Task 11 — Stop a long generation gracefully

The Gradio ASR demo wires a `StopOnFlag()` `StoppingCriteria` to a global flag toggled by the red **Stop** button. When you embed VibeVoice in your own app, install the same pattern:

```python
import threading
from transformers import StoppingCriteria, StoppingCriteriaList

stop = threading.Event()
class StopOnFlag(StoppingCriteria):
    def __call__(self, input_ids, scores, **kwargs): return stop.is_set()

model.generate(..., stopping_criteria=StoppingCriteriaList([StopOnFlag()]))
# From another thread: stop.set()
```

The streaming TTS path accepts a similar pattern via `stop_check_fn=stop_event.is_set` on `model.generate(...)`.

### Task 12 — Process audio segments larger than a single batch

Use `transcribe_with_batching` from `VibeVoiceASRBatchInference` (in `demo/vibevoice_asr_inference_from_file.py`) — it splits a list of files into batches of size `--batch_size`, generates per-batch, and concatenates results. For audio individually longer than the model's context window, pre-segment with FFmpeg first (e.g. `ffmpeg -i in.mp3 -f segment -segment_time 1800 out_%03d.mp3`).

---

## 6. Troubleshooting

### Decision tree

1. Does `python -c "from vibevoice import *"` import cleanly? — **No** → install issue (section 2).
2. Does `ffmpeg -version` print? — **No** → install FFmpeg.
3. Does `nvidia-smi` show a GPU and free VRAM ≥ 16GB for ASR / 8GB for TTS? — **No** → switch `--device cpu` (slow) or `--device mps`.
4. Did the script print "Found N voice presets"? — **No** → run `git lfs pull` from repo root.
5. Did the model load with `flash_attention_2`? — **No** → either `pip install flash-attn` or accept the SDPA fallback.
6. Did the WebSocket close with code 1013? — server is busy with another generation; wait or run more workers.

### Error: `RuntimeError: MODEL_PATH not set in environment`

**Cause:** `demo/web/app.py` reads `MODEL_PATH` at FastAPI startup, but the user launched `uvicorn web.app:app` directly without exporting the variable.

**Solution:**
1. Use the wrapper: `python demo/vibevoice_realtime_demo.py --model_path microsoft/VibeVoice-Realtime-0.5B`.
2. Or export the variables manually: `MODEL_PATH=microsoft/VibeVoice-Realtime-0.5B MODEL_DEVICE=cuda uvicorn web.app:app`.

**Prevention:** Always launch via `vibevoice_realtime_demo.py`; it sets both env vars before importing the app.

### Error: `RuntimeError: Voices directory not found`

**Cause:** `demo/web/app.py` resolves voices from `BASE.parent / "voices" / "streaming_model"`, where `BASE = demo/web/`. If the repo was cloned without Git LFS the `.pt` files are missing.

**Solution:**
1. `cd VibeVoice && git lfs install && git lfs pull`.
2. Verify with `ls demo/voices/streaming_model/*.pt`.

**Prevention:** Add `git lfs pull` to your bootstrap script.

### Error: `RuntimeError: No voice preset (.pt) files found in <dir>`

**Cause:** The voices directory exists but is empty (LFS pointer files only).

**Solution:** Same as above — `git lfs pull`.

### Error: `RuntimeError: Voice preset 'foo' not found`

**Cause:** `_ensure_voice_cached` was passed a `voice` query parameter (or `--speaker_name`) that does not match any preset stem.

**Solution:** Hit `GET /config` to list available voice keys, or `ls demo/voices/streaming_model/`. Use the exact stem (e.g. `en-Carter_man`).

### Error: `RuntimeError: StreamingTTSService not initialized`

**Cause:** `_prepare_inputs` was called before `service.load()` finished. Usually means the FastAPI app received a request during startup.

**Solution:** Wait for the `[startup] Model ready.` log line before sending traffic.

### Error: `ValueError: Multiple voice presets match the speaker name 'X'`

**Cause:** The `VoiceMapper.get_voice_path()` substring match found two presets containing `X`.

**Solution:** Use a more specific name. For example, `Spk0` matches every language pack; use `de-Spk0_man` instead.

### Error: `ValueError: Audio duration (Ns) exceeds the configured limit (3660s)`

**Cause:** The vLLM plugin enforces a hard duration cap to prevent OOM (issue #210).

**Solution:** Either pre-split the audio (`ffmpeg -i input.mp3 -f segment -segment_time 1800 part_%03d.mp3`) or raise the limit: `export VIBEVOICE_MAX_AUDIO_DURATION=7200`.

**Prevention:** Profile peak memory at your target duration before raising the limit.

### Error: `Token indices sequence length is longer than the specified maximum sequence length for this model`

**Cause:** The TTS script you supplied tokenizes to more than the decoder context (issue #332). The 1.5B-64K config supports 65,536 tokens; the 7B-32K supports 32,768.

**Solution:** Split the script into chunks and call the model per chunk, then concatenate WAVs. The streaming demo already streams; for the file-based demo, slice the input.

### Error: `ImportError: soundfile is required to save audio files`

**Cause:** `processor.save_audio()` in `vibevoice_tokenizer_processor.py` falls back to `soundfile` and the package is not installed.

**Solution:** `pip install soundfile`.

### Error: `Error loading the model. Trying to use SDPA. However, note that only flash_attention_2 has been fully tested...`

**Cause:** `flash-attn` failed at import time. Common on Windows or older PyTorch wheels.

**Solution:** This is a **warning**, not a hard failure. The model still loads with SDPA. Audio quality is "lower" per the project authors but typically still acceptable. To regain flash-attn install it from source: `pip install flash-attn --no-build-isolation`.

### Error: `Connection closed with code 1013 (Service busy)`

**Cause:** Another client already holds the server's TTS lock (`asyncio.Lock`).

**Solution:** Wait for the in-flight stream to complete, or run a uvicorn worker pool fronted by a load balancer.

### Error: Gradio ASR demo crashes when you connect over plain HTTP through Docker (#339)

**Cause:** Gradio rewrites internal `/file=` URLs based on the request scheme. Behind some HTTP-only proxies the rewrite fails.

**Solution:** Front the demo with HTTPS, or expose the Gradio port directly without rewriting.

### Error: Constant crash on Windows 11 (#327)

**Cause:** Mixed CUDA driver / VC++ runtime / PyTorch wheel ABIs.

**Solution:** Re-install the latest NVIDIA driver, latest **VC++ Redistributable**, and a PyTorch wheel matching your CUDA version (`pip install torch --index-url https://download.pytorch.org/whl/cu121`).

### Error: `ValueError: Unsupported audio data type: <class '...'>`

**Cause:** The vLLM plugin's audio mapper accepts `str` (path), `bytes`, `np.ndarray`, or `List[str]`. Anything else raises.

**Solution:** Convert your input to one of the four supported types before calling.

### Error: `ValueError: Unsupported file format: .X. Supported formats: .wav, .mp3, .flac, .m4a, .ogg, .pt, .npy, .npz`

**Cause:** `vibevoice_tokenizer_processor.py` rejects unknown extensions when loading audio.

**Solution:** Convert with FFmpeg: `ffmpeg -i input.amr -ar 24000 input.wav`.

---

## 7. FAQ

**Q: What languages does VibeVoice support?**

The shipped voice presets cover German, English, Spanish, French, Italian, Indian English, Japanese, Korean, Dutch, Polish, and Portuguese. ASR can transcribe any language Qwen2.5-7B understands, but accuracy varies by language. Arabic (#108, #253), Vietnamese (#283), and Traditional Chinese (#271) are open community feature requests at the time of this writing.

**Q: Is VibeVoice open source?**

The code is MIT-licensed (see `LICENSE`). Pretrained weights are distributed via Hugging Face under their own terms. See `LICENSE` and the project's `SECURITY.md` for the full picture.

**Q: Can I run VibeVoice without a GPU?**

Yes — pass `--device cpu`. Expect Real Time Factor far above `1.0` (i.e. generation is slower than playback). The 0.5B streaming model is more usable on CPU than the 7B ASR model.

**Q: How do I get speaker diarization in ASR output?**

It is automatic. The ASR model emits JSON segments with `Start`, `End`, `Speaker` (integer ID), and `Content`. Call `processor.post_process_transcription(text)` to get the parsed list.

**Q: How do I bias transcription toward a known vocabulary?**

Pass `context_info` (CLI: `--context_info`) with one hotword per line. The processor injects it into the system prompt; the LLM treats it as a hint. The Gradio demo exposes a textbox; the LoRA finetune script reads `customized_context` from each JSON label.

**Q: What sample rate does VibeVoice use?**

24 kHz mono throughout. All inputs are resampled to 24 kHz at load time and outputs are produced at 24 kHz. The streaming server emits 16-bit PCM at 24 kHz.

**Q: How do I add a new voice preset?**

Run `processor.encode_voice_to_pt(audio_path, output_path)` (or follow the encoding scripts under `vllm_plugin/tools/`). Drop the resulting `.pt` file into `demo/voices/streaming_model/` and it will be discovered at server startup.

**Q: Why does streaming TTS sometimes hang for several seconds before playing?**

The first chunk requires the model's prefill pass to complete. Subsequent chunks stream every few hundred milliseconds. Check the WebSocket log frames: `backend_request_received` to `model_progress` (first instance) is your time-to-first-byte.

**Q: Why does `--cfg_scale 1.0` sound flat?**

CFG below `1.5` reduces guidance strength, biasing toward the unconditional voice prior. `1.5` is the project default; raise to `2.0` for more emphatic delivery, lower for a calmer read.

**Q: Can I use VibeVoice commercially?**

The code is MIT. The pretrained weight licenses live on the Hugging Face model pages — read them carefully before commercial deployment. As of this writing the project explicitly publishes `microsoft/VibeVoice-Realtime-0.5B` and `microsoft/VibeVoice-ASR`; check the model cards for use restrictions.

**Q: Is there a JavaScript or REST API?**

Yes — the streaming server speaks **WebSocket** (`/stream`) and **HTTP GET** (`/`, `/config`). For ASR, the vLLM plugin adds OpenAI-compatible REST endpoints (`/v1/chat/completions` with multimodal audio input). Issue #336 tracks deeper OpenAI API parity. There is no first-party REST endpoint for ASR shipped in this repo — use the vLLM plugin.

**Q: How much GPU memory do I need?**

The 0.5B streaming TTS model needs ~4-6 GB at bfloat16 + cache. The 7B ASR model needs ~16 GB at bfloat16 for a single-batch 1-hour audio run. Multi-batch inference scales linearly.

**Q: Where do generated files end up?**

The file-based TTS demo writes to `--output_dir` (default `./outputs`). The Gradio ASR demo embeds segment audio inline as data URIs (no disk writes per segment). Issue #328 tracks user confusion about file locations.

**Q: Does VibeVoice support voice cloning?**

The streaming TTS path uses **voice presets** that are pre-encoded `.pt` files. The repo ships 23 such presets, plus optional experimental packs. Encoding new voices requires running the audio through the tokenizer pipeline; see the **Admin Guide → Developer Internals** for the workflow.

**Q: Can I quantize the model?**

Issue #297 requests GGUF Q-quantization. As shipped the codebase exercises bfloat16 (CUDA), float32 (MPS/CPU), and the underlying Qwen2 backbone supports the standard `bitsandbytes` 4-bit/8-bit modes via `transformers.BitsAndBytesConfig`. Quantization is community-driven, not first-party.

**Q: Why does the Gradio demo hide the sampling sliders by default?**

`do_sample=False` (greedy) is the recommended default for ASR. Sampling sliders only appear when **Enable Sampling** is checked, to avoid configuration drift in production.

**Q: How do I report a security issue?**

`SECURITY.md` lays out the disclosure path. Issue #341 noted the original guidance was unclear; check the latest copy on `main`.

---

_Generated April 2026 from microsoft/VibeVoice at 4a78d3e. AI-generated documentation. The library is checked by sampling (three repositories so far); this guide was not individually reviewed. Verify against the project before relying on it. Unofficial; not affiliated with or reviewed by the project._