VibeVoice Administrator & Developer Guide

VibeVoice is Microsoft's open-source TTS + ASR research stack. This guide covers system architecture, full configuration reference, deployment + scaling guidance, security posture, performance tuning, developer internals, and integration points. It is written for operators running VibeVoice in a service, developers extending the codebase, and architects evaluating the design.

For end-user usage, scripts, and recipes, see the VibeVoice User Guide.

1. Architecture Overview

VibeVoice is a two-track speech model packaged as a single Python distribution. One track produces speech (TTS), the other consumes it (ASR). Both tracks share an identity — a Qwen2.5 decoder-only LLM backbone wrapped by VibeVoice's own acoustic and semantic tokenizers, plus a diffusion head — but expose different generation modes through different model classes and processors.

High-level dataflow

                            +-----------------------------------------------+
                            |                  TTS Path                     |
                            |                                               |
   text  -> Streaming   ->  | VibeVoiceStreamingProcessor                   |
            Processor        |   - apply_chat_template (Qwen2 tokenizer)   |
                            |   - cached prompt (.pt voice preset)         |
                            |                                               |
                            | VibeVoiceStreamingForConditionalGeneration-  |
                            | Inference                                    |
                            |   - language_model (Qwen2)                   |
                            |   - tts_language_model (parallel head)       |
                            |   - acoustic / semantic tokenizers           |
                            |   - diffusion head + DPMSolverMultistep      |
                            |                                               |
                            |   AudioStreamer (per-batch queue)            |
                            +-----------------------------------------------+
                                              |
                                              v
                                          PCM bytes (24 kHz s16le mono)
                                              ^
                                              |
                            +-----------------------------------------------+
                            |                  ASR Path                     |
                            |                                               |
   audio -> FFmpeg -> 24kHz | VibeVoiceASRProcessor                         |
            stdin pipe       |   - load + normalize audio                   |
                            |   - VAE-tokenize speech (semantic + acoustic)|
                            |   - context_info -> system prompt            |
                            |                                               |
                            | VibeVoiceASRForConditionalGeneration         |
                            |   - LLM-only generate over multimodal seq    |
                            |   - speech tokens injected via               |
                            |     acoustic_input_mask                      |
                            |                                               |
                            |   post_process_transcription (JSON parse)    |
                            +-----------------------------------------------+
                                              |
                                              v
                                  segments [{Start, End, Speaker, Content}]

Both paths share vibevoice/processor/audio_utils.py, vibevoice/modular/configuration_*.py, and the underlying vibevoice/modular/modular_vibevoice_tokenizer.py (acoustic + semantic VAEs). They diverge at the model class — TTS uses streaming inference with a diffusion head; ASR uses LLM generation with multimodal speech embeddings.

Note

The unified VibeVoiceStreamingModel.forward() is intentionally disabled (raises RuntimeError). All inference goes through model.language_model(...), model.tts_language_model(...), or model.generate(...). This is a design invariant that prevents callers from bypassing the streaming machinery accidentally.

Component inventory

Technology stack

Data flow — streaming TTS

1. HTTP/WS request lands at demo/web/app.py:websocket_stream. Query params: text, cfg, steps, voice.
2. The endpoint acquires the per-app asyncio.Lock. Concurrent clients are rejected with WS code 1013.
3. StreamingTTSService.stream() runs the full pipeline:
   • Look up the voice preset (_get_voice_resources).
   • Run VibeVoiceStreamingProcessor.process_input_with_cached_prompt(...) — produces token IDs and attention mask.
   • Spawn a background threading.Thread that calls model.generate(...) with an AudioStreamer(batch_size=1) sink.
   • The generator yields decoded float32 chunks; the WS handler converts to int16 PCM via chunk_to_pcm16 and writes binary frames.
4. JSON log frames are enqueued into a queue.Queue from the worker thread and flushed to the WS by the main coroutine between binary writes.
5. Stop is cooperative: a threading.Event is passed in via stop_check_fn so generation aborts on disconnect.

Data flow — long-form ASR

1. Client supplies an audio file path, bytes, ndarray, or HF-dataset name.
2. audio_utils.load_audio_use_ffmpeg() shells out to ffmpeg -loglevel error -nostdin ... -f s16le -ac 1 -ar 24000 - to produce 24 kHz mono float32. Concurrency is throttled by _FFMPEG_SEM (a threading.Semaphore gated on VIBEVOICE_FFMPEG_MAX_CONCURRENCY).
3. AudioNormalizer applies a target dB-FS scaling.
4. VibeVoiceASRProcessor builds the prompt:
   • System prompt (with optional context_info hotword block).
   • Speech token region delimited by <|object_ref_start|>...<|object_ref_end|> (or by the corresponding IDs in the fast tokenizer).
   • Generation prompt (<|im_start|>assistant\n).
5. VibeVoiceASRForConditionalGeneration.generate() runs greedy / sampled / beam-search decoding.
6. Output text contains a JSON array of {Start, End, Speaker, Content} objects; processor.post_process_transcription() parses it.

Note

Both tokenizers (acoustic and semantic) are frozen during LoRA fine-tuning. Only the Qwen2 attention and MLP projections receive adapter updates. This keeps the speech-token vocabulary stable across fine-tunes — important for swapping LoRA adapters at inference time.

2. System Requirements

Hardware

Software

Network

The repo's only outbound dependencies are:

• Hugging Face Hub (model + processor downloads).
• GitHub release attachments (experimental voice packs in download_experimental_voices.sh).

When deploying behind a firewall, mirror both into your registry and pass local paths to --model_path.

3. Configuration Reference

3.1 CLI flags — every argument across every entry point

3.2 Environment variables

Important

Two of these variables silently change runtime behavior with no log output: OPTIMIZE_FOR_SPEED=1 and VIBEVOICE_USE_MEAN. Set them only after benchmarking — a default deployment should leave both unset.

3.3 Model JSON configs (read by from_pretrained)

Two reference configs ship in vibevoice/configs/. They define the architecture; the model.safetensors carries the weights.

qwen2.5_1.5b_64k.json — used by streaming TTS

qwen2.5_7b_32k.json — used by long-form ASR

Identical structure; key deltas:

The 7B config also sets use_mrope: false on the decoder.

3.4 LoRA configuration (finetuning-asr/lora_finetune.py)

The script forces dataloader_num_workers = 0 and remove_unused_columns = False regardless of TrainingArguments user input, because audio loading is not multiprocessing-safe in this code path.

3.5 HTTP API surface (streaming TTS, demo/web/app.py)

There is no auth, no rate limit, and no CORS configuration baked in — the demo is intended to run behind your own gateway.

Warning

Do not expose demo/web/app.py directly to the public internet. It has zero authentication and a single global lock, so anyone can both (a) tie up your GPU with a long script and (b) trivially DoS the service.

4. Deployment & Scaling

4.1 Single-host streaming TTS

The simplest deployment is a single CUDA host running:

MODEL_PATH=microsoft/VibeVoice-Realtime-0.5B \
MODEL_DEVICE=cuda \
VOICE_PRESET=en-Carter_man \
python demo/vibevoice_realtime_demo.py --port 3000

Front this with nginx for TLS termination + WebSocket proxying. Example nginx block:

upstream vibevoice {
    server 127.0.0.1:3000;
}
server {
    listen 443 ssl http2;
    location / {
        proxy_pass http://vibevoice;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_read_timeout 300s;
        proxy_send_timeout 300s;
    }
}

Important

The default WebSocket idle timeout in many proxies is 60s — too short for long generations. Bump proxy_read_timeout and proxy_send_timeout to at least the longest TTS request you expect (5 minutes is a safe default for read-aloud workloads).

4.2 Horizontal scaling — TTS

Each FastAPI process holds a single global asyncio.Lock. Concurrency therefore equals number of processes (not threads). Two patterns:

• Worker pool: launch --workers N uvicorn processes against the same socket, each with its own model copy. Memory cost: N × ~6 GB on bfloat16. Best when you have a single large GPU and want N concurrent streams.
• Multi-host: deploy several boxes behind a sticky load balancer (Layer 7 with cookie or path-based routing). The streaming WebSocket holds state across messages, so sticky routing is required.

4.3 vLLM serving for ASR

For high-throughput ASR, register the plugin and launch vLLM:

pip install vllm
python -m vllm.entrypoints.openai.api_server \
  --model microsoft/VibeVoice-ASR \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.85 \
  --max-num-seqs 8

The plugin auto-loads via pyproject.toml's [project.entry-points."vllm.general_plugins"] table and registers:

• VibeVoiceConfig with AutoConfig.
• VibeVoiceASRTextTokenizerFast (fast) and Qwen2Tokenizer (slow) with AutoTokenizer.
• Qwen2AudioProcessor with AutoProcessor.
• VibeVoiceForCausalLM with vLLM's ModelRegistry under names VibeVoice and VibeVoiceForASRTraining.

Audio inputs come in via the multi_modal_data parameter on the OpenAI-compatible request (audio/file/bytes/array, see vllm_plugin/inputs.py).

Note

Throughput at 8 concurrent requests on an A100-80GB approaches ~3-4× single-stream PyTorch baseline, mostly from vLLM's continuous batching.

4.4 Sizing the GPU

4.5 Container deployment

There is no first-party Dockerfile in the repo. A minimal recipe:

FROM nvidia/cuda:12.1.0-cudnn8-devel-ubuntu22.04
RUN apt-get update && apt-get install -y python3.10 python3-pip ffmpeg git git-lfs
WORKDIR /app
COPY . /app
RUN pip install -e . && \
    pip install transformers peft accelerate diffusers \
    soundfile pydub fastapi uvicorn[standard] websockets gradio && \
    pip install flash-attn --no-build-isolation
RUN git lfs install && git lfs pull
ENV MODEL_PATH=microsoft/VibeVoice-Realtime-0.5B \
    MODEL_DEVICE=cuda \
    VIBEVOICE_FFMPEG_MAX_CONCURRENCY=4
EXPOSE 3000
CMD ["python", "demo/vibevoice_realtime_demo.py", "--port", "3000"]

Caution

Issue #339 reports the Gradio ASR demo crashing under HTTP-only Docker setups due to URL scheme rewriting. Always front the Gradio demo with TLS in container environments.

5. Operations

5.1 Health checks

# Streaming TTS
curl -fsS http://localhost:3000/config

# Expected
# {"voices": ["de-Spk0_man", ..., "en-Carter_man", ...], "default_voice": "en-Carter_man"}

The /config endpoint dereferences service.voice_presets, which is only populated after service.load() completes. A 500/disconnect therefore indicates the model has not finished loading.

Tip

For Kubernetes, use httpGet with /config for both readinessProbe and livenessProbe. Allow at least 120 seconds for the initial model load before the readiness probe starts.

5.2 Logs

The streaming TTS server emits structured stdout logs:

• [startup] Loading processor from <path> — early init.
• [startup] Found N voice presets — voice scan complete.
• [startup] Loading prefilled prompt from <path> — voice cached.
• [startup] Model ready. — service is accepting traffic.
• Per-request: Client connected, text='...' and JSON event frames over WebSocket (backend_request_received, model_progress, backend_first_chunk_sent, backend_stream_complete, client_disconnected, backend_error).

Pipe to a log aggregator that handles structured JSON; the WebSocket events are valuable telemetry.

5.3 Metrics worth tracking

5.4 Backup & state

VibeVoice is stateless. Voice presets and model weights are content-addressable (Hugging Face hashes); nothing per-tenant lives on disk. The only state worth checkpointing is:

• LoRA adapter weights (after fine-tuning) → push to HF Hub or copy to S3.
• Custom voice presets (.pt files) → copy to S3.

5.5 Runbook checklist for a new deployment

• [ ] Provision a GPU host with CUDA 12.1+ and FFmpeg installed.
• [ ] Clone the repo and git lfs pull to fetch voice presets.
• [ ] Install the package + flash-attn.
• [ ] Pre-warm microsoft/VibeVoice-Realtime-0.5B (or your local copy) via huggingface-cli download.
• [ ] Set MODEL_PATH, MODEL_DEVICE, VOICE_PRESET, VIBEVOICE_FFMPEG_MAX_CONCURRENCY.
• [ ] Launch through demo/vibevoice_realtime_demo.py (or an equivalent supervisord / systemd unit).
• [ ] Wait for [startup] Model ready. before flipping the load balancer.
• [ ] Verify GET /config returns 200 with the expected voice list.
• [ ] Verify a short WebSocket TTS round-trip (10-second script) under target latency.
• [ ] Configure Prometheus / Grafana alerts on per-request latency p95.

Tip

Pre-warm the voice cache by hitting /stream with text=Hi&voice=<preset> for every preset you expect to use. The first request per voice triggers a torch.load(.pt, weights_only=False) from disk; subsequent requests reuse the cached object.

5.6 Upgrade procedure

1. Snapshot any LoRA adapters and custom voices.
2. git fetch && git checkout <new-tag> in a parallel directory.
3. Run a smoke test: python demo/realtime_model_inference_from_file.py ... against the new code.
4. Cut traffic over (DNS or load-balancer flip).
5. Keep the old install for 24h as a rollback.

6. Performance

6.1 TTS latency budget (CUDA + flash-attention-2, RTX 4090)

Time-to-first-byte target: ≤ 250 ms. Real Time Factor at this configuration is ~0.3 — generation runs ~3× faster than playback.

6.2 ASR throughput

vLLM raises throughput sublinearly with batch size. Past ~8 concurrent requests, FFmpeg becomes the bottleneck — set VIBEVOICE_FFMPEG_MAX_CONCURRENCY to your CPU core count.

6.3 Memory profile

6.4 Tuning levers

Tip

Production tuning, in order of impact: (1) attention implementation (flash-attention-2 vs SDPA — measure both), (2) cfg_scale (lower = faster, less expressive), (3) inference_steps (5 default, raise for fidelity), (4) Liger Kernel in the ASR path, (5) OPTIMIZE_FOR_SPEED=1 for tokenizer speedup.

6.5 Profiling

import torch
with torch.profiler.profile(
    activities=[torch.profiler.ProfilerActivity.CPU, torch.profiler.ProfilerActivity.CUDA],
    record_shapes=True,
) as prof:
    out = model.generate(**inputs, max_new_tokens=256)
print(prof.key_averages().table(sort_by="cuda_time_total", row_limit=20))

Hot spots in our profiling: Qwen2 attention (~50%), diffusion head (TTS only, ~25%), tokenizer encode (~10%), audio normalization (~5%). Memory peaks during the prefill pass — tune --max_new_tokens if you see OOM mid-generation.

7. Security

7.1 Threat model

VibeVoice is a research model release. Three primary threat surfaces apply if you deploy it:

1. Model misuse — voice cloning + TTS produces realistic speech in the trained voices. Mitigation: deploy behind authenticated APIs, retain server-side logs of (text, voice) tuples.
2. Data exfiltration via context_info — the ASR system prompt accepts free-form text; in a multi-tenant deployment, this could be used to inject system-prompt content. Mitigation: sanitize / quote-escape user-supplied hotwords before passing through.
3. Resource exhaustion — the streaming server has no rate limit and runs unbounded scripts on the GPU. Mitigation: max-text-length guard, per-IP rate limit, max-concurrency cap, queue rejections.

7.2 Network exposure

The streaming server (demo/web/app.py):

• Exposes WebSocket /stream with no authentication.
• Exposes /config with no authentication (returns voice list, useful for fingerprinting).
• Has no CORS configuration; FastAPI defaults to no CORS allowance.

Warning

Never expose demo/web/app.py directly to the internet. Wrap with an authenticated reverse proxy (oauth2-proxy, Cloudflare Access, your own API gateway). Even the demo index.html page is intended for local-network use.

7.3 Secret management

The repo currently consumes no secrets at runtime:

• No API keys.
• No database connection strings.
• No Hugging Face token requirement (public models).

The only sensitive environment input is MODEL_PATH if it points to a local directory containing weights you do not want to leak. Bind-mount that directory read-only.

7.4 Input validation

Caution

Untrusted audio bytes are decoded by FFmpeg via subprocess.run(...). While this is a sandboxed binary, FFmpeg has had its share of CVEs. Keep FFmpeg patched and consider running the API container with seccomp / --read-only filesystem and a dropped capability set (no CAP_NET_RAW, no CAP_SYS_ADMIN).

7.5 Supply chain

pyproject.toml declares the project. The repo's external dependencies (PyTorch, transformers, diffusers, FFmpeg binary, optional flash-attn / vllm / liger-kernel) all come from public registries. Pin exact versions in your deployment lockfile and mirror to a private index.

The vLLM plugin's register_vibevoice() runs at import time for every vLLM process. Treat this as a code-execution surface — only add the plugin to vLLM environments you trust.

7.6 Reporting vulnerabilities

SECURITY.md is the canonical channel. Issue #341 (closed) noted earlier guidance was unclear; check the latest copy on main for the reporting email address and PGP key.

8. Developer Internals

8.1 Repository structure

VibeVoice/
├── pyproject.toml              # Package metadata + entry points (vLLM)
├── LICENSE                     # MIT
├── SECURITY.md                 # Vulnerability reporting
├── CONTRIBUTING.md             # PR guide
├── README.md                   # (intentionally minimal in this commit)
├── vibevoice/
│   ├── __init__.py            # Re-exports streaming TTS classes
│   ├── modular/                # Model code
│   │   ├── configuration_vibevoice.py             # VibeVoiceConfig + sub-configs
│   │   ├── configuration_vibevoice_streaming.py   # Streaming variant
│   │   ├── modeling_vibevoice.py                  # Base TTS model
│   │   ├── modeling_vibevoice_asr.py              # ASR model (separate path)
│   │   ├── modeling_vibevoice_streaming.py        # Streaming TTS submodel (forward disabled)
│   │   ├── modeling_vibevoice_streaming_inference.py  # Inference wrapper with .generate()
│   │   ├── modular_vibevoice_diffusion_head.py    # Diffusion head MLP
│   │   ├── modular_vibevoice_text_tokenizer.py    # Text tokenizer (Qwen2-derived) + ASR variant
│   │   ├── modular_vibevoice_tokenizer.py         # Acoustic + semantic VAEs
│   │   └── streamer.py                             # AudioStreamer + AsyncAudioStreamer
│   ├── processor/              # Audio + tokenization
│   │   ├── audio_utils.py                         # FFmpeg shell-out + AudioNormalizer
│   │   ├── vibevoice_processor.py                 # Long-form TTS processor (multi-speaker)
│   │   ├── vibevoice_streaming_processor.py       # Streaming TTS processor
│   │   ├── vibevoice_asr_processor.py             # ASR processor (system prompt + speech tokens)
│   │   └── vibevoice_tokenizer_processor.py       # File I/O for .wav/.mp3/.pt/.npy/.npz
│   ├── schedule/               # Diffusion scheduler
│   │   ├── dpm_solver.py                          # DPMSolverMultistepScheduler
│   │   └── timestep_sampler.py                    # Training-time sampling
│   ├── configs/
│   │   ├── qwen2.5_1.5b_64k.json
│   │   └── qwen2.5_7b_32k.json
│   └── scripts/                # (placeholder)
├── vllm_plugin/
│   ├── __init__.py            # register_vibevoice() — entry point
│   ├── inputs.py              # vibevoice_audio_input_mapper
│   └── model.py               # VibeVoiceForCausalLM (vLLM model class)
├── demo/
│   ├── realtime_model_inference_from_file.py
│   ├── vibevoice_asr_gradio_demo.py
│   ├── vibevoice_asr_inference_from_file.py
│   ├── vibevoice_realtime_demo.py
│   ├── vibevoice_realtime_colab.ipynb
│   ├── download_experimental_voices.sh
│   ├── web/
│   │   ├── app.py             # FastAPI + WebSocket
│   │   └── index.html
│   ├── voices/streaming_model/*.pt   # 23 voice presets (Git LFS)
│   ├── text_examples/*.txt
│   └── asr_demo/*.{mp3,mp4,wav}
├── finetuning-asr/
│   ├── lora_finetune.py
│   ├── inference_lora.py
│   ├── README.md
│   └── toy_dataset/{0.json, 0.mp3, 1.json, 1.mp3}
└── docs/                       # Reserved (currently no markdown — pull from upstream branches)

8.2 Module reference

vibevoice/modular/configuration_vibevoice.py

Defines four PretrainedConfig subclasses:

• VibeVoiceAcousticTokenizerConfig (model_type = "vibevoice_acoustic_tokenizer").
• VibeVoiceSemanticTokenizerConfig (model_type = "vibevoice_semantic_tokenizer").
• VibeVoiceDiffusionHeadConfig (model_type = "vibevoice_diffusion_head").
• VibeVoiceConfig (model_type = "vibevoice", composition of the four above plus Qwen2Config).

Implements _convert_dtype_to_string() to coerce torch.dtype to its string form during to_dict() — fixes "Object of type dtype is not JSON serializable" reported in issue #199.

VibeVoiceConfig.get_text_config() returns the inner Qwen2Config so vLLM can read hidden_size, num_attention_heads, etc. for memory profiling.

vibevoice/modular/modeling_vibevoice.py

Base TTS model. Raises NotImplementedError(f'Speech type {speech_type} not implemented') for unknown speech-type tokens in the prefix; NotImplementedError(f'Prediction type {prediction_type} not implemented') for unknown diffusion targets.

vibevoice/modular/modeling_vibevoice_asr.py

VibeVoiceASRForConditionalGeneration — wraps the Qwen2 decoder for ASR. Validates segment_length must be positive (raises ValueError otherwise).

vibevoice/modular/modeling_vibevoice_streaming.py

VibeVoiceStreamingModel.forward() is intentionally disabled — raises RuntimeError('VibeVoiceStreamingModel.forward is intentionally disabled. Use model.language_model(...) or model.tts_language_model(...) instead.').

vibevoice/modular/modeling_vibevoice_streaming_inference.py

Inference wrapper. set_output_embeddings() raises if lm_head is undefined; the unified forward() is disabled (use forward_lm, forward_tts_lm, or generate); loss computation is NotImplementedError.

vibevoice/modular/modular_vibevoice_tokenizer.py

The acoustic and semantic VAEs. OPTIMIZE_FOR_SPEED env var read at module load time (line 30) skips selected normalization/scale steps. Validates several knobs:

• mixer_layer ∈ a hardcoded set (else "Unsupported mixer layer").
• layernorm ∈ {RMSNorm, LayerNorm} (else "Unsupported norm type").
• dist_type ∈ {fix, gaussian} (else "Unsupported dist_type").
• GroupNorm + causal evaluation incompatible (rejected).

vibevoice/modular/streamer.py

AudioStreamer(batch_size, stop_signal, timeout) and AsyncAudioStreamer. Both validate sample_idx < self.batch_size; the streaming TTS path always uses batch_size=1.

vibevoice/processor/vibevoice_processor.py

Long-form TTS processor (multi-speaker dialog, not used by the realtime demo). Validates JSON speaker scripts: a list of speaker entries, with at least one valid speaker line, content non-empty.

vibevoice/processor/vibevoice_asr_processor.py

ASR processor. Builds the prompt:

<|im_start|>system
You are a speech recognition model.
<context_info, if provided>
<|im_end|>
<|im_start|>user
<|object_ref_start|><speech_pad_id × N><|object_ref_end|>
<|im_end|>
<|im_start|>assistant

Speech tokens are mapped to specific Qwen2 special-token IDs:

• speech_start_id → <|object_ref_start|>
• speech_pad_id → <|box_start|>
• speech_end_id → <|object_ref_end|>

The fast tokenizer (VibeVoiceASRTextTokenizerFast from modular_vibevoice_text_tokenizer.py) MUST respect these mappings or ASR quality silently degrades — this is called out in vllm_plugin/__init__.py:register_vibevoice().

Supported language_model_pretrained_name: Qwen, Llama, Gemma. Other backbones raise ValueError.

vibevoice/processor/vibevoice_streaming_processor.py

Streaming TTS processor. __call__ is not implemented — callers must use process_input_with_cached_prompt(text, cached_prompt, ...) which combines the script with a pre-encoded voice preset.

vibevoice/processor/vibevoice_tokenizer_processor.py

File I/O. Accepts .wav, .mp3, .flac, .m4a, .ogg, .pt, .npy, .npz. Validates audio shape (1D or 2D, otherwise raises). save_audio() requires soundfile; if missing it raises ImportError('soundfile is required to save audio files. Install it with: pip install soundfile').

vibevoice/schedule/dpm_solver.py

Adapted from https://github.com/LuChengTHU/dpm-solver. Implements multi-step DPM solvers (1st through 3rd order) with several algorithm_type and final_sigmas_type options. Heavy validation:

• algorithm_type not implemented → NotImplementedError.
• solver_type not implemented → NotImplementedError.
• final_sigmas_type ∈ {zero, sigma_min} (else ValueError).
• prediction_type ∈ {epsilon, sample, v_prediction}.
• timestep_spacing ∈ {linspace, leading, trailing}.
• beta_schedule not implemented → NotImplementedError.
• alpha_transform_type ∈ {cosine, exp, cauchy, laplace}.
• Cannot pass both num_inference_steps and timesteps (or custom_timesteps).
• karras_sigmas and lu_lambdas mutually exclusive with explicit timesteps.

vllm_plugin/inputs.py

vibevoice_audio_input_mapper(ctx, data) — accepts str, bytes, np.ndarray, or List[str]; rejects everything else with ValueError(f'Unsupported audio data type: {type(data)}'). Enforces VIBEVOICE_MAX_AUDIO_DURATION (raises with a specific guidance message instructing the user to set the env var).

vllm_plugin/model.py

VibeVoiceForCausalLM — the vLLM-side model class. Validates: tokenizer config presence (Missing acoustic/semantic tokenizer config in model config), tokenizer config types (acoustic_tokenizer_config has unexpected type, semantic_tokenizer_config has unexpected type), audio item length (Audio at index N is too short to be represented), modality strings (Unsupported modality: {modality}), and embed_tokens discoverability (Cannot find embed_tokens layer).

demo/web/app.py

Entry point for the FastAPI streaming TTS server. Composition:

• StreamingTTSService — owns the model, processor, voice presets, default voice key, voice cache.
• StreamingTTSService.load() — runs at FastAPI startup event; selects dtype + attn implementation per device; reconfigures the noise scheduler to sde-dpmsolver++ + squaredcos_cap_v2; sets DDPM inference steps to 5.
• StreamingTTSService.stream() — generator that yields normalized float32 numpy chunks.
• chunk_to_pcm16 — clips, scales by 32767, casts to int16, encodes to bytes.
• streaming_tts(text, **kwargs) — module-level convenience wrapper.
• websocket_stream — WebSocket handler, owns the asyncio.Lock busy-check.
• index — serves index.html.
• get_config — JSON list of voices and default voice key.

8.3 Code patterns

Config composition. VibeVoiceConfig is a Hugging Face composition config (is_composition = True) with four sub_configs. Each child can be passed as a dict (auto-instantiated) or a config instance. This is how the same code path serves both the 1.5B-64K and 7B-32K checkpoints — only the JSON values differ.

Attention selection. Every model loader in this repo follows the same pattern: try flash_attention_2 first; if it raises (commonly ImportError), fall back to sdpa and print a warning. The fall-through is hardcoded — there is no env-var override.

Device handling. Each script normalizes mpx → mps, then validates MPS availability and falls back to CPU. CUDA gets bfloat16 + device_map='cuda', MPS gets float32 + manual model.to('mps') (because MPS does not yet honor device_map cleanly).

Lazy loading. _voice_cache in StreamingTTSService caches torch.load(.pt) results per voice key. Other parts of the codebase use Hugging Face from_pretrained caching (~/.cache/huggingface/hub/). No explicit cache eviction — restart the server to free voice memory.

Streaming via threads. TTS generation runs in a daemon threading.Thread because the underlying model.generate() is a long blocking call. The main async event loop polls a queue.Queue for log frames between binary writes — await asyncio.to_thread(next, iterator, sentinel) is used to advance the audio iterator without blocking the loop.

Right-padding for training, left-padding for inference. The LoRA finetune collator (VibeVoiceASRDataCollator) right-pads input_ids/labels/speech for training. The processor's default behavior at inference time is left-padding (so causal attention treats the unpadded tokens as the most recent). This split is intentional and load-bearing — see lora_finetune.py:VibeVoiceASRDataCollator for the comment.

Frozen tokenizers. setup_model_for_training walks model.named_parameters() and freezes everything matching acoustic_tokenizer or semantic_tokenizer. Only the Qwen2 attention/MLP weights receive LoRA adapters.

8.4 Testing strategy

The vllm_plugin/tests/ directory contains test_api.py and test_api_auto_recover.py — these exercise the vLLM serving path end-to-end. There is no unit-test infrastructure committed for the rest of the repo; integration testing is performed via the demo scripts (which double as smoke tests). The toy dataset under finetuning-asr/toy_dataset/ (two MP3 + JSON pairs) supports a single-step finetune sanity check.

Note

Test coverage is the only confidence-floor "gap" recorded in this briefing's metadata. Adding pytest coverage for VibeVoiceASRProcessor, audio_utils.load_audio_use_ffmpeg, and DPMSolverMultistepScheduler.set_timesteps would close the loop without major architectural surgery.

8.5 Potential improvements

8.6 Contributing

CONTRIBUTING.md is checked in but minimal. Expect the standard Microsoft OSS flow: fork → branch → PR; sign-off via CLA bot if it triggers; tests + lint required for non-trivial changes. The repo has only a main branch and follows trunk-based development.

9. Integration Points

9.1 Hugging Face Hub

All model and processor classes implement the standard from_pretrained / save_pretrained contracts. The repo references three published Hugging Face IDs:

To deploy without network access, mirror these to your private HF instance and pass local paths to every script's --model_path / --base_model.

9.2 vLLM

pyproject.toml declares an entry point under vllm.general_plugins that maps to vllm_plugin.register_vibevoice. When vLLM starts, it discovers and runs this plugin, which registers the config, tokenizers, processor, and model classes. After registration, vLLM treats microsoft/VibeVoice-ASR as a first-class model — vllm serve microsoft/VibeVoice-ASR works end-to-end.

9.3 PEFT / LoRA

finetuning-asr/lora_finetune.py uses peft.LoraConfig + peft.get_peft_model. inference_lora.py uses peft.PeftModel.from_pretrained(base_model, lora_path). Adapter weights can be merged into the base model for faster inference (model = model.merge_and_unload()), at the cost of swappability.

9.4 FFmpeg

The audio loader requires ffmpeg and ffprobe on PATH. Both are invoked via subprocess.run(..., check=True). The semaphore in audio_utils.py allows operators to cap concurrent subprocesses (set VIBEVOICE_FFMPEG_MAX_CONCURRENCY to ~CPU-core count under heavy load).

9.5 Gradio

demo/vibevoice_asr_gradio_demo.py builds a gr.Blocks interface. Custom CSS styles the Stop button. demo.queue(default_concurrency_limit=3) allows up to three concurrent ASR requests through the Gradio runtime, but the ASR model itself is shared — back-pressure builds on the GPU, not the queue. demo.launch(share=args.share, show_error=True, theme=gr.themes.Soft(), css=custom_css) is the standard Gradio launch call; share=True publishes a Gradio public tunnel.

9.6 FastAPI / Starlette

demo/web/app.py uses FastAPI lifespan via @app.on_event("startup") (the older event-based API). For long-lived production, port to the lifespan async context manager pattern; behavior is equivalent.

9.7 Liger Kernel (optional)

The ASR Gradio demo wraps Qwen2 RoPE / RMSNorm / SwiGLU layers with Liger Kernel if importable:

from liger_kernel.transformers import apply_liger_kernel_to_qwen2
apply_liger_kernel_to_qwen2(rope=True, rms_norm=True, swiglu=True, cross_entropy=False)

cross_entropy=False is intentional — the Liger CE replacement is not yet validated against VibeVoice's loss path. Leave it disabled.

9.8 Pydub (optional)

Used by the ASR Gradio demo to re-encode segment audio as 32 kbps MP3 (smaller than WAV for inline data URIs). If pydub is missing, the demo falls back to WAV embedding. No effect on transcription quality — display-time only.

9.9 Diffusers

The DPM solver inherits from diffusers.schedulers.scheduling_utils.SchedulerMixin + ConfigMixin. This makes it usable in any diffusers pipeline that expects a KarrasDiffusionSchedulers-compatible scheduler. The streaming server reconfigures the scheduler to sde-dpmsolver++ + squaredcos_cap_v2 at startup — those values are not the diffusers defaults; if you build a pipeline outside the streaming server you must reconfigure manually.