0xShug0/audio.cpp

audio.cpp

audio.cpp is a high-performance C++ audio inference framework built on top of ggml, designed to make modern local audio models practical, portable, and fast.

Tired of juggling a dozen Conda environments, hundreds of Python packages, and dependency conflicts just to try a few audio models? audio.cpp gives those paths a shared native runtime instead. Runs on Windows, Linux, and macOS, with support for NVIDIA, AMD, Apple Silicon, and CPU-only machines.

Huggingface main repo: https://huggingface.co/audio-cpp/audio.cpp-gguf

ModelScope repo mirror: https://www.modelscope.cn/models/HereIsMark/audio.cpp-gguf

[!IMPORTANT]

> 2026-09-15 - Release v0.8.0: YuE2 and SheetSage2: YuE2 native song generation and SheetSage2 audio-to-ABC score transcription are now on main and included in release v0.8.0.

> Arena UI: The new Arena tab makes it easier to compare local models side by side for TTS, voice conversion, and ASR. Use one shared input, queue multiple models or GGUF variants, then review outputs with metrics!

> CUDA performance headline: multiple TTS paths already run 1.8x to up to 8x faster than their Python reference paths while cutting end-to-end latency by 45%-85%.

> GGUF performance: all released model families support GGUF loading, and tested Q8 packages can run up to 1.53x faster while reducing peak VRAM by up to about 37% on routes such as Higgs Audio, Fish Audio, and Voxtral. See the GGUF guide for support status and the Q8 performance report for 16-bit vs Q8 measurements.

> Production deployment example: Try Fun-ASR-Nano and SenseVoice with audio.cpp on the FunASR platform!

> VibeVoice 1.5B: generates a 93.9-minute podcast in 18.2 minutes with 10 diffusion steps and without quantization, running about 5.15x faster than real time.

> Supertonic 3: generates about 10 hours of audio in 3 minutes on RTX5090. Up to 200x+ real-time on CUDA, 6x+ real-time on CPU, and 47 ms TTFT in CUDA streaming mode.

Demo: 10 hours of audio generated in 3 minutes.

> Real-world ASR win: In TranscrIA benchmark on messy French meeting audio, audio.cpp’s Nemotron 3.5 ASR matched the same WER as other implementations while using about 1/4 of the wall time.

It is built for real end-to-end execution rather than one-off model demos: the same runtime powers TTS, voice cloning, voice conversion, ASR, diarization, VAD, source separation, alignment, codec-style models, and higher-level workflows through a common framework surface.

Highlights:

• Parity. Strong parity tooling against Python reference paths.
• Performance. Performance-focused execution, reusable sessions, and batch-style offline inference. Optimized for CUDA.
• Portability. A portable native stack centered on ggml, with CUDA, HIP/ROCm, Vulkan, Metal, and CPU backends behind shared CLI and server entry points instead of Python-only deployment paths.
• Pipelines. Experimental JSON pipeline support for higher-level multi-step workflows.
• Audio Utilities. Built-in denoise, enhancement, resampling, and STFT/ISTFT utilities for real production-style task paths. Model-based utilities need separately downloaded weights; see downloads and usage.

The goal of the framework is to provide highly optimized, reusable building blocks for audio-related models, so new model integrations can be brought up faster, shared components can be improved once and benefit many families, and real end-to-end inference paths can stay efficient, maintainable, and portable.

audio.cpp would not be moving this quickly without generous contributors bringing in real fixes, new capabilities, and careful polish. See CONTRIBUTING.md for how to contribute and for a shout-out to the people already helping shape the project.

[!TIP]

Contribution focus: the most helpful contributions right now are improvements to the UI, API server, and pipeline/workflow subsystems. These areas make the existing model surface easier to use, serve, compose, and validate. See CONTRIBUTING.md for more details.

> New model PRs: before starting a new model port, please check the supported model table because several families are already implemented or under testing and read the New Model PRs section in CONTRIBUTING.md. New ports should start under the community models surface, where review is lighter than core models but still needs reproducible validation.

News

[!IMPORTANT]

2026-09-15 - Release 0.8.0: Since v0.7.0, audio.cpp has added YuE2 song generation, SheetSage2 audio-to-score transcription, BreezeTTS 2, CosyVoice3, Kokoro 82M, Moonshine Streaming ASR, Niagara ASR, and VibeVoice ASR Streaming 7B. The community lineup also grows with Audio8 ASR/TTS, Echo-TTS, Chatterbox Turbo, MiraTTS, Sopro, Soprano, sanoTTS, VoxCPM1, VibeASR, and Sortformer v2.1, alongside Irodori v4.1 Anime support. This brings audio.cpp to 80+ model families and 120+ model variants! This release also brings improvements to performance, stability, and usability across the project. Thanks to the community for the contributions, testing, and feedback!

> Optional server frontend modules add MP3 support, HTTPS listeners, and more customization for server integrations.

> Thanks to @DrewThomasson for the Colab UI, @christopherthompson81 for the C ABI, and @XsquirrelC/VibeASR for the official VibeASR port, extending audio.cpp with support for BitNet models!

> 2026-08-26 - Release 0.7: This release adds MiniMax Music 3, MagpieTTS, PersonaPlex, MeanVC2, AudioSR, ControlFoley, FireRedTTS3, FireRedAudio, MiDashengLM-Gen, F5-TTS/Habibi, Granite Speech 5.0 TurboCTC, MMS Forced Aligner, and MOSS-VoiceGenerator, plus DotTTS Edit and ACE-Step 1.5 XL variants, bringing audio.cpp to 62 total model families and 85+ model variants! It also introduces the new Arena UI for side-by-side TTS, voice-conversion, and ASR comparison with shared inputs, queued runs, metrics, and result sorting.

> 2026-08-13 - Release 0.6: This release adds 5 new model families - DotTTS, NeuTTS, MuScriptor, MiniMax-H3, and SenseVoice - bringing audio.cpp to 49 total model families and 70+ model variants, alongside the new native WebUI from @mirek190, expanded GGUF packaging, and more shared framework runtime pieces.

> 2026-07-31 - Release 0.5: audio.cpp reaches 44 model families with 9 new additions, early HIP/ROCm support for AMD GPUs, Nix ROCm/HIP build support, Metal optimizations with tested VoxCPM2 runs up to 2.56x faster on Apple Silicon, and a major GGUF-first WebUI/package-spec usability pass.

2026-06-25 to 2026-07-23 (release 0.1 to 0.4): audio.cpp grew from the first released model wave into broad TTS, ASR, music generation, source separation, VAD, diarization, codec, and voice-conversion coverage, with VibeVoice 1.5B/7B, LoRA adapter loading, initial streaming support, and major CUDA Conv1DTransp speedups.

Supported Models

Task tags: TTS text to speech, Clone voice cloning, VC voice conversion, S2S speech-to-speech, ASR speech recognition, Align forced alignment, VAD voice activity detection, Diar speaker diarization, Codec audio codec, Sep source separation, MIDI audio-to-symbolic MIDI/events, Music music/song generation, SFX sound effects, Video video generation, Edit audio/music editing, Design voice design, Dialogue multi-speaker dialogue TTS, Ctrl TTS/clone voice control such as emotion, style, instruction, caption, or non-verbal tag control.

Runtime tags summarize the supported loading paths. GGUF package precision varies by model and release; check the audio.cpp GGUF repo or docs/gguf.md for the exact package list. Bundled means the tiny runtime asset ships under assets/framework/models and needs no separate model download. Stream means the family exposes a streaming server/session path.

Speech Generation And Conversation

| Family | Task | Lang | Variants | Runtime | |---|---|---|---|---| | breeze_tts | TTS, Clone, Design, Ctrl | zh, en | BreezeTTS 2 instruction-conditioned TTS and prompt-audio voice cloning | GGUF BF16/Q8, Stream | | chatterbox | TTS, Clone, VC| ar, da, de, el, en, es, fi, fr, hi, it, ko, ms, nl, no, pl, pt, sv, sw, tr | Chatterbox with 0.5B backbone | GGUF 16/Q8 | | confucius4_tts | Clone | zh, en, ja, ko, de, fr, es, id, it, th, pt, ru, ms, vi | Confucius4-TTS multilingual voice cloning | GGUF F32, Stream | | cosyvoice3 | TTS, Clone | zh, en, ja, ko, de, es, fr, it, ru, yue | Fun-CosyVoice3 zero-shot, cross-lingual, and instruction-conditioned TTS | GGUF F32/Q8 | | dots_tts | TTS, Clone, Edit, Ctrl | multilingual | DotTTS SOAR
DotTTS MeanFlow
DotTTS Edit | GGUF 16/Q8, Stream | | dramabox | TTS, Clone | en | DramaBox expressive TTS and voice cloning | GGUF Q8 | | fish_audio | TTS, Clone, Ctrl | auto, en, zh | Fish Audio S2 Pro | GGUF 16/Q8 | | firered_audio | ASR, TTS, Clone, Design, Ctrl | zh, en | FireRedAudio multimodal speech/audio model with ASR, understanding, cloning, design, and edit paths | GGUF original/Q8 | | fireredtts3 | TTS, Clone, Design, Ctrl | 24 langs + 21 zh dialects | FireRedTTS3 Base
FireRedTTS3 Instruct/Voicedesign | GGUF original/Q8 | | higgs_audio_tts | TTS, Clone, Ctrl | auto | Higgs Audio v3 TTS 4B | GGUF 16/Q8 | | index_tts2 | TTS, Clone, Ctrl | zh, en, ja, es, ar | IndexTTS-2
IndexTTS-2.5 | GGUF 16/Q8 | | kokoro_tts | TTS | en-us, en-gb, es, fr, hi, it, ja, pt-br, zh | Kokoro 82M, 54 preset voices | Safetensors, local GGUF BF16/Q8 | | irodori_tts | TTS, Clone, Design, Ctrl | ja | Irodori-TTS-v4.1-Small
Irodori-TTS-v4.1-Anime
Irodori-TTS-500M-v3
Irodori-TTS-600M-v3-VoiceDesign | GGUF 16/Q8 | | magpie_tts | TTS | ar-AE, ar-MSA, ar-SA, de, en, es, fr, hi, it, ko, pt-BR, vi, zh | NVIDIA MagpieTTS Multilingual 357M (v2607) with baked speaker prompts and NanoCodec decode | GGUF original/Q8 | | miotts | TTS, Clone | en, ja | MioTTS-1.7B | GGUF 16/Q8 | | moss_tts_local | TTS, Clone, Ctrl | auto, optional language hint | MOSS-TTS-Local-Transformer-v1.5 | GGUF 16/Q8 | | moss_tts_nano | TTS, Clone | auto | MOSS-TTS-Nano-100M | GGUF 16/Q8 | | neutts | TTS, Ctrl | en | NeuTTS 2E with built-in speaker prompts and emotion control | GGUF original precision, Stream | | omnivoice | TTS, Clone, Design, Ctrl | 646+ langs | OmniVoice, Qwen3-0.6B based | GGUF 16/Q8, Stream | | personaplex | Dialogue, S2S | en | PersonaPlex 7B v1 speech-to-speech conversational model with packaged voice/persona prompts | GGUF Q4/Q8, Stream | | pocket_tts | TTS, Clone | en, de, it, pt, es | PocketTTS-100M English/German/Italian/Portuguese/Spanish | GGUF 16/Q8, Stream | | qwen3_tts | TTS, Clone, Design, Ctrl | zh, en, fr, de, it, ja, ko, pt, ru, es | Qwen3-TTS-12Hz-0.6B-Base
Qwen3-TTS-12Hz-1.7B-Base
Qwen3-TTS-12Hz-1.7B-CustomVoice
Qwen3-TTS-12Hz-1.7B-VoiceDesign | GGUF 16/Q8 | | supertonic | TTS | en, ko, ja, ar, bg, cs, da, de, el, es, et, fi, fr, hi, hr, hu, id, it, lt, lv, nl, pl, pt, ro, ru, sk, sl, sv, tr, uk, vi, na | Supertonic 3 | GGUF F32, Stream | | vibevoice | TTS, Dialogue | en, zh | VibeVoice-1.5B
VibeVoice-7B | GGUF 16/Q8 | | voxcpm2 | TTS, Clone, Design, Ctrl | ar, da, de, el, en, es, fi, fr, he, hi, id, it, ja, km, ko, lo, ms, my, nl, no, pl, pt, ru, sv, sw, th, tl, tr, vi, zh | VoxCPM2-2B, 48 kHz | GGUF 16/Q8, Stream |

Speech Recognition And Analysis

| Family | Task | Lang | Variants | Runtime | |---|---|---|---|---| | citrinet_asr | ASR | en | Citrinet-256 | GGUF Q8 | | fun_asr_nano | ASR | auto, zh, en, ja | Fun-ASR-Nano-2512 | GGUF 16/Q8 | | higgs_audio_stt | ASR | en | Higgs Audio v3 STT | GGUF 16/Q8, Stream | | hviske_asr | ASR | da | Hviske v5.3 | GGUF Q8 | | marblenet_vad | VAD | lang agnostic | MarbleNet VAD | Bundled | | moonshine_asr | ASR | en | Moonshine Streaming Tiny/Small/Medium | GGUF Q8, Stream | | nemotron_asr | ASR | 100+ ASR prompt codes incl. auto | Nemotron 3.5 ASR Streaming 0.6B | GGUF 16/Q8, Stream | | niagara_asr | ASR | en | Niagara 19M Batch English
Niagara 38M Batch English | GGUF F32 | | qwen3_asr | ASR | zh, en, yue, ar, de, fr, es, pt, id, it, ko, ru, th, vi, ja, tr, hi, ms, nl, sv, da, fi, pl, cs, fil, fa, el, ro, hu, mk | Qwen3-ASR-0.6B
Qwen3-ASR-1.7B-hf | GGUF 16/Q8, Stream | | qwen3_forced_aligner | Align | zh, yue, en, de, es, fr, it, pt, ru, ko, ja | Qwen3-ForcedAligner-0.6B | GGUF 16/Q8 | | silero_vad | VAD | lang agnostic | Silero VAD | Bundled, Stream | | sortformer_diar | Diar | en | Sortformer-4spk-v1 | - | | vibevoice_asr | ASR | auto | VibeVoice ASR | GGUF 16/Q8 | | vibevoice_asr_streaming | ASR | en, zh, es, pt, de, ja, ko, fr, ru, it | VibeVoice ASR Streaming 7B with persistent decoder state and speaker turns | GGUF BF16/Q8/Q4, Stream | | voxtral_realtime | ASR | auto | Voxtral-Mini-4B-Realtime-2602 | GGUF 16/Q8/Q4, Stream |

Audio Conversion And Processing

| Family | Task | Lang | Variants | Runtime | |---|---|---|---|---| | audiosr | S2S | lang agnostic | AudioSR Basic audio super-resolution package | GGUF F32 | | bs_roformer | Sep | lang agnostic | BS-RoFormer vocal separation checkpoints | GGUF Q8 | | controlfoley | SFX | auto | ControlFoley 44 kHz multimodal Foley generation from text, video, and reference audio conditioning | GGUF F32/Q8 | | htdemucs | Sep | lang agnostic | HTDemucs
HTDemucs_ft | GGUF 16/Q8 | | meanvc2 | VC | lang agnostic | MeanVC2 120 ms/40 ms zero-shot voice conversion | GGUF F32/Q4, Stream | | mel_band_roformer | Sep | lang agnostic | Mel-Band RoFormer MLX vocal separation variants | GGUF 16/Q8 | | miocodec | Codec, VC | lang agnostic | MioCodec v2, 25 Hz, 44.1 kHz | GGUF 16/Q8 | | muscriptor | MIDI | music | MuScriptor Small audio-to-symbolic transcription | GGUF F32, Stream | | rvc | VC | lang agnostic | RVC F16 GGUF with packaged v1/v2 voices and optional retrieval blending | GGUF 16 | | seed_vc | VC | lang agnostic | SeedVC XLS-R + HiFT
SeedVC Whisper-small + BigVGAN | GGUF 16/Q8 | | sheetsage2 | MIDI | music | SheetSage2 audio-to-ABC score transcription | GGUF original precision |

Music, Media, And Editing

| Family | Task | Lang | Variants | Runtime | |---|---|---|---|---| | ace_step | Music, Edit | 50+ langs | ACE-Step 1.5 Turbo
ACE-Step 1.5 Base
ACE-Step 1.5 XL Turbo
ACE-Step 1.5 XL SFT | GGUF 16 | | heartmula | Music | zh, en, ja, ko, es | HeartMuLa-oss-3B with HeartCodec-oss | GGUF 16/Q8 | | midashenglm_gen | Music, SFX | auto | MiDashengLM-Gen structured-prompt generation for speech, music, sound effects, and ambience | GGUF F32/Q8 | | minimax_h3 | Video, Music, TTS/Dialogue | auto | MiniMax-H3 Q4_K with optional INT8 ConvRot DiT | GGUF Q4/INT8 | | minimax_music3 | Music | auto | MiniMax Music 3 text-to-music generation with lyrics conditioning | GGUF Q4/Q8 | | stable_audio | Music, SFX, Edit | en | Stable Audio 3 Small Music
Stable Audio 3 Small SFX
Stable Audio 3 Medium | GGUF 16/Q8 | | vevo2 | TTS, Music, VC, Edit | en, zh | Vevo2 with Qwen2.5-0.5B AR model | GGUF 16 | | yue2 | Music | en | YuE2-3B lyrics-conditioned song generation with optional ABC score planning and conditioning | GGUF BF16/Q8/Q4 |

Some model families in the supported table started as outside contributions before being promoted into the core release surface. Thanks to Mirek @mirek190 for BS-RoFormer, @justinjohn0306 for MOSS-TTS-Local, and @LauraGPT from the official FunASR team for Fun-ASR-Nano.

Community Models

Community model ports live under community_models to make the ownership boundary clear while keeping them available through the normal audio.cpp CLI and server paths. Some community-contributed models graduate into the core model tree when they become part of the main release surface. Huge thanks to the contributors who bring these models in, test them, and keep pushing the framework into new territory. See docs/community_models/models.md for community-model expectations and current entries.

| Family | Task | Lang | Runtime | Contributor | What They Added | |---|---|---|---|---|---| | audio8_asr | ASR | en, zh, yue, ja, ko, fr, de | GGUF Q8, Safetensors | @gqf2008 | Audio8-ASR-0.1B compact multilingual autoregressive ASR reusing the Qwen3-ASR encoder with an MLP-tower adapter and an 8-layer Qwen2-style decoder (CC-BY-NC, local conversion only) | | audio8_tts | TTS, Clone | auto, yue, zh, nl, en, fr, de, it, ja, ko, pl, es | GGUF Q8, Stream | @jasonchen31 | Audio8 TTS Preview 0.6B DualAR multilingual TTS and zero-shot voice cloning with a Qwen backbone and neural codec | | chatterbox_turbo | TTS (testing) | en | GGUF 16/Q8 | @pannagaps | Chatterbox Turbo distilled 350M GPT2 T3 backbone + 2-step meanflow S3Gen decoder; built-in voice | | echo_tts | Clone | en | GGUF 16/Q8 | @5uck1ess | Echo-TTS 44.1 kHz zero-shot voice cloning with EchoDiT latents and Fish S1-DAC decoding | | f5_tts | TTS, Clone | en, ar (Habibi) | GGUF | @tareko | F5-TTS flow-matching DiT synthesis and voice cloning, with Habibi Arabic aliases habibi/habibi_tts | | glm_tts | TTS, Clone | zh, en | GGUF | Mirek @mirek190 | GLM-TTS zero-shot synthesis and voice cloning support | | granite5asr | ASR | en | GGUF Q8 | @ampersandru | IBM Granite Speech 5.0 470M TurboCTC ultra-fast Conformer-CTC ASR with Shaw relative positional embeddings and ByteLevel BPE | | inflect_v2 | TTS | en | GGUF FP32 | Jan @JanWerder | Inflect Micro v2 and Nano v2 native offline synthesis | | kroko_asr | ASR | de, en, es, fr, it, he, nl, pt, sv, tr | Safetensors, GGUF Q8 | Mirek @mirek190 | Kroko Community ASR native offline/streaming Zipformer2/RNN-T transcription with word timestamps | | minimax_h3 | Video, Music, TTS/Dialogue | auto | GGUF Q4/INT8 | @0xShug0 | MiniMax-H3 text-to-audio/video generation with Q4_K and optional INT8 ConvRot DiT | | minimax_music3 | Music | auto | GGUF Q4/Q8 | @0xShug0, @JoeMattie | MiniMax Music 3 text-to-music generation with lyrics conditioning | | mira_tts | TTS, Clone | en | Local conversion | Mirek @mirek190 | MiraTTS experimental native Qwen2 + ECAPA/Perceiver zero-shot voice cloning with progressive segment streaming (CC-BY-NC-SA-4.0 weights) | | mms_forced_aligner | Align | nl (nld), en (eng); pre-romanized Latin | Safetensors, GGUF 16/Q8 | @LysanderdeJong | MMS-300M-1130 Forced Aligner word-timestamp alignment from a wav2vec2 CTC checkpoint (safetensors or local GGUF) | | moss_voicegen | Voice Design | en, zh | GGUF | Joost @jrohde | MOSS-VoiceGenerator speech in a voice designed from a written instruction | | outetts | TTS, Clone | en, ar, zh, nl, fr, de, it, ja, ko, lt, ru, es, pt, be, bn, ka, hu, lv, fa, pl, sw, ta, uk | GGUF | Mirek @mirek190 | Llama-OuteTTS-1.0-1B TTS and voice cloning support | | parakeet_tdt | ASR | auto, bg, cs, da, de, el, en, es, et, fi, fr, hr, hu, it, lt, lv, mt, nl, pl, pt, ro, ru, sk, sl, sv, uk | GGUF F32/16/Q8, Stream | @dleiferives | Parakeet-TDT 0.6B v3 offline, long-form, and buffered-streaming ASR support, plus the Orukeet r3 weight variant | | sanotts | TTS | en, vi, id, cs, de, es, fr, it, pt, ro, ru, tr, ne, hi | GGUF FP32 | Ashish @voidash | sanoTTS voice family eighteen voices from 294k to 2.27M parameters, native offline synthesis | | sense_asr | ASR | auto, zh, en, yue, ja, ko, pt, ru, es, it, fr, de, nl, pl, tr, ar, hi, vi, th, id, ms, fa, nospeech | GGUF Q8, Stream | Jason Chen @jasonchen31, @LauraGPT / FunASR | SenseVoice-Small offline/streaming SAN-M + CTC transcription with event/emotion/language tags and ITN | | sopro_tts | TTS, Clone | en, pt, fr, de | Safetensors, GGUF, Stream | @Brioch | Sopro V2 Turbo 120M zero-shot voice cloning: style-prefix semantic LM over FSQ tokens, rectified-flow acoustic DiT, Vocos ISTFT vocoder at 24 kHz | | soprano_tts | TTS | en | GGUF Q8, Stream | @drzsdrtfg | Soprano-1.1-80M ultra-lightweight TTS with Qwen3 LM + Vocos decoder | | sortformer_diar_v2 | Diar | multilingual | GGUF F32/mixed F16, Stream | @LysanderdeJong | NVIDIA Sortformer v2.1 four-speaker streaming diarization; local conversion only pending redistribution approval | | vietneu_tts | TTS, Clone | vi, en | GGUF | Phuoc @phuocnguyen90 | VieNeu-TTS-v3-Turbo TTS and voice cloning support | | vibeasr | ASR | en | GGUF I8_S + I2_S | @XsquirrelC/VibeASR | VibeASR fully quantized port of VibeASR.cpp: VibeVoice acoustic/semantic tokenizers on INT8 weights and INT8 activations, feeding a ternary BitNet Qwen2 decoder. Offline, CPU only | | voxcpm1 | TTS, Clone | zh, en, ja, ko | GGUF Q8, Stream | @jasonchen31 | [VoxCPM1](docs/community_models/voxcpm1.