sgl-project/sglang-omni

--------------------------------------------------------------------------------

Blog | Documentation | Quick Start | Cookbook | SGLang | Join Slack

⭐ Star SGLang-Omni to help more builders discover open infrastructure for multimodal and speech serving!

News

• [2026/10] 🎙️ Introducing Voxt, an open-source voice input tool for Mac, now integrating SGLang-Omni for faster local speech recognition on Apple Silicon. Combine ASR with LLM-based text refinement and personal system prompts for accurate technical terminology and personalized voice input, with fully local or remote inference options. [Download][[Demo]](https://x.com/GenAI_is_real/status/2109419016205017181)
• [2026/10] 🚀 SGLang-Omni v0.1.8 is on PyPI. Install with uv pip install --prerelease=allow "sglang-omni==0.1.8". \[Installation\]
• [2026/09] 🐧 Day-0 support for AuK and AuK-Flash: text + voice instructions → 24 kHz speech on /v1/audio/speech, audio + editing instructions → edited speech on /generate. \[Cookbook\]
• [2026/08] 🎵 Day-0 support for MiniMax Music 3: lyrics + caption → 32 kHz stereo song on /v1/audio/speech. \[Cookbook\]
• [2026/08] 🚀 TTS architecture refactor: shared pipeline state, engine construction, reference encoding, capability metadata, and vocoder scheduling. \[Roadmap\] \[Blog\]
• [2026/06] 🔥 MOSS-TTS Local Transformer v1.5 on SGLang-Omni with native-streaming 48 kHz speech. \[Blog\] \[Cookbook\]
• [2026/06] 🔥 Higgs Audio v3 TTS for real-time, controllable speech. \[Blog\] \[Cookbook\]

About

SGLang-Omni is a multi-stage serving runtime for omni, speech, and TTS models. Its design target is multi-stage decoding: generation split across heterogeneous stages with different compute patterns, dependency structures, and resource needs. SGLang-Omni owns the pipeline topology, stage lifecycle, inter-stage transport, model-family integration layer, and OpenAI-compatible serving surface, while composing with SGLang for high-performance autoregressive scheduling and model execution where applicable.

• Multi-stage runtime: SGLang-Omni models generation as coordinated stages: preprocessing, encoders, autoregressive engines, talkers, decoders, vocoders, and aggregators.
• Stage-specialized scheduling: Each stage runs behind a scheduler matched to its workload, from SGLang-backed autoregressive scheduling to lightweight preprocessing and streaming vocoder loops.
• Transport-aware execution: A control plane coordinates requests while the relay data plane moves tensor payloads across shared-memory, NCCL, NIXL, and Mooncake backends.
• API surface: OpenAI-compatible endpoints expose multimodal chat, speech generation, batch speech, streaming speech, uploaded voices, and transcription.

What SGLang-Omni Serves

• Omni chat and speech: Qwen3-Omni, Ming-Omni — multimodal in, text/audio out.
• Music generation: MiniMax Music 3 — lyrics + caption → 32 kHz stereo song.
• Speech generation: Higgs Audio v3, MOSS-TTS, MOSS-TTS Local, Fish Speech S2-Pro, Qwen3-TTS, Voxtral TTS, Ming-Omni-TTS, dots.tts, ZONOS2 — /v1/audio/speech, batch, streaming, uploaded voices.
• Audio transcription and diarization: Qwen3-ASR, Fun-ASR, ARK-ASR, Nemotron 3.5 ASR, MOSS-Transcribe-Diarize via /v1/audio/transcriptions. MOSS-TD supports speaker labels and timestamps (response_format=verbose_json).
• SGLang-Omni Router: Multi-worker OpenAI-compatible front door — health, readiness, lifecycle, capability discovery. Router guide.

Hardware Support

| Backend | Status | Notes | |---------|--------|-------| | NVIDIA CUDA | Supported | Default backend with full model coverage. | | Apple Silicon | Experimental | Qwen3-ASR runs through native MLX or Torch MPS on macOS arm64. Install with install.sh and follow the Qwen3-ASR guide. | | Intel GPU (XPU) | Experimental | Intel Arc GPUs via PyTorch XPU. Qwen3-ASR, Qwen3-TTS, ZONOS2, and Qwen3-Omni serve end-to-end (ZONOS2 with decode graphs; Omni thinker via multi-XPU tensor parallelism). Install per Intel XPU guide; the backend is auto-detected. |

Additional model guides, including experimental and research-oriented paths, are available in the Cookbook.

Quick Start

• macOS Apple Silicon: from a checkout, run ./install.sh for a one-command Homebrew + uv setup. See installation.
• Installation
• TTS usage
• Qwen3-Omni usage
• Qwen3-ASR cookbook
• Nemotron 3.5 ASR cookbook
• MOSS-Transcribe-Diarize cookbook
• Omni router
• Developer reference

Community & Support

SGLang-Omni welcomes contributors working on inference systems, kernels, scheduling, inter-stage communication, model runners and cache efficiency, model integration, benchmarking, production deployment. Join the SGLang Slack or read the developer reference.

Organizations interested in supporting SGLang-Omni, TTS, or omni model serving can contact Chenyang Zhao at zhaochenyang@lmsys.org.

Acknowledgments

SGLang-Omni builds on the SGLang ecosystem and on open model work from the TTS, speech, and omni-model communities. We thank the model teams, systems contributors, and partner organizations helping make open multimodal serving faster, more reliable, and easier to extend.