espnet/espnet

• ESPnet 202604 —

• ESPnet 202511 —

• ESPnet 202509 —

• ESPnet 202506 —

• ESPnet 202503 —

Full history: Releases.

pip install "espnet[all]"                       # optional dependencies
pip install git+https://github.com/espnet/espnet  # latest master

• Full setup (recipes, DNN training, Kaldi-style tooling): see the

• Docker: see docker/ and the Docker docs.
• Task-specific tools live in tools/installers.
• ESPnet1 is no longer supported — use ESPnet2 (egs2/) or ESPnet3 (egs3/). See the ESPnet1 notice.

|system/pytorch ver.|2.9.1|2.10.0|2.11.0| | :---- | :---: | :---: | :---: | |ubuntu/python3.12/pip|||| |ubuntu/python3.13/pip|||| |debian12/python3.12/conda|||| |windows/python3.12/pip|||| |macos/python3.12/pip|||| |macos/python3.12/conda||||

Kaldi-style complete recipe

• Support numbers of ASR recipes (WSJ, Switchboard, CHiME-4/5, Librispeech, TED, CSJ, AMI, HKUST, Voxforge, REVERB, Gigaspeech, etc.)
• Support numbers of TTS recipes in a similar manner to the ASR recipe (LJSpeech, LibriTTS, M-AILABS, etc.)
• Support numbers of ST recipes (Fisher-CallHome Spanish, Libri-trans, IWSLT'18, How2, Must-C, Mboshi-French, etc.)
• Support numbers of MT recipes (IWSLT'14, IWSLT'16, the above ST recipes etc.)
• Support numbers of SLU recipes (CATSLU-MAPS, FSC, Grabo, IEMOCAP, JDCINAL, SNIPS, SLURP, SWBD-DA, etc.)
• Support numbers of SE/SS recipes (DNS-IS2020, LibriMix, SMS-WSJ, VCTK-noisyreverb, WHAM!, WHAMR!, WSJ-2mix, etc.)
• Support voice conversion recipe (VCC2020 baseline)
• Support speaker diarization recipe (mini_librispeech, librimix)
• Support singing voice synthesis recipe (ofuton_p_utagoe_db, opencpop, m4singer, etc.)

ASR: Automatic Speech Recognition

• State-of-the-art performance in several ASR benchmarks (comparable/superior to hybrid DNN/HMM and CTC)
• Hybrid CTC/attention based end-to-end ASR
• Fast/accurate training with CTC/attention multitask training
• CTC/attention joint decoding to boost monotonic alignment decoding
• Encoder: VGG-like CNN + BiRNN (LSTM/GRU), sub-sampling BiRNN (LSTM/GRU), Transformer, Conformer, Branchformer, or E-Branchformer
• Decoder: RNN (LSTM/GRU), Transformer, or S4
• Attention: Flash Attention, Dot product, location-aware attention, variants of multi-head
• Incorporate RNNLM/LSTMLM/TransformerLM/N-gram trained only with text data
• Batch GPU decoding
• Data augmentation
• Transducer based end-to-end ASR
• Architecture:
• Custom encoder supporting RNNs, Conformer, Branchformer (w/ variants), 1D Conv / TDNN.
• Decoder w/ parameters shared across blocks supporting RNN, stateless w/ 1D Conv, MEGA, and RWKV.
• Pre-encoder: VGG2L or Conv2D available.
• Search algorithms:
• Greedy search constrained to one emission by timestep.
• Default beam search algorithm [[Graves, 2012]](https://arxiv.org/abs/1211.3711) without prefix search.
• Alignment-Length Synchronous decoding [[Saon et al., 2020]](https://ieeexplore.ieee.org/abstract/document/9053040).
• Time Synchronous Decoding [[Saon et al., 2020]](https://ieeexplore.ieee.org/abstract/document/9053040).
• N-step Constrained beam search modified from [[Kim et al., 2020]](https://arxiv.org/abs/2002.03577).
• modified Adaptive Expansion Search based on [[Kim et al., 2021]](https://ieeexplore.ieee.org/abstract/document/9250505) and NSC.
• Features:
• Unified interface for offline and streaming speech recognition.
• Multi-task learning with various auxiliary losses:
• Encoder: CTC, auxiliary Transducer and symmetric KL divergence.
• Decoder: cross-entropy w/ label smoothing.
• Transfer learning with an acoustic model and/or language model.
• Training with FastEmit regularization method [[Yu et al., 2021]](https://arxiv.org/abs/2010.11148).

• CTC segmentation
• Non-autoregressive model based on Mask-CTC
• ASR examples for supporting endangered language documentation (Please refer to egs/puebla_nahuatl and egs/yoloxochitl_mixtec for details)
• Wav2Vec2.0 pre-trained model as Encoder, imported from FairSeq.
• Self-supervised learning representations as features, using upstream models in S3PRL in frontend.
• Set frontend to s3prl
• Select any upstream model by setting the frontend_conf to the corresponding name.
• Transfer Learning :
• easy usage and transfers from models previously trained by your group or models from ESPnet Hugging Face repository.
• Documentation and toy example runnable on colab.
• Streaming Transformer/Conformer ASR with blockwise synchronous beam search.
• Restricted Self-Attention based on Longformer as an encoder for long sequences
• OpenAI Whisper model, robust ASR based on large-scale, weakly-supervised multitask learning

• Real-time ASR demo with ESPnet2
• Gradio Web Demo on Hugging Face Spaces. Check out the Web Demo
• Streaming Transformer ASR Local Demo with ESPnet2.

TTS: Text-to-speech

• Architecture
• Tacotron2
• Transformer-TTS
• FastSpeech
• FastSpeech2
• Conformer FastSpeech & FastSpeech2
• VITS
• JETS
• Multi-speaker & multi-language extension
• Pre-trained speaker embedding (e.g., X-vector)
• Speaker ID embedding
• Language ID embedding
• Global style token (GST) embedding
• Mix of the above embeddings
• End-to-end training
• End-to-end text-to-wav model (e.g., VITS, JETS, etc.)
• Joint training of text2mel and vocoder
• Various language support
• En / Jp / Zn / De / Ru / And more...
• Integration with neural vocoders
• Parallel WaveGAN
• MelGAN
• Multi-band MelGAN
• HiFiGAN
• StyleMelGAN
• Mix of the above models

• Real-time TTS demo with ESPnet2
• Integrated to Hugging Face Spaces with Gradio. See demo:

• kan-bayashi/ParallelWaveGAN
• r9y9/wavenet_vocoder

SE: Speech enhancement (and separation)

• Single-speaker speech enhancement
• Multi-speaker speech separation
• Unified encoder-separator-decoder structure for time-domain and frequency-domain models
• Encoder/Decoder: STFT/iSTFT, Convolution/Transposed-Convolution
• Separators: BLSTM, Transformer, Conformer, TasNet, DPRNN, SkiM, SVoice, DC-CRN, DCCRN, Deep Clustering, Deep Attractor Network, FaSNet, iFaSNet, Neural Beamformers, etc.
• Flexible ASR integration: working as an individual task or as the ASR frontend
• Easy to import pre-trained models from Asteroid
• Both the pre-trained models from Asteroid and the specific configuration are supported.

• Interactive SE demo with ESPnet2
• Streaming SE demo with ESPnet2

ST: Speech Translation & MT: Machine Translation

• State-of-the-art performance in several ST benchmarks (comparable/superior to cascaded ASR and MT)
• Transformer-based end-to-end ST (new!)
• Transformer-based end-to-end MT (new!)

VC: Voice conversion

• Transformer and Tacotron2-based parallel VC using Mel spectrogram
• End-to-end VC based on cascaded ASR+TTS (Baseline system for Voice Conversion Challenge 2020!)

SLU: Spoken Language Understanding

• Architecture
• Transformer-based Encoder
• Conformer-based Encoder
• Branchformer based Encoder
• E-Branchformer based Encoder
• RNN based Decoder
• Transformer-based Decoder
• Support Multitasking with ASR
• Predict both intent and ASR transcript
• Support Multitasking with NLU
• Deliberation encoder based 2 pass model
• Support using pre-trained ASR models
• Hubert
• Wav2vec2
• VQ-APC
• TERA and more ...
• Support using pre-trained NLP models
• BERT
• MPNet And more...
• Various language support
• En / Jp / Zn / Nl / And more...
• Supports using context from previous utterances
• Supports using other tasks like SE in a pipeline manner
• Supports Two Pass SLU that combines audio and ASR transcript

• Performing noisy spoken language understanding using a speech enhancement model followed by a spoken language understanding model.
• Performing two-pass spoken language understanding where the second pass model attends to both acoustic and semantic information.
• Integrated to Hugging Face Spaces with Gradio. See SLU demo on multiple languages:

SUM: Speech Summarization

• End to End Speech Summarization Recipe for Instructional Videos using Restricted Self-Attention [[Sharma et al., 2022]](https://arxiv.org/abs/2110.06263)

SVS: Singing Voice Synthesis

• Framework merge from Muskits
• Architecture
• RNN-based non-autoregressive model
• Xiaoice
• Tacotron-singing
• DiffSinger (in progress)
• VISinger
• VISinger 2 (its variations with different vocoders-architecture)
• Support multi-speaker & multilingual singing synthesis
• Speaker ID embedding
• Language ID embedding
• Various language support
• Jp / En / Kr / Zh
• Tight i