SpeechT5 TTS

microsoft/speecht5_tts is a text-to-speech model built on the SpeechT5 unified-modal encoder-decoder framework, fine-tuned for speech synthesis on the LibriTTS dataset. The model was developed by Microsoft and published at ACL 2022.

Architecture

SpeechT5 uses a shared encoder-decoder network with six modal-specific pre-nets and post-nets for speech and text. The text encoder pre-net maps input tokens to hidden representations (similar to BERT), while the speech decoder employs a Tacotron2-inspired pre-net (linear layers for log mel spectrograms) and a post-net (residual refinement of spectrograms). A cross-modal vector quantization mechanism aligns speech and text representations in a unified semantic space.

Pre-training Data

The model was pre-trained on approximately 960 hours of LibriSpeech audio data combined with the LibriSpeech LM text dataset, both English. This self-supervised pre-training spans automatic speech recognition, speech synthesis, translation, voice conversion, enhancement, and speaker identification tasks.

Inference Details

Output is a mono 16 kHz waveform. The model requires a speaker embedding vector (e.g., from CMU Arctic xvectors) to control voice characteristics. No latency or throughput benchmarks are reported in the original card.

Research Basis

The underlying paper is “SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processing” (arXiv:2110.07205). The model code and weights are released under the MIT license.