MagpieTTS Multilingual 357M

| |

🤗 Hugging Face MagpieTTS Multilingual demo: magpie_tts_multilingual_demo

💻 NeMo Speech Framework: github.com/NVIDIA-NeMo/Speech

July 21, 2026: MagpieTTS v2607 was released with support for 3 new languages (Arabic, Korean, Portuguese).

For the older checkpoints, refer to below tags:

Released on March 2026, v2602.

Released on January 2026, v2512.

Model Details

Developed by: NVIDIA
Model type: Multilingual text-to-speech (encoder–decoder transformer, 364M parameters)
Languages: ar, de, en, es, fr, hi, it, ja, ko, pt, vi, zh
License: NVIDIA Open Model License
Version: v2607 (latest); previous: v2602, v2512
Library: NeMo Speech

Model Description

MagpieTTS is an end-to-end multilingual neural text-to-speech model that synthesizes speech using 5 English speaker voices — Aria, Jason, Leo, Sofia, and John Van Stan — across 12 languages: Arabic (ar), Chinese (zh), English (en), French (fr), German (de), Hindi (hi), Italian (it), Japanese (ja), Korean (ko), Portuguese (pt), Spanish (es), and Vietnamese (vi). The model adopts a transformer encoder–decoder architecture that autoregressively predicts discrete audio codec tokens, using multi-codebook prediction (typically 8 codebooks) with frame stacking (factor = 2) and a local transformer for fine-grained refinement of high-fidelity audio. To improve robustness and controllability, training incorporates attention priors for stable text-to-audio alignment, classifier-free guidance (CFG) for stronger conditioning, and Group Relative Policy Optimization (GRPO) for preference-aligned generation. At inference time, MagpieTTS supports batched synthesis of complete utterances as well as long-form generation of extended text via a sliding-window mechanism; the predicted codec tokens are then decoded into speech waveforms by a frozen pretrained audio codec model (NanoCodec). This release also removed zero-shot voice-cloning capability for security reasons, and added IPA grapheme-to-phoneme (G2P) support for custom dictionaries and code-switching, and updated G2P support for English-to-Katakana code-switching.

This model is ready for commercial use.

Key Features

Multilingual Support — Synthesizes natural speech across all 12 supported languages with consistent speaker identity.
Expressive Voices — Multiple voice options with emotional tones and gender variations including 4 proprietary voices and 1 public voice.
Text Normalization — Built-in text normalization for handling numbers, abbreviations, and special characters for all 12 languages.
Efficient High-Fidelity Decoding — A local transformer with frame stacking (factor = 2) performs multi-codebook refinement on stacked frames, improving audio quality while reducing sequence length for faster generation.

Model Sources

Demo: magpie_tts_multilingual_demo
Repository: NVIDIA NeMo Speech
Enterprise API: MagpieTTS NIM
Voice-agent examples: NVIDIA voice-agent-examples
Nemotron: Overview · Developer

Uses

Direct Use

MagpieTTS is for developers, researchers, and product teams building multilingual speech applications that need consistent speaker voices across 12 languages. Typical applications include cascade voice agents, audiobook and content narration, accessibility tools, dubbing and localization pipelines, and interactive media. IPA grapheme-to-phoneme (G2P) support for custom dictionaries and code-switching (including English-to-Katakana) also enables mixed-language content and domain-specific pronunciation.

MagpieTTS acts as a dedicated speech-generation layer that plugs into existing AI pipelines without changing upstream language models or downstream audio handling. In cascade voice-agent setups, it converts Large Language Model (LLM) text into natural, real-time speech for user playback. It can also replace or extend existing NVIDIA TTS integrations when multilingual coverage from a single unified model is required.

Deployment Geography: Global

Out-of-Scope Use

This model is not intended for zero-shot voice cloning, languages outside the 12 supported languages, or use cases that bypass the NVIDIA Open Model License terms. See Technical Limitations & Mitigations for additional constraints.

Model Architecture

Architecture Type: Transformer Encoder, Transformer Decoder, Local Transformer, and Feedforward Layers

Figure 1: MagpieTTS Model Architecture

Network Architecture:

Causal Transformer Encoder with 6 layers, learnable positional encoder of length 2048, and 1 Layer Normalization output layer.
Causal Transformer Decoder with 12 layers, learnable positional encoder of length 2048, and 1 Layer Normalization output layer.
Local Transformer for multi-codebook refinement, operating on stacked frames with a frame stacking factor of 2 to improve audio quality and reduce sequence length.

Number of model parameters: 3.64 x 10^8 (364M parameters)

Inputs & Outputs

Input Type(s): Text
Input Format: String
Input Parameters: One-Dimensional (1D)
Other Properties Related to Input: Text input is UTF-8 encoded; text normalization is required.

Output Type(s): Audio
Output Format: WAV
Output Parameters: One-Dimensional (1D)
Other Properties Related to Output: Mono, PCM-encoded 16 bit audio; sampling rate of 22.05 kHz; Audio output with dimensions (B x T), where B is batch size and T is time dimension.

How to Get Started with the Model

Choose a path below: Hosted API Quickstart (no GPU required) or Local Inference with NeMo Speech (Method 1 for a single utterance, Method 2 for batch evaluation).

Hosted API Quickstart

Synthesize speech using the hosted NVIDIA NIM API on Magpie TTS Multilingual — no local GPU, Docker, or checkpoint download required.

1. Get a free API key: Open Magpie TTS Multilingual and choose Get API Key.

2. Install the Riva client:

pip install nvidia-riva-client

3. Synthesize speech to a WAV file:

import wave

import riva.client
from riva.client.proto.riva_audio_pb2 import AudioEncoding

auth = riva.client.Auth(
    uri="grpc.nvcf.nvidia.com:443",
    use_ssl=True,
    metadata_args=[
        ["function-id", "877104f7-e885-42b9-8de8-f6e4c6303969"],
        ["authorization", "Bearer nvapi-YOUR_API_KEY"],
    ],
)

service = riva.client.SpeechSynthesisService(auth)

sample_rate_hz = 22050
resp = service.synthesize(
    "Hello from the Magpie multilingual hosted API.",
    "Magpie-Multilingual.EN-US.Sofia",
    "en-US",
    sample_rate_hz=sample_rate_hz,
    encoding=AudioEncoding.LINEAR_PCM,
)

with wave.open("out.wav", "wb") as wf:
    wf.setnchannels(1)
    wf.setsampwidth(2)
    wf.setframerate(sample_rate_hz)
    wf.writeframesraw(resp.audio)

Or use the CLI (list voices, then synthesize):

git clone https://github.com/nvidia-riva/python-clients.git
export NVIDIA_API_KEY="nvapi-YOUR_API_KEY"

python python-clients/scripts/tts/talk.py \
    --server grpc.nvcf.nvidia.com:443 --use-ssl \
    --metadata function-id "877104f7-e885-42b9-8de8-f6e4c6303969" \
    --metadata authorization "Bearer $NVIDIA_API_KEY" \
    --list-voices

python python-clients/scripts/tts/talk.py \
    --server grpc.nvcf.nvidia.com:443 --use-ssl \
    --metadata function-id "877104f7-e885-42b9-8de8-f6e4c6303969" \
    --metadata authorization "Bearer $NVIDIA_API_KEY" \
    --text "Hello from Magpie." \
    --voice "Magpie-Multilingual.EN-US.Sofia" \
    --language-code en-US \
    --sample-rate-hz 22050 \
    -o out.wav

Note: Voice IDs and supported sample rates follow the deployed NIM. Use --list-voices against this endpoint, or see the API Reference on the Magpie TTS Multilingual page.

Local Inference with NeMo Speech

To train, fine-tune or perform TTS with this model, you will need to install NVIDIA NeMo Speech. We recommend you install it after you've installed latest PyTorch version and Python version ≥ 3.10.12.

pip install nemo_toolkit[tts]@main
pip install kaldialign

The model is available for use in the NVIDIA NeMo Speech Framework, and can be used as a pre-trained checkpoint for inference or for fine-tuning on another dataset.

Two inference paths are available: Method 1 (single utterance) and Method 2 (batch inference and evaluation).

Method 1 — Single utterance inference

Synthesize one (text, language) pair at a time. Text normalization can be applied for all 12 languages. Load the open-source MagpieTTS checkpoint from Hugging Face and call the model's do_tts(transcript: str, language: str, apply_TN: bool, use_cfg: bool, speaker_index: int) method. This returns the generated audio and the length of the audio.

from nemo.collections.tts.models import MagpieTTSModel

speaker_map = {
    "Aria": 0,
    "Jason": 1,
    "John": 2,
    "Leo": 3,
    "Sofia": 4,
}
transcript = "Hello world from NeMo Text to Speech."
language = "en"
speaker = "Sofia"
speaker_idx = speaker_map[speaker]

# Load the latest checkpoint (from the `main` branch).
model = MagpieTTSModel.from_pretrained("nvidia/magpie_tts_multilingual_357m")
audio, audio_len = model.do_tts(transcript, language=language, apply_TN=False, speaker_index=speaker_idx)

# To apply custom phoneme customization in supported languages like English
# Surround the IPA string in a '|' character and add a space token between each IPA character.
ipa_transcript = "Hello world from | ˈ n ɛ m o ʊ | Text to Speech."
audio, audio_len = model.do_tts(ipa_transcript, language=language, apply_TN=False, speaker_index=speaker_idx)

Choosing a model version

from_pretrained(...) always loads the latest checkpoint from the main branch. To load a specific release instead, download the .nemo file for that version tag and restore it with restore_from(...). Available tags: v2607 (latest, on main), v2602, and v2512.

from huggingface_hub import hf_hub_download
from nemo.collections.tts.models import MagpieTTSModel

# Pin a specific release by its tag (branch, tag, or commit hash).
model_path = hf_hub_download(
    repo_id="nvidia/magpie_tts_multilingual_357m",
    filename="magpie_tts_multilingual_357m.nemo",
    revision="v2602",
)
model = MagpieTTSModel.restore_from(model_path)

Method 2 — Batch inference and evaluation

Run batch inference and optional evaluation with examples/tts/magpietts_inference.py. The script supports:

Batch inference from .nemo files or .ckpt checkpoints
Optional evaluation with metrics — Character Error Rate (CER), Speaker Similarity (SSIM), and UTMOSv2
Multiple datasets in a single run

Dataset configuration (examples/tts/evalset_config.json)

The script requires a JSON configuration file that defines the metadata for the datasets to process.

Format

{
    "dataset_name_1": {
        "manifest_path": "/absolute/path/to/manifest.json",
        "audio_dir": "/",
    },
    "dataset_name_2": {
        "manifest_path": "/path/to/another_manifest.json",
        "audio_dir": "/base/audio/path",
    }
}

Fields

Field	Required	Description
`manifest_path`	Yes	Absolute path to the NeMo manifest JSON file
`audio_dir`	Yes	Base directory for audio files. Use `"/"` if manifest contains absolute paths
`whisper_language`	No	Language code for ASR evaluation (default: `"en"`)

Example

{
    "libritts_test_clean": {
        "manifest_path": "/data/libritts/test_clean_manifest.json",
        "audio_dir": "/",
        "whisper_language": "en"
    },
    "vctk": {
        "manifest_path": "/data/vctk/manifest.json",
        "audio_dir": "/data/vctk/wav48",
    }
}

Manifest format

The manifest is a JSON-lines file where each line is a JSON object representing one utterance.

Minimum required fields

For models with fixed speaker context embeddings (no audio/text conditioning needed):

{
    "audio_filepath": "/path/to/audio.wav",
    "text": "The transcript text.",
    "duration": 3.5
}

Field	Type	Description
`audio_filepath`	string	Path to the target audio file
`text`	string	Text transcript to synthesize
`duration`	float	Audio duration in seconds

Run inference and evaluation

# Basic inference (no evaluation)
python examples/tts/magpietts_inference.py \
    --nemo_files "nvidia/magpie_tts_multilingual_357m" \
    --datasets_json_path /path/to/evalset_config.json \
    --out_dir /path/to/output \
    --codecmodel_path "nvidia/nemo-nano-codec-22khz-1.89kbps-21.5fps" \
    --use_cfg \
    --cfg_scale 2.5

# Inference with evaluation
python examples/tts/magpietts_inference.py \
    --nemo_files "nvidia/magpie_tts_multilingual_357m" \
    --datasets_json_path /path/to/evalset_config.json \
    --out_dir /path/to/output \
    --codecmodel_path "nvidia/nemo-nano-codec-22khz-1.89kbps-21.5fps" \
    --run_evaluation \
    --use_cfg \
    --cfg_scale 2.5

Outputs and evaluation metrics

After running, you'll find:

Generated audio files in <out_dir>/<checkpoint_name>/
Evaluation metrics in metrics.json
Visualization plots (if evaluation enabled)

When --run_evaluation is enabled, the following metrics are computed:

Metric	Description
CER	Character Error Rate (lower is better)
WER	Word Error Rate (lower is better)
SSIM (pred-gt)	Speaker similarity between predicted and ground truth
SSIM (pred-context)	Speaker similarity between predicted and context
UTMOSv2	Audio quality score (higher is better, requires `utmosv2` package)
RTF	Real-time factor (processing time / audio duration)

Software & Hardware

Runtime Engine(s):

NVIDIA NeMo Speech Framework 25.11
NVIDIA Riva 2.24.0

Acceleration Engine: Tensor(RT)-LLM, Triton

Supported Hardware Microarchitecture Compatibility:

NVIDIA Ada Lovelace L4
NVIDIA Ada Lovelace L40
NVIDIA Ampere A10
NVIDIA Ampere A30
NVIDIA Ampere A100
NVIDIA Hopper H100

Preferred/Supported Operating System(s):

Linux
Linux 4 Tegra

Our AI models are designed and/or optimized to run on NVIDIA GPU-accelerated systems. By leveraging NVIDIA’s hardware (e.g. GPU cores) and software frameworks (e.g., CUDA libraries), the model achieves faster training and inference times compared to CPU-only solutions.

The integration of foundation and fine-tuned models into AI systems requires additional testing using use-case-specific data to ensure safe and effective deployment. Following the V-model methodology, iterative testing and validation at both unit and system levels are essential to mitigate risks, meet technical and functional requirements, and ensure compliance with safety and ethical standards before deployment.

Training and Evaluation Datasets

Training Dataset

Language	Dataset
Arabic (UAE, ar-AE)	UAE Arabic Speech Recognition Corpus (Desktop)
Arabic (MSA, ar-MSA)	Publicly available internet-scale data (Internal)
Arabic (Saudi Arabia, ar-SA)	Classical Arabic Text-to-Speech (ClArTTS)
	Arabic (Saudi Arabia) Speech Recognition Corpus (Desktop)
	Publicly available internet-scale data (Internal)
Arabic (Syrian, ar-SY)	Arabic Speech Corpus
Chinese (zh)	AISHELL-3
	Emilia-YODAS Chinese
	Publicly available internet-scale data (Internal)
	Riva Speakers (Proprietary)
English (en)	David AI (Internal)
	GLOBE-V2
	HiFiTTS
	HiFiTTS-2
	LibriTTS
	Publicly available internet-scale data (Internal)
	Riva Speakers (Proprietary)
French (fr)	CML-TTS French
French (fr)	Riva Speakers (Proprietary)
German (de)	CML-TTS German
German (de)	German Speech Recognition Corpus (Desktop)
Hindi (hi)	IndicSUPERB
	Multilingual and code-switching ASR Challenge Dataset – sub-task2
	Publicly available internet-scale data (Internal)
Italian (it)	CML-TTS Italian
Japanese (ja)	Japanese Anime Speech Dataset V5
	Common Voice Script Speech 17.0 Japanese
	Emilia-YODAS Japanese
	Japanese Conversational Speech Recognition Corpus (Mobile)
Korean (ko)	Emilia-YODAS Korean
Portuguese (pt)	CML-TTS Portuguese
	Brazilian Portuguese Merged Speech Dataset
	TTS-Portuguese Corpus
Spanish (es)	CML-TTS Spanish
	Spanish (Spain) Conversational Smartphone (ESP_ASR003)
	Riva Speakers (Proprietary)
Vietnamese (vi)	Riva Speakers (Proprietary)
	InfoRe-1
	Publicly available internet-scale data (Internal)

Audio Training Data Size

54,305 Hours

Properties:

Number of data items in training set: ~54.3k hours
Data modalities: text and audio
Nature of the content: Audiobooks, Daily Conversations, podcast interviews, News, Twitter, Youtube
Languages: 12 languages (Arabic (ar), Chinese (zh), English (en), French (fr), German (de), Hindi (hi), Italian (it), Japanese (ja), Korean (ko), Portuguese (pt), Spanish (es), Vietnamese (vi))
Sensor Type for Data Collection: Microphones

Evaluation Dataset

Language	Dataset
Arabic (ar)	Internal
Chinese (zh)	Internal
English (en)	LibriTTS test-clean
English (en)	GLOBE-V2 accent test set
French (fr)	CML-TTS French
German (de)	CML-TTS German
Hindi (hi)	Internal
Italian (it)	CML-TTS Italian
Japanese (ja)	Internal
Korean (ko)	Internal
Portuguese (pt)	CML-TTS Portuguese
Spanish (es)	CML-TTS Spanish
Vietnamese (vi)	Internal

Evaluation data spans all 12 supported languages with the same modalities, content types, and collection method as the training data.

Evaluation

We report two metrics on per-language held-out test sets:

CER — Character Error Rate (%), lower is better.
SSIM — speaker similarity between prediction and context, higher is better.

Language / dataset	CER↓ (%)		SSIM↑ (pred-context)
Language / dataset	v2602	v2607	v2602	v2607
Arabic (ar)	—	1.62	—	0.806
Chinese (zh)	—	3.17	—	0.833
English (en)	0.34	0.37	0.835	0.822
English (en) accented	—	0.34	—	0.746
French (fr)	2.70	1.54	0.703	0.747
German (de)	0.66	0.80	0.626	0.742
Hindi (hi)	—	1.23	—	0.788
Italian (it)	—	2.19	—	0.773
Japanese (ja)	—	1.40	—	0.775
Korean (ko)	—	2.69	—	0.807
Portuguese (pt)	—	2.91	—	0.753
Spanish (es)	1.14	0.60	0.715	0.793
Vietnamese (vi)	—	0.59	—	0.725

Technical Limitations & Mitigations

There are two modes of inference, namely, standard and long-form. In standard mode, this model can generate up to 20 seconds of speech at a time in any of the 12 supported languages. In long-form mode, the model performs optimally when the input text contains punctuation and capitalization. The model was trained on a mix of publicly available speech datasets and internally recorded datasets in 12 languages. As a result, it is not suitable for speech generation in any language other than the 12 languages mentioned. We have removed zero-shot capabilities of this model for this release. Text normalization is required.

Ethical Considerations

NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their internal model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse.

Please report model quality, risk, security vulnerabilities or NVIDIA AI Concerns here.

License / Terms of Use

GOVERNING TERMS: Use of this model is governed by the NVIDIA Open Model License Agreement.

References

Shehzeen Hussain et al., "Align2speak: Improving TTS for Low Resource Languages via ASR-Guided Online Preference Optimization", ICASSP, 2026.
Roy Fejgin et al., "Frame-Stacked Local Transformers for Efficient Multi-Codebook Speech Generation", ICASSP, 2026.
Shehzeen Hussain et al., "Koel-TTS: Enhancing LLM based Speech Generation with Preference Alignment and Classifier Free Guidance", EMNLP, 2025, ACL.
Edresson Casanova et al., "NanoCodec: Towards High-Quality Ultra Fast Speech LLM Inference", Interspeech, 2025.
Ryan Langman et al., "HiFiTTS-2: A Large-Scale High Bandwidth Speech Dataset", Interspeech, 2025.
Edresson Casanova et al., "Low Frame-rate Speech Codec: a Codec Designed for Fast High-quality Speech LLM Training and Inference", ICASSP, 2025.
Paarth Neekhara et al., "Improving Robustness of LLM-based Speech Synthesis by Learning Monotonic Alignment", Interspeech, 2024.

Downloads last month: 1,335

Model tree for nvidia/magpie_tts_multilingual_357m

Finetunes

6 models

Quantizations

1 model

Datasets used to train nvidia/magpie_tts_multilingual_357m

Spaces using nvidia/magpie_tts_multilingual_357m 7

Collection including nvidia/magpie_tts_multilingual_357m

Nemotron Speech

Collection

Open, state-of-the-art, production‑ready enterprise speech models from the NVIDIA Speech research team for ASR, TTS, Speaker Diarization and S2S • 13 items • Updated 5 days ago • 59

Papers for nvidia/magpie_tts_multilingual_357m

Frame-Stacked Local Transformers For Efficient Multi-Codebook Speech Generation

Paper • 2509.19592 • Published Jan 22

Align2Speak: Improving TTS for Low Resource Languages via ASR-Guided Online Preference Optimization

Paper • 2509.21718 • Published Sep 26, 2025

NanoCodec: Towards High-Quality Ultra Fast Speech LLM Inference

Paper • 2508.05835 • Published Aug 7, 2025 • 1

HiFiTTS-2: A Large-Scale High Bandwidth Speech Dataset

Paper • 2506.04152 • Published Jun 4, 2025

Koel-TTS: Enhancing LLM based Speech Generation with Preference Alignment and Classifier Free Guidance

Paper • 2502.05236 • Published Feb 7, 2025 • 1

Article mentioning nvidia/magpie_tts_multilingual_357m

Scaling Real-Time Voice Agents with Cache-Aware Streaming ASR