Speech + LLMs @nvidia | Previously: @Meta MSL, @jhuclsp, @IITGuwahati
New York, NY
Joined September 2009
- Tweets2.4K
- Following2.3K
- Followers4.6K
- Likes8.6K
Looking at earlier speech models, their training data, and how people use them in products and everyday use cases, one thing stood out: the same recording can need different outputs.
Sometimes you want every word and filler. Sometimes you want clean, formatted text. Sometimes you want to add context, like names or technical terms, to help the model recognize them. Or you want a translation or an answer to a question about the recording.
Users should be able to ask for what they need.
This idea helped shape #Granary-v2.
We expanded from 25 to 48 languages and improved label quality through better ASR labeling, transcript refinement, and quality filtering. We also built labels and instruction pairs for different transcription styles, contextual ASR, code-switching, translation, transcript-based captions, and question answering.
#Granary-v2 is part of the training data behind #Nemotron-3.5-Transcribe, which lets users control transcription style, provide context, translate speech, and ask questions about audio.
We’re sharing model weights, training recipes, a public subset of #Granary-v2 labels, and the curation pipeline built with NeMo Curator.
A lot of work from the NeMo Speech team. Excited to finally share it!
🤗 Model:
huggingface.co/nvidia/NVIDIA…
📊 Granary-v2 dataset:
huggingface.co/datasets/nvid…
📝 Granary-v2 technical report:
research.nvidia.com/labs/nem…
🛠️ Granary-v2 pipeline:
github.com/NVIDIA-NeMo/Curat…
#NVIDIA #NeMo #SpeechAI #SpeechRecognition #SpeechData #OpenSource
We ran @nvidia Nemotron 3.5 Transcribe through our speech-to-text benchmark.
It has the lowest word error rate on real phone calls of the 20 models we tested, at 2.48%. We score real calls separately from clean test clips, on conversation recordings licensed from Ocular.
Congrats to the NVIDIA team.
Full results: benchmarks.cekura.ai/stt
One SpeechLLM to do all kinds of transcription.
Fully open weight!
Check out details in the tech report: research.nvidia.com/labs/nem…
Congrats to @PiotrZelasko and team on a great release!
Meet NVIDIA Nemotron 3.5 Transcribe 🎙️
An open-weight, promptable 30B MoE model with ~4B active parameters that brings transcription in 40 languages, speaker attribution, translation, contextual prompting and audio understanding into one checkpoint.
Watch it keep track of who said what when everyone’s talking at once.
1/ 🧵
Desh Raj retweeted
NVIDIA's Nemotron 3.5 Transcribe ties for #1 among open weights Speech to Text models with 2.8% WER, using just 4B active parameters
Nemotron 3.5 Transcribe is NVIDIA's new open weights Speech to Text model for non-streaming transcription, with 30B total and 4B active parameters, and the most accurate NVIDIA model on our non-streaming Speech to Text leaderboard.
Key takeaways
➤ Accuracy: Nemotron 3.5 Transcribe scores 2.8% on the AA-WER Index, tied with Mistral's Voxtral Small for #1 among open-weight models. It leads Voxtral Small on AA-AgentTalk, our dataset of voice agent conversations, at 1.8% vs. 2.3%.
➤ Model size: With 4B active parameters (30B total), Nemotron 3.5 Transcribe sits on the open weights frontier of AA-WER vs. active parameters. It matches the accuracy of Mistral's Voxtral Small with 6x fewer active parameters (4B vs. 24B), while other open weights models of a similar size sit at around 4% WER.
Congratulations to @NVIDIAAI on the release!
See more details below ⬇️
Desh Raj retweeted
Meet NVIDIA Nemotron 3.5 Transcribe 🎙️
An open-weight, promptable 30B MoE model with ~4B active parameters that brings transcription in 40 languages, speaker attribution, translation, contextual prompting and audio understanding into one checkpoint.
Watch it keep track of who said what when everyone’s talking at once.
1/ 🧵
Yeah, I'm also curious from an academic perspective. Let's see if they release more technical details.
My critique was that you are claiming one of two things:
1. Training on spoken dialogue without text is novel.
--> not true, this has been done 4 years ago
2. Training on spoken dialogue without text is "better" for some definition of better
--> you don't have any empirical validation to claim this
Yup, there is a ton of paralinguistic information in speech which is not in text. And that's exactly why it's better to spend your compute learning that differential instead of the basic knowledge which can be gleaned from text, no?
A lot is being made of the fact that Dessa was trained only on human conversations, without any text or pretrained models.
First, dGSLM (arxiv.org/abs/2203.16502) already did this in 2022 with just 2k hours of Fisher data. Vendors now provide thousands of hours of high-quality 2-channel dialog off-the-shelf.
Second, why do you even want to do this? Humans spent hundreds of thousands of years before inventing writing systems, and then optimized it over 5k years to store knowledge very efficiently. dGSLM had to train only on dialogs because at the time, we didn't know have unified models that could train jointly on text and speech, but this problem was solved by autoregressive transformers. You can cut your training ~10x by letting the model learn from text first, then spoken dialog, like Moshi (and basically every other modern S2S model). Different != smart.
Making very natural chit-chat bots is not where the frontier lies. Demos are cool, but the real challenge is intelligence, agentic behavior, and delegation. I would have been blown away if the naturalness was accompanied by GPT-Live style function calling, but alas!
All voice data vendors now provide thousands of hours of 2-channel dialog data off the shelf. This was nonexistent two years ago.
Desh Raj retweeted
When we unveiled Moshi two years ago, we also showed an early checkpoint we'd trained just for fun, directly on Fisher (phone calls from the late 90s and early 2000s). As a result, every time you called, you'd get to chat with a random person straight out of that era. That was trained on just 2k hours of data. Today, you can get 100s of thousands of hours of full-duplex conversational data. There's an obvious scaling law at play here!
We believe voice is the most natural way to interact with intelligence, so we created Dessa, the most authentic full-duplex speech model ever built. Talking with it feels like talking with a person…full of emotion, nuance, pauses, and a natural cadence.
block.xyz/dessa
Do people not know that Moshi has been around for over 2 years?
We believe voice is the most natural way to interact with intelligence, so we created Dessa, the most authentic full-duplex speech model ever built. Talking with it feels like talking with a person…full of emotion, nuance, pauses, and a natural cadence.
block.xyz/dessa
Desh Raj retweeted
idk I try to help youngsters high schoolers etc in my DMs if I can be helpful
this is the reason why I don't like SF, feels like it's full of social hierarchies where numbers/markers matter
not as much in Europe
