Speech + LLMs @nvidia | Previously: @Meta MSL, @jhuclsp, @IITGuwahati
New York, NY
Joined September 2009
- Tweets2.4K
- Following2.3K
- Followers4.6K
- Likes8.6K
Looking at earlier speech models, their training data, and how people use them in products and everyday use cases, one thing stood out: the same recording can need different outputs.
Sometimes you want every word and filler. Sometimes you want clean, formatted text. Sometimes you want to add context, like names or technical terms, to help the model recognize them. Or you want a translation or an answer to a question about the recording.
Users should be able to ask for what they need.
This idea helped shape #Granary-v2.
We expanded from 25 to 48 languages and improved label quality through better ASR labeling, transcript refinement, and quality filtering. We also built labels and instruction pairs for different transcription styles, contextual ASR, code-switching, translation, transcript-based captions, and question answering.
#Granary-v2 is part of the training data behind #Nemotron-3.5-Transcribe, which lets users control transcription style, provide context, translate speech, and ask questions about audio.
We’re sharing model weights, training recipes, a public subset of #Granary-v2 labels, and the curation pipeline built with NeMo Curator.
A lot of work from the NeMo Speech team. Excited to finally share it!
🤗 Model:
huggingface.co/nvidia/NVIDIA…
📊 Granary-v2 dataset:
huggingface.co/datasets/nvid…
📝 Granary-v2 technical report:
research.nvidia.com/labs/nem…
🛠️ Granary-v2 pipeline:
github.com/NVIDIA-NeMo/Curat…
#NVIDIA #NeMo #SpeechAI #SpeechRecognition #SpeechData #OpenSource
We ran @nvidia Nemotron 3.5 Transcribe through our speech-to-text benchmark.
It has the lowest word error rate on real phone calls of the 20 models we tested, at 2.48%. We score real calls separately from clean test clips, on conversation recordings licensed from Ocular.
Congrats to the NVIDIA team.
Full results: benchmarks.cekura.ai/stt
One SpeechLLM to do all kinds of transcription.
Fully open weight!
Check out details in the tech report: research.nvidia.com/labs/nem…
Congrats to @PiotrZelasko and team on a great release!
Meet NVIDIA Nemotron 3.5 Transcribe 🎙️
An open-weight, promptable 30B MoE model with ~4B active parameters that brings transcription in 40 languages, speaker attribution, translation, contextual prompting and audio understanding into one checkpoint.
Watch it keep track of who said what when everyone’s talking at once.
1/ 🧵
I don't have any accepted papers at NeurIPS, and not attending InterSpeech in Sydney either.
But I advanced to French level B1 yesterday, and I'm happy to show you my new, big-boy textbooks 🚀
A good diarization system is at the heart of any speech data processing pipeline.
Check this out 👇
Nemotron 3 Diarization ranked #1 of 12 systems in @voicearena_ai's initial Diarization-Bench results. Its 14.72% error rate was ~24% lower than the runner-up.
Here’s a four-speaker comparison from the benchmark.
Full results: voicearena.com/diarization-b…
This is the time of your life if you want to make a career in academia with minimal effort. The algorithm is simple:
1. Take any published paper.
2. Go to the "Future Work" section.
3. Task an agent with designing an experiment based on the future work text and run it.
4. Once the experiment provides results that are at least marginally better than those shown in the other paper, task an agent with writing a follow-up paper.
5. Submit to all second- and third-tier conferences.
6. Repeat the process thousands of times per month.
The law of large numbers always works.
A good way to think about this is backwards from the use case. Right now, there are 2 big applications of these systems: (1) enterprise voice agents, and (2) personal companion (including assistants, language coach, tutor, etc.).
In both cases, evals should be along 2 axes: (A) accuracy and (B) experience, and the goal should be to move the pareto frontier (for example, see the figure below from EVA-Bench).
For the voice agent use case, measuring accuracy is easy by tracking the application state and checking task completion; this is what Tau-Voice pioneered. It is not so well defined for the companion use case, this is a good problem for academic researchers to think about.
Measuring experience is more difficult, but some combination of response latency/rate and yield latency/rate is usually measured as a proxy. Personally, I think "experience" should be scored on the ELO scale through pairwise comparisons between sessions, since an "ideal" response/yield rate for an enterprise agent would be very different from a language tutor.
Beyond the metrics themselves, we also need good user simulators to run auto-evals during development. This goes hand-in-hand with model improvement on instruction following; the original Tau-Voice used a cascade with a text LLM and ElevenLabs, but GPT-Live uses GPT-Live itself as the user. I think they should at least use a different version of GPT-Live trained with a different (more diverse) set of voices. Building good open-source user voice simulators is another interesting and impactful direction.
A big question is about the *several* τ-Voice numbers:
- 86.2% on OpenAI blog
- 81.7% on τ-Voice leaderboard
- 67.9% on Artificial Analysis
Stochasticity explains some of this, but 20%?
The backend also matters a lot (see Astra v/s Sol difference).
But OpenAI's CRAWL/WALK/RUN eval harness is worth reading for anyone interested in voice agent evals! Check it out at developers.openai.com/cookbo…
Full post ↑
