
Speech + LLMs @nvidia | Previously: @Meta MSL, @jhuclsp, @IITGuwahati
New York, NY
Joined September 2009
- Tweets2.4K
- Following2.3K
- Followers4.6K
- Likes8.6K
I’m happy to share that I’m starting a new position as Senior Research Scientist at @nvidia!
Looking forward to open science for speech full-duplex models :)
After 2 wonderful years, I left Meta this week. During this time, I worked on several projects related to speech and LLMs:
- Built the first multi-channel audio foundation model with M-BEST-RQ (arxiv.org/abs/2409.11494)
- Made ASR with SpeechLLMs faster (arxiv.org/abs/2409.08148) and more accurate (ieeexplore.ieee.org/document…)
- Shipped the first production-ready full-duplex voice assistant (about.fb.com/news/2025/04/in…)
- Improved Moshi’s reasoning capability with chain-of-thought (arxiv.org/abs/2510.07497)
I am grateful to my managers for having my back on critical projects, and fortunate to have collaborated with several brilliant researchers and engineers during this time.
As to what's next, I am still in NYC and continuing to do speech research. More on that later!
Looking at earlier speech models, their training data, and how people use them in products and everyday use cases, one thing stood out: the same recording can need different outputs.
Sometimes you want every word and filler. Sometimes you want clean, formatted text. Sometimes you want to add context, like names or technical terms, to help the model recognize them. Or you want a translation or an answer to a question about the recording.
Users should be able to ask for what they need.
This idea helped shape #Granary-v2.
We expanded from 25 to 48 languages and improved label quality through better ASR labeling, transcript refinement, and quality filtering. We also built labels and instruction pairs for different transcription styles, contextual ASR, code-switching, translation, transcript-based captions, and question answering.
#Granary-v2 is part of the training data behind #Nemotron-3.5-Transcribe, which lets users control transcription style, provide context, translate speech, and ask questions about audio.
We’re sharing model weights, training recipes, a public subset of #Granary-v2 labels, and the curation pipeline built with NeMo Curator.
A lot of work from the NeMo Speech team. Excited to finally share it!
🤗 Model:
huggingface.co/nvidia/NVIDIA…
📊 Granary-v2 dataset:
huggingface.co/datasets/nvid…
📝 Granary-v2 technical report:
research.nvidia.com/labs/nem…
🛠️ Granary-v2 pipeline:
github.com/NVIDIA-NeMo/Curat…
#NVIDIA #NeMo #SpeechAI #SpeechRecognition #SpeechData #OpenSource
We ran @nvidia Nemotron 3.5 Transcribe through our speech-to-text benchmark.
It has the lowest word error rate on real phone calls of the 20 models we tested, at 2.48%. We score real calls separately from clean test clips, on conversation recordings licensed from Ocular.
Congrats to the NVIDIA team.
Full results: benchmarks.cekura.ai/stt
One SpeechLLM to do all kinds of transcription.
Fully open weight!
Check out details in the tech report: research.nvidia.com/labs/nem…
Congrats to @PiotrZelasko and team on a great release!
Meet NVIDIA Nemotron 3.5 Transcribe 🎙️
An open-weight, promptable 30B MoE model with ~4B active parameters that brings transcription in 40 languages, speaker attribution, translation, contextual prompting and audio understanding into one checkpoint.
Watch it keep track of who said what when everyone’s talking at once.
1/ 🧵
Desh Raj retweeted
NVIDIA's Nemotron 3.5 Transcribe ties for #1 among open weights Speech to Text models with 2.8% WER, using just 4B active parameters
Nemotron 3.5 Transcribe is NVIDIA's new open weights Speech to Text model for non-streaming transcription, with 30B total and 4B active parameters, and the most accurate NVIDIA model on our non-streaming Speech to Text leaderboard.
Key takeaways
➤ Accuracy: Nemotron 3.5 Transcribe scores 2.8% on the AA-WER Index, tied with Mistral's Voxtral Small for #1 among open-weight models. It leads Voxtral Small on AA-AgentTalk, our dataset of voice agent conversations, at 1.8% vs. 2.3%.
➤ Model size: With 4B active parameters (30B total), Nemotron 3.5 Transcribe sits on the open weights frontier of AA-WER vs. active parameters. It matches the accuracy of Mistral's Voxtral Small with 6x fewer active parameters (4B vs. 24B), while other open weights models of a similar size sit at around 4% WER.
Congratulations to @NVIDIAAI on the release!
See more details below ⬇️
Meet NVIDIA Nemotron 3.5 Transcribe 🎙️
An open-weight, promptable 30B MoE model with ~4B active parameters that brings transcription in 40 languages, speaker attribution, translation, contextual prompting and audio understanding into one checkpoint.
Watch it keep track of who said what when everyone’s talking at once.
1/ 🧵
A lot is being made of the fact that Dessa was trained only on human conversations, without any text or pretrained models.
First, dGSLM (arxiv.org/abs/2203.16502) already did this in 2022 with just 2k hours of Fisher data. Vendors now provide thousands of hours of high-quality 2-channel dialog off-the-shelf.
Second, why do you even want to do this? Humans spent hundreds of thousands of years before inventing writing systems, and then optimized it over 5k years to store knowledge very efficiently. dGSLM had to train only on dialogs because at the time, we didn't know have unified models that could train jointly on text and speech, but this problem was solved by autoregressive transformers. You can cut your training ~10x by letting the model learn from text first, then spoken dialog, like Moshi (and basically every other modern S2S model). Different != smart.
Making very natural chit-chat bots is not where the frontier lies. Demos are cool, but the real challenge is intelligence, agentic behavior, and delegation. I would have been blown away if the naturalness was accompanied by GPT-Live style function calling, but alas!
Desh Raj retweeted
When we unveiled Moshi two years ago, we also showed an early checkpoint we'd trained just for fun, directly on Fisher (phone calls from the late 90s and early 2000s). As a result, every time you called, you'd get to chat with a random person straight out of that era. That was trained on just 2k hours of data. Today, you can get 100s of thousands of hours of full-duplex conversational data. There's an obvious scaling law at play here!
We believe voice is the most natural way to interact with intelligence, so we created Dessa, the most authentic full-duplex speech model ever built. Talking with it feels like talking with a person…full of emotion, nuance, pauses, and a natural cadence.
block.xyz/dessa
Do people not know that Moshi has been around for over 2 years?
We believe voice is the most natural way to interact with intelligence, so we created Dessa, the most authentic full-duplex speech model ever built. Talking with it feels like talking with a person…full of emotion, nuance, pauses, and a natural cadence.
block.xyz/dessa
Desh Raj retweeted
idk I try to help youngsters high schoolers etc in my DMs if I can be helpful
this is the reason why I don't like SF, feels like it's full of social hierarchies where numbers/markers matter
not as much in Europe
I love full-duplex models and GPT-Live is awesome, but this claim is dubious at best.
From Josh's post, most of the models hold up under "street noise" and "cafe chatter", but fail under "background TV", despite all of them being added at low energy levels. The only conclusion that we can draw from this is that most models are trained with noise augmentation resembling ambient noises or on in-the-wild data where it is unlikely to find TV noise.
On the TV noise scenario itself, GPT-Live degraded by ~39% while Grok degraded only by 15%, despite not being a full-duplex model. From xAI's blog post: "We've focused on making Grok Voice Think Fast 2.0 perform exceptionally well in real-world settings, with substantial background noise and telephony compression." In short, they trained with a lot of real-world noise augmentation.
Speech folks have worked on noise robustness and target-speaker recognition for a decade now, and came up with several cool architectural tricks. But if you look at ASR recipes from Kaldi/ESPNet, the only constant is noise augmentation and far-field simulation.
Desh Raj retweeted
How to kill a voice agent? Just turn on the TV ☠️
We benchmarked 14 voice agents with a TV playing in the background.
Task completion fell from 38.7% to 8.6%
🧵/
Introducing ChatGPT for Dishwashing
GPT-6 achieves state-of-the-art results on dishwashing benchmarks for long-running jobs and difficult stains.
The Navier–Stokes Millennium Prize Problem is a $1 million challenge about whether the equations describing fluid motion can break down. In this video, I accidentally found a counter-example while trying to optimize my dishwasher. But in reality, OpenAI used 10,000 AI agents to find a proof in 88 hours: openai.com/index/navier-stok…
Parody of the original GPT-6 Astra ad
@OpenAI #ChatGPT_Partner #chatgpt
Credits:
Written and Directed by Joma
Co-Writer - Henry Connor Coan IV @coan_henry
Production by Dauntlus Studios @dauntlus.studios
Producer/Co-Director - Marcus Liew @dauntlus
Camera Operator - Amos Lee @amoselijahlee
Art Director - Alethea Soo @aletheasoo
Art Assistant - Alicia Lim @aliciatkl_
Assistant Editor - GPT-6 Astra
Music: Tensions Run High by Soundridemusic
youtube.com/watch?v=Ly9H63SL…
I think I'm going to get a lot of mileage out of this one.
🤖 Made with AI
Mathematicians are coping so hard they might abolish irrational numbers again and turn the field into a guild
terrytao.wordpress.com/2026/…
Desh Raj retweeted
love how in 90s and 2000s romcoms, nyc was never just a backdrop but a character of its own
this was not in my bingo card for 2026
PSA: @mervenoyann is destined to teach @pewdiepie about local agents and post-training.
upvote her plea and make a dream come true.
github.com/odysseus-dev/odys…
We’re looking for our next cohort of graduate fellows.
Our 2027–2028 Graduate Fellowship Program is open to Ph.D. students worldwide with awards up to $60,000, plus mentorship and technical support.
Apply by October 30: nvda.ws/4rGKDhv
🤖 Made with AI
Desh Raj retweeted
Sigh.
I wrote this post below, but then I didn't post it because I am tired of the consistent pushback I get from folks with the same safety worldview as Anthropic. I guess that means I should send it. Here goes!
This post is pretty solipsistic and doesn't properly take recent events in cyber risks into it's discussion. It is really hard for me to watch how the US frontier labs like Anthropic don't have the ability to consider other approaches to safety and ways things could play out.
It insinuates that Z ai (Chinese lab, builds GLM series) doesn't really care about safety and is reckless to take their business strategy.
Saying things like "Given this evidence, we think it's likely both state and non-state actors will use models like GLM-5.3 to cause real-world harm." and "This is unlike any other similarly capable AI model, all of which were released with safeguards or through limited access programs." while closed models have been used on more of the documented cyber attacks is just bowing out of the interesting question.
A plausible view is that open model weights and closed model apis (with some safe guards) are both far closer to being easy to mis-use, rather than API models being closer to safe. The trope "Open Dangerous, Closed Safe" may be closer to "Open Unsafe, Closed Unsafe"
Closed models have stronger capabilities and stronger safeguards, but the stronger capabilities part could matter more in net harm if both the safeguards are porous.
At the same time, as the authors do acknowledge (thanks - thats progress!), open models without extreme cyber guardrails are important to rapidly diffuse cyber readiness in the economy -- as programs like project glasswing are not perfect in getting all critical industry onboarded.
There will be more issues like the time when Fable was released and Amazon found a workaround, which allowed them to access the full capabilities of the model served readily at an API.
The blog overall is reasonable in it's narrow line, but it's a very effective tool in a complicated, rapidly evolving media ecosystem to reinforce a certain type of safety thinking.
Desh Raj retweeted
Who wants more speech training data?
YODAS v3 is now available @huggingface
It’s 1.1M hours - the biggest audio dataset ever. And the first at this scale with stereo audio at 48kHz, with timestamped transcripts and translations. 100+ langs
hf.co/blog/espnet/yodasv3
CC-BY-3.0































