Working to make machines understand the world like human beings. Words are my own.
Santa Clara
Joined July 2020
- Tweets265
- Following805
- Followers7.9K
- Likes491
Pinned Tweet
Better multimodal models need better spatial data.
Today we’re open-sourcing LocateAnything-Data — a high-quality, training-ready spatial data collection with 12M images, 138M queries, and 785M annotations spanning multi-object detection, referring expression grounding, physical AI, GUI understanding, OCR, and document understanding.
With the community response to LocateAnything, we believe that making its training pipeline more reproducible would be valuable to the community. Alongside the data, we’re also releasing:
• Unified spatial annotations
• JSONL annotations
• Indexed WebDataset TARs
• Megatron-Energon configs
• Source-media mappings
• Visualization & data loading examples
LocateAnything-Data integrates many outstanding public datasets under a unified spatial annotation format while fully respecting their original licenses and usage terms. For datasets with redistribution restrictions, we provide annotations, media mappings, and local build tools instead of redistributing the original images. Huge thanks to all the researchers and open-source contributors whose datasets, tools, and models made this release possible.
📦Dataset: huggingface.co/datasets/NVEa…
📄Paper: arxiv.org/abs/2605.27365
🤗Model: huggingface.co/nvidia/Locate…
Zhiding Yu retweeted
Hi all, I have a new result I'm excited to share. It turns out that *all existing implementations* of softmax and cross-entropy (that's JAX, PyTorch, textbooks, etc) have a subtle bug --- two bugs, actually. A simple rewrite fixes both, and eliminates "slingshots" when grokking:
Zhiding Yu retweeted
Interested in Internship at NVIDIA? Apply for a 2027 internship at LPR research.nvidia.com/labs/lpr…! 🤖
We’re looking for interns to help shape the future of intelligent systems that interact with the physical world.
Apply: forms.gle/afgoy2doFT7akpqJ6
#EmbodiedAI #NVIDIA #Internship
Zhiding Yu retweeted
Introducing LAGEN: Latency-Aware Agents.
A 1-frame delay is invisible to humans, but that delay can make an agent lose 95% performance on real-time tasks.
AI models like Astra and VLAs can already get to real-time loops. But to keep them alive, we need Latency Awareness.🧵
An important step towards building powerful data infra for GR00T and robotics community.
Congrats @DrJimFan and team!
We built a robotics data engine with Astra-level performance at 125x lower cost. And we are open-sourcing the weights today! Introducing Flora, a suite of open models to parse the physical world and turn raw footage into rich training signals.
Flora already powers our in-house data engine, labeling 200K hours of videos a day on 500 GPUs. We created it to scale the frontier for robot learning. Now you own the weights too.
Flora gives you:
- High-quality, timestamped captions to curate and filter human egocentric and robot data;
- Dense progress rewards for quality control and RL post-training;
- Rubrics scoring for real-world benchmarking
Try it today, links in thread:
Zhiding Yu retweeted
Introducing Flora, an open model suite for scaling robotics data engines.
It watches robot and human video and annotate what happened (timestamped instructions) and how well it went (rewards and failure reasons).
🔗 research.nvidia.com/labs/gea…
We use it to train NVIDIA GR00T. 🧵
Zhiding Yu retweeted
🚀 The future of Physical AI lies not in scaling more RGB data, but in scaling physics.
Introducing PrismWAM: a new WAM that decouples semantic context from physical state.
No action pretraining. SOTA results across 5 benchmarks and real world.
🔗 prism-wam.github.io
Zhiding Yu retweeted
We are open-sourcing V1.0 of our agentic robotics framework, ASPIRE, with full real robot skill discovery and self improvement capabilities! On the day Astra arrived, our lab switched the backend model and saturated all the benchmarks to ~99% over night.
And the frontier is already on the move:
- Opus 5.5 slightly edges out Astra on our robot simulation bench.
- Sonnet 5.5 hits the Pareto frontier of cost vs. robot capability, $12/task.
We first introduced ASPIRE and /robot skill library in June, way before the big models could reliably do robotics. A rising frontier tide lifts all boats, but Astra was a 0->1 tsunami on many tasks.
Now you can also feel the Physical Bitter Lesson at home like we did on that fateful day ;)
links in thread
Zhiding Yu retweeted
Excited to share a new repo: Sooool!
Sol itself playing games, frame by frame. No code-as-a-policy!
The environment changes while Sol processes the images. When an action arrives, it may no longer be correct. We built a research loop that accounts for this delay:
- Run up to eight model requests in parallel. Sol keeps looking at new frames and choosing actions while its earlier decisions are still in flight.
- Profile the target API and simulate its delays using cheaper inference during development. We only simulate the delay itself: Sol still sees the frames and chooses what to do at each step.
- Give Sol latency mean, std, P90 and P95. When it looks at a frame, it can plan for where things will be when its action arrives.
- Run episodes, inspect failures, update Sol’s instructions, and let it play again. Sol decides what lessons can enter later prompts.
We expect to spend about $50 per task on paid API calls, primarily for final verification rather than development. Most development runs use cheaper inference with simulated latency.
Touhou ↓
Sol chooses movement, shooting, focus and bombs from frames.
Findings from Sol in the auto-research loop:
- Predict where the boss will be, not where she was.
- Check the move AND the waiting position.
- More damage is not enough if it costs more hits..
Repo, experiment details, and the reproduction guide below. ↓
Zhiding Yu retweeted
Welcome to the era of embodied AI.
We are launching AA-Robotics v0.1, our research preview that measures frontier language model performance on understanding, planning, and acting in the physical world. In v0.1, each model is given control of a 6-axis robot arm (5 joints and a gripper) and a simple objective. We’re releasing results for our first eight tasks today.
We find that frontier language models in coding agent harnesses have recently become capable of directly manipulating robot arms for simple tasks. Today, they are far too slow and expensive for any kind of production usage - nearly two orders of magnitude worse than human baselines. We expect this to change quickly.
GPT-6 Astra places first in AA-Robotics v0.1, followed by GPT-6.1 Sol, Gemini 4 Argon and Claude Opus 5.5.
AA-Robotics v0.1 details:
➤ Our first eight tasks test a range of capabilities: control and dexterity (precise grasping and object handling), spatial reasoning (how objects interact with one another) and adaptability (how models respond to the unexpected)
➤ Models get only brief instructions: how to connect to the robot arm (via a simple HTTP API), the arm's name and how to submit an attempt. They can view the scene through four cameras and write and run code to plan and send movements
➤ Once a model submits an attempt, its end state is graded against the task's pass criteria. A pass scores 100; an incomplete attempt earns partial credit. All attempts are zero-shot
Key findings:
➤ The GPT family tops the leaderboard: GPT-6 Astra scores 69, ahead of GPT-6.1 Sol at 65 and Gemini 4 Argon at 55. Claude Opus 5.5 (49) and Claude Fable 5.1 (46) trail
➤ Today’s models are incredibly slow at our tasks, taking ~30 minutes on average to complete each tabletop task
➤ Cost varies widely: GPT-6.1 Sol is the cheapest at $1.93 per task on average; GPT-6 Astra, the most expensive, costs $16.36, more than eight times as much
➤ Compared with a human, today's best-performing frontier language models are ~100x slower and ~10x more expensive
More results, clips, and how AA-Robotics v0.1 works follow in the thread below.
Check out this cool work from @Isabella__Liu and our NVIDIA colleagues! An important step towards autonomy in robot data scaling.
Can we scale robot learning without scaling human labor? 📈
Demonstrations, resets, failure recovery, constant supervision—collecting robot data still takes substantial human effort.
🔑 Three keys let agents take on more of the work:
1. Real2Sim2Real: Agents explore and learn in digital twins, where trial and error is safer and cheaper.
2. Autonomous Recovery: Robots learn to recover from unexpected failures instead of waiting for human resets.
3. Human-on-Demand Learning: Multiple robots run in parallel, and humans step in only when autonomous recovery falls short.
🥑 Recova brings these together to get more robot learning out of every hour of human effort.
research.nvidia.com/labs/gea…
Joint work with @anjjei, Johan Bjorck, @ZhidingYu, @yin_hongxu, @jankautz, @DrJimFan, @yukez, and @Sifei30488L.
👇 Here's how it works 🧵
One thing we never properly said: thank you.
We were genuinely humbled by the response to LocateAnything-3B. Seeing it reach #1 on Hugging Face Trending and remain near the top for nearly three weeks was far beyond what we expected.
Thank you to everyone who tried the model, shared feedback, opened issues, cited our work, or built upon it. Your support and encouragement made this release possible.
We hope LocateAnything-Data and the accompanying training pipeline will make it easier for the community to reproduce, extend, and build even stronger multimodal models.
Zhiding Yu retweeted
For my first post, I’m sharing a letter @NVIDIA signed on why open models matter.
AI will transform every industry, power every company, and be built by every country.
Open models strengthen safety and cybersecurity, accelerate innovation and diffusion, and enable sovereignty.
The world needs both frontier closed models and frontier open models.
images.nvidia.com/pdf/Open-W…
Zhiding Yu retweeted
Meet the first Sound World Model from Sonilo.
Generated video can look incredible, but without the right sound, it still feels distant.
Sonilo takes a video and creates the music and sound effects it needs, timed to the scene, motion, mood, and environment.
And in head-to-head benchmarks, Sonilo outperforms leading models across both music and sound effects.
With Sound Effects 1.0, we’re bringing music and sound effects together into one sound layer for videos.
Music carries emotion.
Sound effects create presence.
Together, they make worlds feel real, immersive, and alive!
Sound Effects 1.0 is live.
#SoundWorldModel #Sonilo #AIAudio
Zhiding Yu retweeted
Today, we enable AutoResearch in the physical world for the first time! Introducing ENPIRE: we give 8 Codex agents a fleet of robots, an allocation of GPUs, and generous token budget. We set them free with a simple goal: solve the task as quickly as possible, keep the robots busy but stay safe, don't waste precious compute. Make no mistake.
Then humans step aside and our watch begins. The robot fleet starts to come alive: they learn to look for visual clues, reset the scene, practice novel skills, tinker with control stack, read papers online, debate, reflect, get stuck, and try again directly on the hardware. All we did is to give Codex an API to the world of atoms, and the rest is emergence.
ENPIRE is able to solve high-precision tasks like tying zip-ties, organizing fine pins, and installing GPUs all by itself. We also discovered a new type of "physical scaling": 8 robots exploring in parallel improves significantly faster than fewer ones.
A part of our NVIDIA GEAR lab now self-improves tirelessly over night. We just read the reports in the morning.
/goal: we all take a holiday and Jensen wouldn't even notice ;)
We will be open-sourcing everything, so you can host your self-running robot lab at home too! Deep dive in the thread:
Zhiding Yu retweeted
Congrats to the @MiniMax_AI team on the release of MiniMax M3, a long-context multimodal model for text, image, and video reasoning. 🙌
Try it today with our free GPU-accelerated endpoint on build.nvidia.com.
Details: nvda.ws/4v4BWhD
MiniMax M3, Open-Weight, Now On Hugging Face , with only ~428B parameters and ~23B activated parameters
Weights:
huggingface.co/MiniMaxAI/Min…
MiniMax Sparse Attention:
huggingface.co/papers/2606.1…
We are presenting Fast-ThinkAct at 10:45am this morning at Poster #469.
Welcome to drop by and discuss!
🚀 Excited to share that our paper Fast-ThinkAct has been accepted to #CVPR2026! 🎉
Efficient Vision-Language-Action reasoning via verbalizable latent planning — enabling embodied agents to think fast internally without lengthy textual reasoning.
⚡ Achieves 9.3× faster inference (89% latency reduction) than ThinkAct-7B — bringing Reasoning VLA closer to real-time robotic control.
📄 arxiv.org/abs/2601.09708
🎥 jasper0314-huang.github.io/f…
🙌 Huge congrats to @chipinhxyz, @yunzeman, @ZhidingYu, @CMHungSteven, @jankautz, Yu-Chiang Frank Wang, @FuEnYang1
#EmbodiedAI #PhysicalAI #VLA #Robotics #NVIDIAResearch @NVIDIAAI @NVIDIARobotics
