Learning representations for intent-to-action. PhD student @ ZJU World Models · JEPA · INTACT CUPT Special Prize · CUPEC National 1st [email protected]
Joined July 2026
- Tweets186
- Following210
- Followers228
- Likes411
Pinned Tweet
LeWM predicts action effects, but searches 9,000 sequences to act.
Introducing INTACT: an end-to-end unified JEPA that completes the other half of LeWM: intent to action.
Only 1 epoch. 0 search.
95.33% Direct macro SR.
2.9-5.5 ms inference.
~300× lower planning latency.
Junhan Sun retweeted
🌟 Excited to share that our paper, RoboReason, has been accepted to EMNLP 2026!
🤖 We study how well Vision-Language Models (VLMs) perform on spatiotemporal ordering tasks that use real-world robotics images.
Our key findings:
🔹 Large VLMs often underperform previously reported results when evaluated on spatiotemporal ordering tasks using real-world robotics images.
🔹 Overlaying specialized computer vision model outputs and increasing reasoning effort generally help improve large VLM performance.
🔹 Finetuning smaller models using GRPO and SFT on examples generated by specialized computer vision models often helps them achieve comparable results to much larger VLMs.
📍 Presenting with @sagnikanupam at @emnlpmeeting in Budapest 🇭🇺 tomorrow!
🗓️ Poster Session C | 9:00–10:30 AM
#EMNLP2026 #VLMs #LLMs #Robotics #AI
Junhan Sun retweeted
Yann LeCun's (@ylecun ) latest talk at ETH Zürich
Scaling LLMs to reach AGI is "impossible"
A large language model is trained on about 30 trillion tokens, which is roughly 10^14 bytes of text and would take a person about 400,000 years to read.
A 4-year-old child receives about the same amount of data, 10^14 bytes, through vision alone in about 1 year and 10 months.
In his view, intelligence is the ability to learn new tasks quickly or perform them without prior training, as a teenager learns to drive in about 20 hours. Scaling increases stored knowledge, but it does not produce this ability to adapt.
----
From "Perfology Clips" YouTube channel, (link in comment)
Junhan Sun retweeted
Hi all, I have a new result I'm excited to share. It turns out that *all existing implementations* of softmax and cross-entropy (that's JAX, PyTorch, textbooks, etc) have a subtle bug --- two bugs, actually. A simple rewrite fixes both, and eliminates "slingshots" when grokking:
Junhan Sun retweeted
Can a language model be based purely on self-conditioning? 🤔
Yes! Meet LAW 👋 (a.k.a. Loopholing All the Way!)
🧵1/8
Junhan Sun retweeted
1/7 Introducing DeXplicit: making physical state explicit in dexterous manipulation world models.
Given an initial scene and a proposed hand trajectory, DeXplicit 3D forecasts how objects move, articulate and deform. The DeXplicit renderer turns that geometry into future video.
The intermediate 3D state makes the physical response inspectable and provides a control signal for visual generation. Joint-aware actions, coherent object motion and transported appearance connect the two stages.
Junhan Sun retweeted
In order to build an intelligent embodied agent, we need to be able to effectively remember what we have seen in the past.
Excited to share MEMI, where we treat memory learning as an inverse problem: given a set of demonstrations, we infer what an agent needs to remember to reproduce the demonstrated decisions.
MEMI is able to reach 97% success on RoboMME’s memory-dependent manipulation tasks.
Junhan Sun retweeted
Carrying a box while crouching under a low beam...
Looks like Amazon is gradually building up the skill set a delivery humanoid would need.
FIBER, a research project involving Amazon FAR, Carnegie Mellon University, and Stanford, using Unitree G1 to explore how humanoids can string different skills together.
The demos are worth a look--G1 picks up a box thrown to a random spot, grabs boxes from different positions, pushes open a heavy door by leaning its body into it, and crouches under a low beam while carrying a box.
Think about a delivery run-- bending down inside a delivery van to pick up a parcel, carrying it out, walking through a corridor, opening a door, navigating around obstacles, and getting the package to the right place. Each step calls for a different skill, and the robot has to move from one to the next smoothly to finish the job.
FIBER tackles this by training separate skills, bringing them into a shared behavior model, then using a planner to decide what to do next. A high-level planner updates at 5 Hz, while the lower-level model adjusts the robot’s movements at 50 Hz.
The system is trained in simulation and tested on a real Unitree G1.
It’s still early. The average success rate across three simulated tasks is around 42%, and end-to-end delivery hasn’t been demonstrated.
But the direction is interesting. Every new skill expands what a humanoid could handle in a warehouse or on a delivery route.
Junhan Sun retweeted
How should we approach research, when the very field we work on is going through rapid and potentially fundamental changes?
I recently gave a talk at the Stanford Vision Seminar, reflecting on my PhD journey and how AI is changing the way I think about research. I adapted part of it into a new blog post, organized around three questions:
1. What should we work on?
2. How should we work with AI?
3. What is a PhD for?
I hope some of my reflections are helpful. The talk ended with an open discussion, and I’d love to continue it here; curious how others are thinking about these questions.
Personally -- as much as there's FOMO, I also think life is very long, and we have time.
toruowo.github.io/blog/posts…
Junhan Sun retweeted
Researchers introduce RLCR (Reinforcement Learning with Calibration Rewards) to train language models on both correctness and confidence, significantly improving calibration and reducing hallucinations.
is.gd/9tqJNw
#ML #AI #LLM #DL #AgenticAI
ALT Beyond Binary Rewards: Training LMs to Reason about Their Uncertainty
Yuejiang, this is what I am looking for!
Instant learning！
Junhan Sun retweeted
A Fix for Softmax and Cross-Entropy
Barron et al. identify numerical bugs in standard softmax and cross-entropy, providing fixes that eliminate the "Slingshot" effect and prevent student networks from drifting from teachers during distillation.
arxivb.org/abs/2610.31642
Junhan Sun retweeted
MLP-SPLATTING: 3D SCENES WITH A FRACTION OF THE MEMORY COST
A new method for creating photorealistic 3D scenes uses compact neural elements instead of millions of individual Gaussians.
The reported gains are significant:
6x less memory usage.
2x faster rendering.
Editable scenes: select, move, recolor, or remove individual objects.
One detail stands out: objects can be selected without manually segmenting the scene beforehand.
That could make editing complex 3D environments much easier, especially when working with large scenes and limited computing resources.
For AI-powered content production, this matters because generating a scene is only part of the workflow. Being able to efficiently edit and reuse its components is what makes the output practical.
Lower memory requirements and faster rendering could make 3D workflows more accessible without sacrificing visual quality.
Research and code:
github.com/kxhit/mlp-splatti…
Junhan Sun retweeted
A key difference between continuous and discrete diffusion for language is that the former is (in principle) able to represent uncertainty in intermediate noisy states (i.e. superpositions of possible outcomes).
Binary discrete diffusion is another way to achieve this. Neat!
Junhan Sun retweeted
You hold an egg and a hammer with almost the same hand shape.
The force behind each grip is nothing alike.
A camera can't see that difference.
I came across this demo of Physical AI that pairs vision with sEMG (surface electromyography): a wristband that senses muscle activation, fatigue and coordination in real time.
What amazes me is how accurate the hand tracking is.
If you thought vision alone could track a hand well, this proves otherwise.
Vision is necessary. It tells you the pose of the hand.
But it doesn't give you the full picture.
sEMG tells you the force behind that pose. And for manipulation, force is everything. Grip the egg like the hammer and it's gone.
This kind of multimodal input matters even more for the foundation models learning manipulation. Vision alone is not going to solve it.
The real question now: will it generalize?
Junhan Sun retweeted
Let robots speak physics. 🤖
Language and code have shown the power of reusable abstractions. We believe physical experience needs a vocabulary, too.
For robots, that experience lives in the details: lifting thin paper without slipping, inserting a shuttlecock without jamming, and adapting as contact changes.
Introducing PhysWord: A Vocabulary for Structuring Latent Reasoning
in Robot Manipulation.
Physical Words that connect actions such as “insert” with predictions of future motion and touch.
• SOTA simulation success: 99.6% on LIBERO-Long with vision only, and 90.3% on UniVTAC with vision + touch.
• 71.7% average success across four real-world tasks, from dexterous manipulation to long horizon tasks including coordinated packing and powered screw driving.
• More than 2× FTP-1’s success rate on extended real-world packing sequences absent from training.
Learned action primitives, reused as tasks grow longer and more complex.
Our longer-term bet: Physical Words can become an interface between agent planning and robot control, connecting language semantics with the continuous dynamics of motion and contact.
Website: nvidia-isaac.github.io/video…
Paper: arxiv.org/abs/2610.26854
Kudos to the amazing collaborators Jiaming Liu, Xinghao Zhu, Yiming Zhang, Shanghang Zhang, Yichu Yang, Yinxi Wang, Zhuoyang Liu, Dantong Niu, @SohaPouya , JUEKUN LI, Yan Chang
#PhysicalAI #Robotics #TactileSensing #NVIDIA
Junhan Sun retweeted
i wanted to experiment with how temperature actually affects token sampling in LLMs, so i wrote a small PyTorch experiment using the exact same logits across different temperatures.
the results were pretty interesting.
at T=0.2, the most likely token had a 99.73% probability. increasing the temperature to 5.0 dropped that to just 15.58%.
the logits never changed, only the temperature.
mathematically, softmax divides each logit by T before exponentiation. lower temperatures amplify the differences between logits, while higher temperatures flatten the distribution.
what's interesting is how drastically this changes sampling diversity without changing the underlying model predictions.
a pretty neat example of how much the decoding strategy matters in language models.
Agree!
The rapid progress in the JEPA field is exciting 🚀.
But when naming becomes a game for eye-catching buzzwords rather than a reflection of scientific essence, the field starts to lose its way.
I firmly believe in the power of community 🤝, but it’s time for JEPAs to slow down, focus on the fundamentals, and tackle the truly essential problems.
