Check out the amazing work by @junli_r84995 @YinghuiLi_rob.
NO MoCap! Onboard camera enables portable outdoor performance.
Also presenting the stable version with MoCap, more than 200 rallies.
ArXiv is coming soon.
🚀 Introducing SMASH — the world’s first high-dynamic humanoid robot for outdoor table tennis, fully autonomous with onboard perception.
No motion capture needed—SMASH uses only onboard sensors for stable, full-body human–robot interaction in real-world settings.
Embodied intelligence is stepping out of the lab.
🔗mmlab.hk/Smash/
#HumanoidRobots #TableTennis #RobotLearning #SportsTech #AI #OpenDriveLab #Algorithms
3
224
Humanoid robots have been prisoners of the lab. We set them free — with human data.
We present EgoHumanoid: The first endorsement of human-to-humanoid transfer for whole-body loco-manipulation.
🔗 Home: opendrivelab.com/EgoHumanoid
📑 Arxiv: arxiv.org/abs/2602.10106
🧵👇
ilnehc retweeted
【1/5】❓Why do Vision-Language Navigation (VLN) agents need "step-by-step" language instructions? No.❌ We only want to give a intent. 📍
Meet SparseVideoNav: a new paradigm that uses video generation to navigate like a human. Tell the destination. Imagine the path. Reach the goal. 🌄🤖
Paper: arxiv.org/abs/2602.05827
Page:opendrivelab.com/SparseVideo…
Github:github.com/OpenDriveLab/Spar…
ilnehc retweeted
Glad to share me and my folks work of the last research cycle, good luck and have fun. Upvote on HF is appreciated!
χ0: huggingface.co/papers/2602.0…
χ0, a resource-efficient robotic manipulation framework addresses distributional shifts through model arithmetic, stage-aware advantage estimation, and train-deploy alignment to achieve long-horizon task reliability.
RISE: huggingface.co/papers/2602.1…
RISE, the first work to use a world model as an effective learning environment for challenging real-world manipulation, enabling policy improvement on tasks that demand high dynamics, dexterity, and precision.
EgoHumanoid: huggingface.co/papers/2602.1…
EgoHumanoid, the first endorsement of human-to-humanoid transfer for whole-body loco-manipulation.
SparseVideoNav: huggingface.co/papers/2602.0…
Vision-language navigation systems traditionally require detailed instructions but can be improved by incorporating video generation models with sparse future planning for faster, more efficient real-world deployment.
ilnehc retweeted
🧐Applying world models to improve real-world policy on challenging manipulation tasks used to be considered out of reach.
😌After sustained effort, we’re now seeing encouraging progress.
🚀Thrilled to introduce RISE: Self-Improving Robot Policy with Compositional World Model
opendrivelab.com/kai0-rl/
arxiv.org/abs/2602.11075
RISE is, to our knowledge, the first work to use a world model as an effective learning environment for challenging real-world manipulation, enabling policy improvement on tasks that demand high dynamics, dexterity, and precision.
Incredible teamwork with @lin_kunyang111 @francislee2020 @YueXiangyu @HaoZhao_AIRSUN @smch_1127
ilnehc retweeted
🦾 【MM-Hand 1.0】Open-Source, Lightweight, High-DoF, Multimodal, Modular Design for Easy Disassembly and Modification
🔗 Details: mmlab.hk/research/MM-Hand
🎁 Beta Program: forms.gle/QhaHGCigY6buuSNfA
📮 Inquiries: [email protected]
#MMHand #DexterousHand #Robotics #OpenSource #EmbodiedAI #TactileSensing
ilnehc retweeted
We’re releasing our paper and code, check:
arXiv: arxiv.org/abs/2602.09021
GitHub: github.com/OpenDriveLab/kai0
Community: discord.gg/weUGdEPpsF
Challenge: opendrivelab.com/challenge20…
🧥 Live-stream robotic teamwork that folds clothes. 6 clothes in 3 minutes straight.
χ₀ = 20hrs data + 8 A100s + 3 key insights:
- Mode Consistency: align your distributions
- Model Arithmetic: merge, don't retrain
- Stage Advantage: pivot wisely
🔗 mmlab.hk/research/kai0
Check out our generalized driving world model V3!
TLDR: integrating hazardous data from Sim with large-scale Real data to enable precise action-conditioned simulation, and transferring rewards in Sim to Real!
🤔 How to reliably simulate future driving scenarios under a wide range of ego behaviors, especially for rare and non-expert ones?
😭 Challenge of data shortage: Non-expert data with hazardous actions are scarce and unsafe to gather in the physical world. Without such data in training, the learned world model would struggle to follow hazardous actions.
💡Key insight: We can gather such data flexibly and effectively within a driving simulator! By incorporating simulated, non-expert data in training together with the real-world corpus, the resulting world model can successfully transfer the dynamics of simulated situations into real-world scenarios.
Introducing ReSim, a driving world model that enables Reliable Simulation of diverse open-world driving scenarios under various actions, including hazardous non-expert ones. A Video2Reward model estimates the reward from ReSim’s simulated future. The high-fidelity prediction, accurate action-following, and reward estimation abilities of ReSim can be applied to various driving applications.
Title: ReSim: Reliable World Simulation for Autonomous Driving
arXiv: arxiv.org/abs/2506.09981
Project page: opendrivelab.com/ReSim
Team work by @jiazhi_yang2024, Kashyap Chitta, Shenyuan Gao, Long Chen, Yuqian Shao, Xiaosong Jia, @francislee2020, Andreas Geiger, @YueXiangyu, and @ilnehc
ilnehc retweeted
🌟 Previewing the UniAD 2.0
🚀 A milestone upgrade on the codebase of the #CVPR2023 best paper UniAD.
🔥 Key Upgrades:
✅ Framework Upgrade. Fully migrated to mmdet3d 1.x & PyTorch 2.x (industry-standard stack).
✅ Seamless Workflow. Run our Evaluation Example to replicate state-of-the-art results!
✅ Expanded Datasets. Now supports nuPlan + NAVSIM for richer scenario training.
👉 Check out this branch github.com/OpenDriveLab/UniA…, and we will get you more details soon
ilnehc retweeted
📢DriveLM-Carla update:
🚗 Open Source code of our expert „PDM-lite“ which is the base of the two top performing models on the Carla Leaerboard 2.0.
🗃️ Release of a full driving dataset based on this expert.
💬 Release of VQA labels together with the code to generate own data.
ilnehc retweeted
🎉 Delighted to share that our research on driving world models, Vista, has been accepted at #NeurIPS2024 @NeurIPSConf! Kudos to our incredible team with @francislee2020 @AutoVisionGroup @ilnehc @kashyap7x @jiazhi_yang2024.
Sincere thanks to all reviewers and AC for their constructive feedback.
🌎Vista is a 2.5B generalizable driving world model trained on 1740 hours of real-world driving videos. It features a wide capability spectrum including:
(a) predicting high-fidelity futures (576×1024 pixels in 10Hz) in various scenarios
(b) extending its predictions to continuous and long horizons up to 15s
(c) controlling the agent with multi-modal actions
(d) providing rewards for different actions without accessing ground truth states
🤗We open-released all code, model, and dataset to support future discoveries of the community.
Paper: arxiv.org/abs/2405.17398
Code: github.com/OpenDriveLab/Vist…
Video Demo: vista-demo.github.io/
Dataset: github.com/OpenDriveLab/Driv…
Read more on: medium.com/@opendrivelab/tow…
📹Data: extensively dig YouTube and clean out 1700+ hours driving videos, combined with 300+ hours from public datasets
🤖Model: a large-scale diffusion video prediction model, with multiple application attempts
⏰More: a lightweight and more powerful version in the near future
🚘Introducing GenAD, the first large-scale video prediction model in the autonomous driving discipline. To empower the generalization ability of our model, we acquire over 2000 hours of driving videos from the web, spanning areas all over the world with diverse weather conditions and traffic scenarios. Inheriting the merits from recent latent diffusion models, GenAD handles the challenging dynamics in driving scenes with novel temporal reasoning blocks. We showcase that it can generalize to various unseen driving datasets in a zero-shot manner. GenAD can be adapted into an action-conditioned prediction model or a motion planner, holding great potential for real-world driving applications.
Paper: arxiv.org/abs/2403.09630
Dataset: github.com/OpenDriveLab/Driv…
6
410
ilnehc retweeted
More tracks! More awards! More Fun! We are excited to introduce the Autonomous Grand Challenge at @CVPR #CVPR2024 which has seven tracks and a total cash pool of $120,000. Participants will get the chance to earn over $27,000 for a single track.
Together with @huggingface we are setting up brand new test servers to level up your experience.
Please stay tuned at our challenge website: opendrivelab.com/challenge20…
ilnehc retweeted
🚀We are excited to announce the Autonomous Driving Challenge at CVPR 2024 Workshop. The preview content is on docs.google.com/document/d/1….
Please stay tuned for the official announcement and more details of the Challenge.
As a beta testing user, I would say it is really helpful and amazing! It saves my (every) day by sorting the huge amount of arXiv papers. Every researcher should use this tool. 🫨🫨👍🏻👍🏻
This Post is from an account that no longer exists. Learn more
5
471
🕯️🕯️🕯️RIP Professor. Will never forget your encouragement to me back in July.
Today, I am deeply shocked and saddened by the news that my long-time dear friend, professor Xiaoou Tang of CUHK, advisor to Dr. Kaiming He and my predecessor at MSRA, peacefully passed away at home yesterday. I just had dinner with him last Wednesday in Shanghai! RIP... 🙏🙏🙏
4
181
Nice work! Thanks @bern_jaeger !
This Post is from an account that no longer exists. Learn more
3
138
ilnehc retweeted
A nice survey of end-to-end learning methods for autonomous driving.
This Post is from an account that no longer exists. Learn more
ilnehc retweeted
This is a fantastic, comprehensive and forward-looking survey of academic literature about end-to-end machine learning for autonomous driving. It is a very timely publication as the field is exploding with interest right now.
I'm aligned with the paper's conclusions on open algorithmic challenges. There's loads of insight around opportunities like world modelling, language, foundation models and long-tail robustness. This paper also exposes that academic literature under-appreciates significant industry challenges right now, such as (1) safety, reward modelling and policy alignment against human expectations and risk, or (2) the significance of establishing a synthetic/real-world data engine for training/validation, which are critical to the success of any machine learning system. I'd love to see more work in these areas.
Great to see @AutoVisionGroup @francislee2020, well done!
This Post is from an account that no longer exists. Learn more
ilnehc retweeted
We wrote a survey on End-to-end Autonomous Driving with the latest update and challenges in this field. Check it out! arxiv.org/abs/2306.16927 – at Beijing, People's Republic of China
















