creating intelligent robots @microagi | bitter-lesson pilled
SF/LDN
Joined April 2015
- Tweets86
- Following288
- Followers371
- Likes539
One of the most important graphs to look at. Thank you @chooi_jeq!
Too many people still judge models in physical AI by what they are today.
But models have always become more intelligent, faster, and cheaper.
In robotics we have to bet on the same trajectory.
Introducing MicroPlanBench:
One of the most important skills for a robot brain is judgment on what to do next.
Goals often take hundreds of subtasks that build upon each other. Doing the right subtask next is critical for achieving the goal.
We tested SOTA multimodal models on their ability to judge what should be the next task based on prior actions. (All tasks were consequential and not ambiguous)
None of them scored 70% - this means that for a goal spanning 13+ tasks, the success rate is <1%.
We believe this is a critical skill for physical AI models to become more useful in the real world.
One of the most interesting findings is that adding the goal of a task did not make a difference to the success of next task prediction.
We will publish more benchmarks in the future and correlate them to real-world success.
You can read more here: microagi.ai/benchmarks/micro…
We will open-source one half of the evaluation set soon and keep the other half private to prevent it from getting into training sets.
Also, huge shoutout to @angus_harley_, @saliherdemkocak, @ffschumann & @OwenBrande who have been working on this with me.
Please make this happen @PimDeWitte and @elonmusk
Will organize a watch party in London then.
Could do it in all 7 cities we have offices in with some heads-up.
🥨 & ARR
Very proud of what we are building and very thankful for every customer choosing our data or our deployments.
We will keep delivering and raising the bar.
moritz💭 retweeted
Very proud that Odyssey-3 is the most physics-accurate world model in...the world!
Robotics has not internalized the learnings from LLMs.
When GPT-3 came out a lot of scaffolding was built to get single use cases to work.
This scaffolding became irrelevant with the next model release.
I think everyone should bet on models becoming smarter, cheaper, and faster.
It's ironic how this praises coding agents as great end-to-end systems while advocating for robotics to use modular skill graphs instead.
While early robotics benefits from these scaffolds (as early coding tools did), I don't see why the frontier won't become end-to-end.
Still a great fan of the article, everything about coding agents for simulations is super relevant to us as well.
A lot of research rn feels like trying to fix current model shortcomings outside of the models which will become irrelevant as models become better.
For LLMs it was always right to bet on smarter + cheaper models in the future.
In case someone wants to work from any of our cool offices (Munich, SF, Zurich, …), my DMs are open :)
Most labs today don’t really look at their data or just evaluate by looking at samples.
And they don’t have to.
Since they only keep purchasing if data leads to model improvements in experiments, this pushes quality control onto the vendor.
OS doesn’t have this incentive.
We (@PantheonInc) have been working on a robotics data quality pipeline that uncovered a series of major problems in public robotics datasets, especially for world modeling.
To improve the quality of data available to open-source robotics, we're publishing annotations for four of the most popular datasets. Some examples of issues, and our report 🧵
moritz💭 retweeted
Today we release FLUX 3 Action - an open-weight 7B world-action model, achieving SOTA performance and efficiency on various leaderboards such as RoboLab.
I am incredible proud of the team - and both backbone and embodiment-specific finetunes are open weight, alongside the training recipes!
Introducing FLUX 3 Action.
An open weights 7B World Action Model that achieves first place on the RoboLab benchmark.
It outperforms the previous best open model by 6.1 percentage points while using 56% fewer parameters and running up to 3.95x faster.
FLUX 3 Action removes the usual trade-off between world action model performance and VLA speed: it still predicts video and actions together, but plans more than twice as far ahead and runs faster per second of robot motion than the strongest open VLA.
Teams can fine-tune FLUX 3 Action on their own demonstrations to create policies for a particular robot and task. Together with @nvidia, we also integrated FLUX 3 Action natively into @huggingface's LeRobot, with fine-tuning recipes included and edge deployment on NVIDIA Jetson.
Beyond robotics, we’re also seeing promising results training task-specific policies for acting in simulated environments like gaming, controlling a vehicle, computer use, and wherever else a model needs to understand a visual environment and then choose what to do next.
FLUX 3 Action builds on the same image, video, and audio pretraining as FLUX 3, but uses a smaller architecture designed for practical deployment. In midtraining, we trained the model to predict actions and future frames together.
We’re releasing the weights, code, fine-tuning recipe, benchmarks, and reproducible examples so researchers and developers can build on the model with their own robots, environments, and tasks (see below).
This is a good reflection of many of the challenges in human data for robotics.
Coming out of LLM expert data, I really appreciate how much easier operations were: One good labeler could drive 100.000s of revenue.
In robotics, diversity is the most important.
Coordinating 100.000s of people collecting millions of tasks in 10.000s of environments with custom hardware you have to develop and source yourself is actually really hard.
And even after solving that, you need to reinvent the complete computer vision stack to make the data actually valuable.
The only thing I disagree with is the outlook. I believe that data will remain valuable and crucial for much longer - although the formats might chance.
Overall, I‘m very happy with what we as a team have built to collect millions of hours enabling researchers to explore scaling laws.
Really looking forward to deploying ever more robots with these models in the future :)
After 2+ years in the robotics data space, we are shutting @Eidon_AI down.
The thesis was right. But the business is brutally hard.
We close this chapter by open-sourcing everything we built and sharing lessons for anyone venturing into the space.
If you wonder why 20-year-olds run laps around giant corp.
Enterprises are late-stage Versailles: People care much more about status than results
Insanely proud to have this team.
We just beat every existing other SLAM out there.
Today we're announcing microSLAM, a monocular SLAM system built by our computer vision team in Zürich with ETH Zurich's Computer Vision and Geometry Lab. It ranks first on the LaMaria benchmark for monocular SLAM, and holds up against systems that carry many more cameras and IMUs. Ours runs on a single RGB stream.
Robot learning is bottlenecked on data, and the largest untapped source is people going about their work. A camera on someone's head for an afternoon in a plant is a record of how that plant actually operates, including all the parts nobody writes down.
That data is only useful if you can recover the geometry, and geometry from one moving camera in a world that will not sit still is the hard version of the problem. Solving it monocular is what makes the data cheap. Rigs do not scale to every worker. Glasses do.
microSLAM turns that footage into interactive 3D environments where behavioral foundation models can be trained. One person walks a factory floor. A robot learns to navigate it.