Moritz Schiebold (@m_schiebold) header
Moritz Schiebold (@m_schiebold) profile picture

Building the data stack for Physical AI at https://teapawt.net/t.co/vVh4lJnGvd. I admire people who are building things

Stockholm, Sweden
Joined November 2014
Photo by Moritz Schiebold (@m_schiebold)
Moritz Schiebold retweeted
And so the rerun team offsite begins 🤖
2
8
437
Moritz Schiebold retweeted
I found another cool @NVIDIARobotics paper, but this time to do with slam. They basically took the original deep patch visual odometry paper and added mogev2 depth priors and dinov3 features to it. Means it can generate metric outputs as well as TSDF fused meshes! I ported this to @rerundotio as well as a @Gradio space. Link here - <huggingface.co/spaces/pablov…>
Because I'm using both egocentric and exocentric views I needed a strong ego hand model. Originally I was using Wilor BUT then @NVIDIARobotics dropped the egohands model which performed lots better on my initial testing. So I decided to use this as my "calibration" target when synchronizing egocnetric views with exocentric views. I went ahead and built a @huggingface space with @Gradio so others could try it out themselves. a few things as I built a @rerundotio visualization for this 1. I did a bunch of optimization work. Converting to NVFP4/FP8 mixture helps a ton in making things fast (but only works on blackwell gpus) this made it so I got ~7ms | 140 fps inference time on my 5090 for the full end to end run (decode -> detection network -> perspective croppping -> pose network) 2. I included nvidia's soma-hand instead of mano. Has the nice property of being commercially friendly + allowing choice of levels of detail. Its also technically compatible with mano which makes it even more useful! 3. it also has an okay detection model built on rt-detr, I think a much better one could be trained but I just haven't had the inclination to do it yet. Overall its a better model at least on initial vibes than wilor and produces good outputs. Try out the space and let me know what ya'll think! Next up is, how do we take sapiens2 + egohand and use them to calibrate all of the egocentric + exocentric views? Thanks to @UmarIqb, @jiefengli_jeff, and team for the great model!
5
17
183
6,570
Moritz Schiebold retweeted
Full-stack (from data collection to training) robot learning built on top of Rerun 🔥🔥🔥🔥
Wanted to get into manipulation. Decided to first build a full stack that scales. Plan: copy @GeneralistAI's gripper. Fun weekend project. It was not. So @t_k_233 and I documented every design choice and lesson along the way. Meet Specialist: fully open-source hardware, data pipeline, training, and deployment code. specialist-ai.github.io/
1
1
9
821
Moritz Schiebold retweeted
Really nice usage of Rerun as the storage layer through the whole pipeline
Wanted to get into manipulation. Decided to first build a full stack that scales. Plan: copy @GeneralistAI's gripper. Fun weekend project. It was not. So @t_k_233 and I documented every design choice and lesson along the way. Meet Specialist: fully open-source hardware, data pipeline, training, and deployment code. specialist-ai.github.io/
3
14
876
Moritz Schiebold retweeted
I'll be in SF next week and at CoRL week after. hmu if you're in robotics and want to hang out!
12
6
65
4,382
Moritz Schiebold retweeted
Robotics is solved when world GDP has grown by a hundred orders of magnitude and we've collonized the galaxy, not when SOTA on some benchmark goes from 42.2% to 63.1%
29
56
14
593
32,508
Moritz Schiebold retweeted
Alright so we now have opensource versions of 1. on device multicam egocentric visual inertial odometry 2. multi-iphone recording software (that are timesynced with ntp) 3. Gaussian splatting at acceptable FPS in @rerundotio I decided to take some data I had previously collected that was quite messy and see how far Opus 5.5 could fix it with the rerun mcp and the context of all of the different projects I've worked on so far. I'll break this down into multiple parts (Today just part 1) 1. hardware <- this thread 2. collection 3. time and spatial alignment 4. Post processing 5. Data cleaning
Splatting is going to be a critical part of the ego collection pipeline IMO, so I'm working on improving our current official support. code here - - <github.com/rerun-io/examples…> The current shipped version does well for less than 1M splats, but not so much for larger ones such as the iconic garden scene (5.2M splats). I ported over brushes kernels to @rerundotio and now on my M5 I'm getting ~40-50fps when recording or a solid 60fps when not. This is basically a 10x improvement from the current version and this huge improvement that wouldn't have been possible without all the verification loops we've added with the rerun skills and MCP. I'll break down what was required to make this work ⬇︎
1
15
3
119
8,953
Moritz Schiebold retweeted
Should Europe leave frontier AI to the Americans and the Chinese? The idea has gained ground in Europe. It is a fundamental mistake. Build or surrender: for now, the choice is still ours. Full article below.
270
776
232
3,893
451,536
Moritz Schiebold retweeted
Ultra has been a close partner and customer of ours for a while now and they are just killing it! They are one of the most disciplined deployment focused modern robotics companies out there and an amazing example I hope a lot more companies follow
Robotics has a distribution problem. Today, Ultra is announcing $62M in funding to solve it, including a $50M Series A led by @hiFramework. We believe that deployment is the bottleneck in robotics. It’s relatively easy to make a robot look impressive in a controlled environment. It’s much harder to put a robot into the open world, make it useful, keep it running, earn the trust of the people working alongside it, and do that over and over and over again. So that’s what we’re focusing on.
1
31
1,406
Moritz Schiebold retweeted
Robotics has a distribution problem. Today, Ultra is announcing $62M in funding to solve it, including a $50M Series A led by @hiFramework. We believe that deployment is the bottleneck in robotics. It’s relatively easy to make a robot look impressive in a controlled environment. It’s much harder to put a robot into the open world, make it useful, keep it running, earn the trust of the people working alongside it, and do that over and over and over again. So that’s what we’re focusing on.
103
94
81
722
236,492
Moritz Schiebold retweeted
I wanted to share something I've been working towards since forever. I've finally built out a full slam + hand reconstruction pipeline that works IN REAL TIME at 30fps on a single rockchip 3588. All streaming directly to @rerundotio from a @BitRobotNetwork robocap. This is similar to what a Quest3 or Project Aria 2 provides, but opensource 🙂 I'm so excited I finally don't have to really on closed source software like the meta quest or others. I finally have a realtime performant pipeline that works on an edge computer, but importantly can also scale to a high performance one when needed.
As I'm working on finding more datasets to test my pipeline, I found a really cool one from Meta. This is Show3D and it does ego + exo data capture on a mobile capture rig, its focused on hand/object interaction in the willd. I've so far registered it to the @rerundotio OSS catalog, but I think I found a few issues. The tracking confidence is more unstable then I would expect and the object often has low tracking confidence on large swaths of of the segment. I'll spend more time on it and see if its a me error, or something with the data labels. Also sadly, they only provide the stereo images, no IMU =/
10
22
2
225
17,639
Moritz Schiebold retweeted
How do you scale 2 million hours of ego data without sacrificing quality? Earlier this week, we revealed the world’s largest egocentric dataset. Our frontier pipeline has also turned it into the most labeled, annotated, and human QA’ed robotics ego dataset in the world. We’ve focused on some of the hardest problems in computer vision and ego data to ensure every clip contains SOTA action supervision directly helpful for training dexterous manipulation.
6
16
2
121
10,846
Moritz Schiebold retweeted
It's really cool how easily you can visualize complex datasets nowadays. Thanks to @rerundotio!
2
4
388
Moritz Schiebold retweeted
The LIBERO dataset shows up in many robot-learning benchmarks. Curious what the data behind the benchmark numbers actually look like? We converted LIBERO to RRD, so you can explore the tele-operated demonstrations directly with Rerun. Pick a task and watch the motion in 3D alongside camera views and robot states. Explore the data using our query APIs. Train on it using our dataloader. Data and code 👇
1
6
1
49
3,831
Moritz Schiebold retweeted
Replying to @rerundotio
@rerundotio'ing fruit flies to poke at object policies too much fun!
2
1
6
498
Moritz Schiebold retweeted
Here it is! I rebuilt a visual iniertial multicamera pipeline in rust + python, alll with support for @rerundotio OSS data catalog + visualization. code here - <github.com/rerun-io/examples…> The core is rust to make sure it runs fast, but you can also easily call it from python. I've been unhappy with pretty much all of the other vio pipelines out there, as none of the ones I know about hit the requirements I wanted
As part of my exo ego pipeline, I haven't yet found a VIO/Slam solution that hits all of the features I want. Now that agents make it super easy to write low level systems code, I've almost finished building a portable rust/python slam pipeline that works on both a 5090 AND a Rockchip-class computer. To really make this work, I needed a good ground truth, and the wonderful work from Collabora and @mateodemayo on the monado slam dataset has been indispensable. I registered it as a @rerundotio dataset in the OSS catalog to make it very easy to verify that the GT actually looks right and use it as an eval, and it has worked REALLY well. I'll have more on the VIO pipeline here in the coming days
5
30
2
239
22,387
💯 this! We can only build our way out of this. Complaining or expecting the old guard to change is not a valid strategy at this point. That's why I'm so pumped to see @esra_robotics become a thing.
We can’t complain about Europe lagging in AI while dismissing every effort to catch up. Surprise: no single initiative will fix everything. Progress has to start somewhere. I'm increasingly bothered by people dismissing and joking about every effort as “not enough”. If you do that without contributing yourself or suggesting an improvement, then you're part of the problem. Start supporting the people who are making an effort if you want something to change.
5
138
Moritz Schiebold retweeted
Rerun 0.37 is out. Register a robot mesh once per dataset and share it across segments. The Viewer caches the asset, so switching episodes doesn't mean downloading the same mesh again.
1
4
21
1,682
Moritz Schiebold retweeted
We have been working on an experimental rosbag2 storage plugin. Now you can try use rosbag2 to record directly to RRD, Rerun’s native file format. ros2 bag record -s rrd This way, your recording is both a rosbag and a Rerun recording. You can stream to the viewer while you record, register to a Rerun server when you’re done, and query and train directly from the bagfile. You can even add more layers later, and the original remains a bagfile. This sits alongside our existing MCAP support and log SDK as one more way to get robotics data into Rerun. RRD stores data as indexed, columnar chunks of Apache Arrow record batches. A query reads only the columns and time ranges it needs, and Arrow gives direct paths into tools like DataFusion and out to formats like Parquet. The same columnar access feeds Rerun’s experimental PyTorch dataloader, so you can train a model straight from your registered bags with no export step. This is still experimental, but we are excited to make it available to test. Issues and PRs are welcome. @OpenRoboticsOrg github.com/rerun-io/rosbag2_…
6
26
1,908