
Babysitting robots with model-ready data at @openmind_agi @fabricFND | prev @PKU1898 @HKUniversity
SF (travel in Asia now)
Joined June 2022
- Tweets6K
- Following4.6K
- Followers7K
- Likes10.2K
Pinned Tweet
Here is a guide to choose robotics data collection hardware.
Robotics data quality starts with the collection stack. It’s not just about camera resolution. What matters is whether the system can reliably produce synchronized, precise, multimodal, and scalable data.
Explaining precision, data type, continuous collection and efficiency below, using Genrobot as an example.
ClaraChengGo retweeted
I’m starting a new early-stage AI fund, doing what I love: partnering with founders from company formation on bold bets, long before consensus.
The fund’s LPs will include Sequoia, Jeff Dean, Sachin Katti, Philip Wong (my Stanford professor and Chief Scientist adviser to TSMC), and others whose work has shaped our industry and how I think about AI’s future.
I’ll remain a partner at Sequoia and keep serving on my boards on Sequoia’s behalf.
Sequoia has been my home for the past 11 years. I joined as the youngest investor the firm had ever hired. I grew up here. Through years of intense debate and shared work, I found lifelong mentors and friends who took me under their wing and taught me this craft. They taught me to aim high and take big swings. I’ll carry that into what I build: outsized ambition, relentless truth-seeking and curiosity, and a refusal to become complacent.
Working alongside founders has made me want to build something of my own: a firm shaped from day one around the convictions I’ve developed over the past decade.
Life has taught me not to take time for granted. This is how I want to spend mine.
Why now
We’re seeing a new way of building companies in today’s AI era. Research and products often advance together. A breakthrough can open a market or accelerate a business far faster than expected. Some companies require extraordinary capital and compute early on. Others reach meaningful scale with remarkably small teams. The rules for how big these businesses can get, and how fast, are being rewritten.
But early on, the technology is unproven, the resources required enormous, and the vision seemingly implausible.
This is what I love most: seeing the potential right from the beginning, and helping founders realize their full scale of ambition.
More than ever, it starts with an extreme willingness to believe early, with bold, independent conviction in a founder’s technical insight before any product or market validation. Then, it means helping founders take the biggest swings, giving the rare, long-term patience to see those bets through uncertainty, ambiguity, and turbulence, and serving as a level head when the stakes are highest.
I learned this kind of partnership at Sequoia. It’s the foundation of the firm I’m building.
Believing before consensus
My fascination with AI’s potential took root at Stanford in the early-to-mid 2010s, studying computer science and working in Andrew Ng’s lab. The friendships I formed connected me to a community whose people and ideas would help shape modern AI and build some of our generation’s defining companies. What stayed with me was the gap between what those people saw was possible and what the wider world could imagine.
I’m drawn to founders pursuing ambitions that sound impossible today.
I start with a simple question: “What if everything goes right?”
Then we work backward together to make it happen.
In 2023, Deepak and Abhinav set out to build a general-purpose robot brain at @SkildAI, before physical AI was a consensus bet. Their sim-to-real breakthroughs gave me conviction where earlier robotics approaches had fallen short. But the data flywheel required deployment at scale. We made commercialization as urgent as research and brought in strategic partners like NVIDIA and SoftBank to support the ambition. Three years in, Skild is advancing the frontier of physical AI with nine figures in annualized revenue.
In 2024, Ioannis and Misha founded @reflection_ai with an insight into scaling reinforcement learning. As that grew into the ambition to build a Western frontier lab for open models, the hard question was how a startup competes with labs spending tens of billions. An introduction to NVIDIA and many compute calls and recruiting events later, the team has built Beam, now one of the leading US open models.
In 2025, Anna and Azalia, the co-creators of AlphaChip, founded @RicursiveAI to design models and hardware together. From day one, we made early customers and ecosystem partners a priority. Today, they are showing order-of-magnitude improvements on real workloads, working with some of the industry’s most consequential players, and reaching a defining inflection point.
The ambition keeps growing as these companies scale.
@linear has always had a kind of magic to it. Now it’s the system a new generation of AI companies builds on, for teams and agents alike.
@tavus saw early that we’d one day talk to AI the way we talk to each other. Today, it’s building the human computing interface for a world of AI colleagues.
@Mach_Industries began as hydrogen experiments in an MIT dorm room. It has since become one of the most exciting next-generation defense companies.
Making the ambition possible
Being a first believer is only the beginning. Compute falls through. A deep-pocketed incumbent makes your mission its top priority. A breakthrough changes the opportunity and the capital required overnight.
These are stomach-churning moments, and early belief carries a responsibility to help make the bet work.
This is the partnership I love: turning the full ambition into concrete steps. Raising capital, securing compute and strategic partners, recruiting exceptional people, and winning the first customers together. Thinking several moves ahead while solving what’s in front of us. And as breakthroughs open new possibilities, helping founders dream even bigger.
Thank you
To the founders I’ve worked with: thank you for trusting me and Sequoia to be your first partner and first believer. It’s been the joy of a lifetime, and I can’t wait to keep building together.
Thank you to Alfred and Pat for your leadership of Sequoia and your support of my next chapter. To Roelof, for bringing me into Sequoia and believing in me from the beginning. Alfred and Roelof, you have been mentors in investing and in life. So much of what I know, and the investor I’ve become, comes from your guidance and generosity over the years. To Bill, for the close partnership and wealth of knowledge, always wrapped in humility. To Shaun, for showing me how to think deeply and imaginatively. To Michael Moritz, for inspiring 22-year-old me to dream big. To Doug, for teaching me to keep things simple. To Bogie and Jess, for bringing joy to the everyday and perspective on what matters. And to all my Sequoia teammates, for the friendship, camaraderie, and lessons I’ll carry with me.
To my family, especially my husband and my brother, who have each started companies of their own: thank you for giving me the courage to take this leap.
Let’s get to work!
ClaraChengGo retweeted
We’ve spent the past year developing new primitives for GPUs. We believe this enables new algorithms to change how machines learn, open new markets, and dramatically reshape AI’s cost structure.
We have raised a $1 billion Series A, co-led by Index and Lux.
This is what a good launch video should be.
"Introducing XXX" +a pure lecture is getting cliche and a demo context, like contact at composing, that maximizes your value proporsition is niche.
For tactile gloves, look at accuracy in joint, accuracy in contact and scalability. S1 from @origami_robotic sets a good example.
I have been trying all types of gloves since I am in Asia these days (FYI this is a TLDR about gloves xitter.cf/ClaraChengGo/status/20…). I think for egocentric headset, hardware design is no long the moat. The moat is workflow design including its app, manufacturing capability thus lower costs and the delivery efficiency.
But The tactile data is still blocked by the glove itself, like sensor precision, adjustment to electronic devices for EMF gloves, drift, and wearability for large scale deployment.
Tactile is early that even the recently launched sharpa 400h dataset is already big, while for ego and UMI it's 10k+ hours of data.
Congrats to @DanielXieee and look forward to more sharings from the team!
Here is prolly why only @dreamlabs_ai can achieve the insanely cheap $2 per hour API cost.
First let's get a glimpe about how cheap.
The cost for Dream-1 is $2-3 per hour. Take the experiement from @se3labs as an example, one task by GPT6 Astra takes about 3min and $3.
So if as Dream-1 blog claims it achieve similar performance as Astra (or even better), then Dream-1 is 60 times cheaper than Astra. Crazy.
Here is where things get interesting.
Consider model companies like OpenAI/Anthropic, one of the largest costs are usually training compute during RnD and inference compute during deployment. So the API cost as the revenue should justify these expanding infra that supports high performance.
But what if the model team gets cheap access to huge amount of compute? I am not talking about a few data centers, but priority to access compute now and in the future.
Yeah don't forget Dream-1 is from Dreamzero, and Dreamzero is from Nvidia, and Nvidia dominates compute.
Please note that good hardware is not just a tool, but an entire workflow.
From connecting devices, deploy and monitor, abnormal alert during capture, to data upload and even purchase of the suite of cable ...... Check here for more of my hands-on lesson learned. xitter.cf/ClaraChengGo/status/21…
Believe it or not, you will get into all types of trouble in your deployment site, so the good hardware that optimizes the workflow and solve troubles for you is super helpful for sustainable data collection.
DM is open about robotics data. Thanks Varun again for the featuring!
Sharpa just launched one of the biggest public tactile dataset. Here are the noteworthy:
> 300h and i believe it's one of the biggest available. FYI the leading dataset is around 10k+ h for UMI and egocentric, but 300h is pretty big for tactile. Also the price for tactile data is 10x than egocentric. Now you know the supply and demand.
> It comes with vision (2 head + 2 wrist camera), joint state, action, joint torque, TCP state and semantic annotation. Torse 7D is 0 to focus on manipulation (not WBC)
> Multi-modal data is aligned to 30fps, although the native sampling rate of tactile is higher. I expect the loss of precision?
> Compared to vision data, tactile is far from standardized, even the format. In this dataset, tactile is captured as tactile deformation video and raw tactile observation (to put it simple, how the tactile points look like when moving or still)
300 hours. 28,993 demonstrations. 59 tasks.
Touch on all 10 fingertips, in every frame.
We're thrilled to share SharpaDex v1.0 with the robotics community! To our knowledge, SharpaDex is the largest publicly available real-robot dexterous-hand manipulation dataset with fingertip tactile sensing, measured by publicly released demonstration hours containing tactile data as of October 2026.
This release brings together:
• Diverse manipulation: tool use, assembly, deformable-object handling, and long-horizon tasks.
• Synchronized sensing: multi-view RGB, fingertip tactile images, fingertip force/torque, and 65-dimensional joint actions.
• Contact-rich data: approximately 195 hours with at least one fingertip in contact, including 55 hours of simultaneous contact across both hands.
• Hierarchical language annotations: task descriptions, 217,642 temporally grounded subtask segments covering 89.3% of frames, and 45 distinct skill labels.
What excites us most is what comes next: researchers using these data to explore how robots can learn from touch, coordinate both hands, and perform increasingly complex tasks.
We hope SharpaDex becomes a useful resource for visual-tactile VLA models, tactile representation learning, and general-purpose dexterous manipulation. We can't wait to see what you build—and learn alongside you.
Available in LeRobot v3.0 and v2.1 formats under CC BY-NC 4.0.
Explore the data, tasks, and annotations through our report and interactive visualizations, or download the dataset to get started:
Dataset on Hugging Face:huggingface.co/datasets/Shar…
Look at the math and timeline.
US has approximately 3.6 million newborns every year. If Elon Musk gave one SpaceX share to every newborn in America, that would mean distributing roughly 3.6 million shares annually. 0.027% of the total supply.
Based on the Mcap now, the total annual value distributed is around $526M.
11:07 AM — @AnthropicAI announced the public benefit fund.
1:16 PM — @elonmusk recommended giving the shares directly to the public.
5:39 PM — Elon said that's what he would be doing.
12:52 AM (next day) — Elon pinned the post saying he would give SpaceX shares to every kid in America.
Only takes about 24h then all kids in America will be born with @SpaceX stock.
Everybody needs you but they just don’t know who you are. Climb to the top of the mountain and tell the world who you are.
With the productivity of AI what we need is not labor but expertise, taste and uniqueness.
Super impressive @_darshi !
Hi, it's Darshita. I am a designer.
And I'm pitching myself in public 'cause I'm done with a year full of anxiety.
Everyone says design is dead and AI killed it. But the real parasite was hiding in plain sight.
So here's my film, DESIGN IS DEAD. She's bold, fearless and very PINK 💖
Hiring? My DMs are open.
Tactile represented by image is a new type of "world state".
Rhoda is Causal Video Model + IDM. But now the Causal Video Model is designed to output tactile signal to be converted to action by IDM.
Dear otaku, remember those nights when you stayed up gaming until you finally beat the game, then went straight to the forums to share your tips, or copy other's tips?
This is how robots are picking up skills now.
Robot execution engine + expanding skill library + evolutionary search.
We are open-sourcing V1.0 of our agentic robotics framework, ASPIRE, with full real robot skill discovery and self improvement capabilities! On the day Astra arrived, our lab switched the backend model and saturated all the benchmarks to ~99% over night.
And the frontier is already on the move:
- Opus 5.5 slightly edges out Astra on our robot simulation bench.
- Sonnet 5.5 hits the Pareto frontier of cost vs. robot capability, $12/task.
We first introduced ASPIRE and /robot skill library in June, way before the big models could reliably do robotics. A rising frontier tide lifts all boats, but Astra was a 0->1 tsunami on many tasks.
Now you can also feel the Physical Bitter Lesson at home like we did on that fateful day ;)
links in thread
Lab researchers never understand why data collection is hard.
If you haven't personally carried egocentric data headset and fly to another country, looking for the whole suite of power bank, usb-typec cable, multiple adapters, drive, HDMI, WLAN, SD card readers, second hand computer/camera/any hardware etc., and talk to the factory owners about deployment, data upload and storage, SOP for workers, settlement of incentives, also take the religious requirement, national holiday, local outfit on their head into consideration, as well as the coordination between factories around the world and the labs in US ...... (i can go forever)
Then you never understand why you can spin up data collection so easily in China, why stereo module gets expensive, why people who are good at playing overcooked may be the good fit for factory management, why you can't get the data every friday, why hardware design is usecase-specific, why only startups can do this, why I am bullish in certain hardware/team but bearish in some...... (i can go forever)
And above all, unlike LLM data, why those who established robotics data as the infrastructure will win in long term.
(The video is an AMSR cuz it’s the lunch break in the factory.)
Follow the First Batch of Robots supported by @physical_int Pi API:
AgRobotics
Actor @ActorLabs
Almond @almond_robotics
Argon Robotics
Chef Robotics @ChefRobotics
Dairy partner
CloudChef
Deft @DeftRobo
DM Dream Machines @dm_robots
Formic @goformic
Foundry @FoundryRobotics
Frontier Machines
General Instinct @gen_instinct
Intuition.dev @intuitioncore
Omakase(CEO @masa_zeals、CTO @NGO275)
robot.com @Robotcom
Ultra @Ultraroboticsco
Weave @weaverobotics
Yondu @yonduai
ClaraChengGo retweeted
The winning robotics data business has a much higher terminal value than investors are underwriting today.
Robotics will be a long race with many twists and turns, but when the ChatGPT moment hits, the winning data player will be in a very strong strategic position to capture token value. They will have large revenue, a strong balance sheet, talent density, deployments experience, and hardware supply chain knowledge. When it comes time to blitzscale they will be amongst the players that can capture the most value.
The reason this is misunderstood today is because most people are using traditional LLM data businesses as a prior for robotics data businesses and what their terminal value could be. I posit this is incorrect framing. Because the model and harness are integrated in LLMs, there is very little room for data players to play meaningfully in the token value flow in the long term.
In robotics this is exactly the opposite, the model and harness (hardware) are separate, the pretraining era is just beginning, there are many more dependencies on distributors/deployers, and there is a large surface area for other players than just models to capture value. Open source will also play a larger role earlier in the robotics race, as all prior LLM learnings will be incorporated as the race is run. This also creates opportunities for long term infrastructure players.
There is a massive amount of terminal value in winning robotics data and participating in the path of the industry from a position of strength.
From first principles:
The endgame of robotics will take place over the next decade. Empirically, today, no player has enough data, enough model intelligence, or sufficient hardware to facilitate a fast revenue takeoff in the field. No player has reasoned through the right sales motion either, which can take years to develop pilots and move to production. Serious players are correctly focused on scaling up pretraining/deep learning vs. who has the best model, deployment, or hardware today.
The robotics data business takes participants on a global hunt to build data, supply chains, and hardware. Along the way it also teaches deployment and deep knowledge of SOTA model development. Winning data participants also get the first crack at the still nascent robotics RL market, which will likely be the next large revenue stream after pretraining. As they compete to win data, data players must develop incredible talent density across both Asia and the US. The winning data business will have both Asian and North American DNA. Canada 🇨🇦 encompasses an interesting cultural mixture of both of these parts of the world. Bullish Canadian Robotics Mafias, more specifically Waterloo/Toronto.
So:
What would constitute a strong strategic position to capture value once the ChatGPT moment of robotics hits? You need a large revenue base, a strong balance sheet, and talent density + experience across data, deployments, hardware, supply chain, and RL. The hyperscalers, the frontier labs, and the winning data player all qualify.
Anyone under time pressure in what will be a long race is in a weaker strategic position in my view. Given the costs of integrated hardware+software+deployment, this would actually be most robotics-native players. They will depend on VC interest, which can be fickle.
When it comes to dense captioning of robotics data, please note that we are talking about Event-based Density, not Time-based Density,
I notice some annotation teams using annotation per as the criteria to measure the caption quality. But it's not ideal.
In this blog, they increase the sampling rate from 4ps to 16fps while the success rate remains the same.
Imo mainly because:
1. More frames don’t necessarily lead to better action understanding. Models still need to understand motion, causality, and the sequence of interactions across frames.
2. More visual input can introduce noise. For example, hand occlusions, redundant frames, and irrelevant movements.
3. The bottleneck isn’t just missing frames. Even when models can see the hands and tools, they may still misinterpret the actions being performed.
(Also, in addition to action captioning, I am also curious about annotation for semantic context, which is particularly important for ICL and failure recovery of agentic robotics. If you know any pls lmk!)
Great breakdown about dense captioning for egocentric videos. Shout out to @tkkong and @MeckaAI team!
Today, we're sharing results from our partnership with @MeckaAI to see if frontier models can label egocentric videos for robot learning.
We had five frontier models caption 200 clips with human grading.
GPT-6 Astra which performed the best got 43.5% of clips fully correct.
Chips next door. Super cool video about how you convert an ordinary wooden backyard shed into a functional semiconductor cleanroom.
DIY setup but with the massive, multi-billion-dollar semiconductor fabs.
Even a single dust particle can ruin a semiconductor chip. So I’m surprised that you could construct cleanroom environment by yourself.
Thank @pronounced_kyle for sharing the video!
Some notes about failure recovery for agentic robotics / ICL.
- Failure sample doesn't mean recovery data. Recovery data may not cover the reason of failure.
- Traditionally the demonstration is teaching the robots from normal to success, but now agentic robotics need more from unexpected to correct.
Failure:
- Most failures are categorizable.
- Typical failure types are gripper 6D-pose, gripper state and task planning. P1 is a good breakdown from ViFailback Dataset.
Recovery:
- Great recovery data should cover task finished or not, key frame and subtask location of the failure, type of task and failure reason.
- Efficiency of training lies in the switch from random failure collection to policy-driven recovery data collection.
- Efficiency of the data collection lies in the switch from expensive teleop/online intervention by human to 1) human recovery data + few robot data for grounding 2) sim data to generate failure specific task recovery.
Take a guess of my sign language. I will send you $555 if you are right.
Dex hand and gloves from @wuji_global .
Money from my wallet.
Again, the idea from @OpenAI about token reset or product update and put it publicly on X is genius.






















