Human behavior contains far more than pixels. Every interaction can be transformed into structured, robot-ready data. At EgoScale, we collect, process, and deliver large-scale egocentric datasets for robotics, VLA models, and embodied AI—including video, annotations, metadata, and robot-ready data assets. Looking for off-the-shelf datasets or custom data collection? 🌐 egoscale.com 📩 [email protected]

A Stargate for Data Labs are on a trajectory towards >$100B/year of data spend by 2030. As we begin the trillion-dollar compute project, we need to think about the equivalent civilizational-scale effort for the other core ingredient: data. At the foundation of the scaling revolution is a simple empirical law: deep neural networks improve smoothly, near magically, as you scale two things in proportion — (1) the size of the model and (2) the amount of data you train on. And despite the scaling laws being brutally diminishing, we’ve successfully bitten the bullet of logarithmic scaling with exponentially larger clusters and datasets, and received incredible new capabilities in return. But this exponential scaling is bound to hit some limits. Oddly enough, compute has compounded fairly smoothly without limit, with trillions flowing into hypercluster buildout. Instead, we’re starting to hit the limits of an exponential demand for data. Gone are the days of being purely in the compute-limited regime, where we had effectively infinite internet data but never enough GPUs, we’re now entering a data-limited regime. Luckily, this limitation is coinciding with staggering improvements in AI capabilities. Incredibly, we seem to have a real line of sight towards automating a majority of knowledge work with the methods we have today. RL + pretraining, and the data for each, will be generally sufficient to achieve most economically valuable tasks, given some minimal algorithmic progress and continued compute scaling. In a data-limited world, economic progress & scientific acceleration will be directly bottlenecked by our coverage in each domain. We need to see data collection as imperative, deserving the same civilizational ambition we’ve given compute. The internet as a one-time subsidy It’s underrated how much all progress in AI owes everything to the blessing of the internet, this one-time civilizational subsidy to deep learning, decades of unintentional accumulation of a perfect dataset: every book, blog post, image, video, paper, discussion, etc. all digitized and freely available. Without the internet, we’d likely see comparably minimal progress in AI today, and in fact, if you notice where systems currently underperform, it’s almost always a domain where web coverage is limited and data is private, expensive, non-digitized, or non-existent. But we’re running out of it. There are only about 300 trillion tokens of useful public human text, and the internet doesn’t produce nearly enough new high-quality data to match what scaling demands — we’re soon to hit the limits of public data for pretraining. And though the advent of RL bought us reprieve — chain-of-thought RL needed a new form of untapped data, gradable math & coding tasks, also available online — we’re quickly running dry of hard tasks for RL as well. Why do we need so much data anyways? Humans learn comparably in far less time, needing just one textbook where language models might need the equivalent of hundreds to learn a new topic. It’s possible we discover methods that are massively more data efficient — synthetic data, data efficient architectures, other exotic algorithms — but fundamental progress is slow and highly unpredictable, and the recipe we have just works today. And, while I’m wary of getting too deep here, even arbitrary data efficiency can’t replace data that just doesn’t exist in the first place. There’s a massive amount of missing information on the web: the dark matter of the internet — tacit knowledge, undocumented processes, etc. — most of which was never published and lives only inside organizations, the physical world, or just in people’s heads. I’ll leave it here and say, for reasons far longer than I can fit in this post [1], it’s best to operate on the assumption that our insatiable desire for data will continue as it has for the last decade. There will be >$100B/year in data spend by 2030 We’re not screwed yet, of course. Only a fraction of useful data in the world is on the public internet, the rest is stored inside private datasets, corporations, personal archives, universities, governments, and otherwise. Labs can and will continue to license these private datasets, or create them from scratch, like Anthropic’s book scanning project. And we’ll increasingly task human experts to manufacture new high-quality data, with a large fraction of hard RL training tasks already being sourced this way. But collecting this data, unlike before, will be expensive. As the free internet dries up and demand for data rises, we should see labs investing equally in data as compute, likely spending a significant fraction of their compute budgets on data. As we see trillions spent on compute, we should also expect hundreds of billions spent on data (human data & collection budgets), given their equivalent importance. And, notably, data spend is already tracking this way: total data spend across vendors, not counting internal lab efforts, is already roughly $7 billion per year. It’s quite reasonable we’ll see >10x by 2030. Data is the moat Data becoming increasingly private will also majorly shift the competitive landscape. While compute is a commodity — everyone buys the same chips and builds the same clusters — data really isn’t. The big reason why frontier models have felt eerily similar to one another, until now, is they were trained on substantially the same internet (pretraining data variability across labs seems pretty low). As labs diverge onto more exclusive, manually collected corpora, I think models will begin to increasingly diverge. OpenAI pulling ahead in mathematics and Anthropic in cybersecurity isn’t an accident. I really think laser-focused collection of high-quality midtraining tokens, custom RL tasks, environments, with dedicated research effort, has driven much of the visible progress in the last year. James Betker has an excellent blog about “the ‘it’ in a model is the dataset”: model architecture and compute buy you efficiency and order-of-magnitude performance, but ultimately, models, of any architecture, are such incredible approximators of their dataset that the core meat of a model boils down to just that, nothing else. Data is a major moat. AGI long, ASI short As I’ve tweeted before, I’m confident that, despite the narrative, the data labeling industry will continue to fuel great businesses and be an excellent AGI long, ASI short. The argument is just: By the time the AGI labs no longer need data, it’s probably over for everything else too [2]. In this frame, the last companies left should be the data companies, as the last speck of economically relevant data is sucked in. And these companies are already among some of the fastest-growing companies in history: Mercor, founded three years ago, is rumored to be doing $2 billion in revenue with something like a few million expert labelers under contract. While these businesses are very non-stationary, what type of data is needed shifts constantly, I don’t think that diminishes their value. The long-tail of the economy is long, and the value isn’t diminishing as you extend farther into more obscure information: as models get more capable, the value of the marginal dataset goes up, not down. Automating a full job means covering its full distribution of tasks, tools, edge-cases, and long-horizon loops. There’s some O-ring logic to it: a dataset that buys a 1% bump can justify a previously unjustifiable collection cost when it’s the difference between a system that does 99% of a job and one that does all of it [3]. The competitive dynamics of the data industry are still evolving but as demand for data is increasingly niche, ultra high-quality, expert-generated, I think we’ll see real consolidation. Again, contra-narrative, we’ll probably see true competitive differentiation built on brand, quality control of data (which, from personal experience, can vary massively), as well as in network effects from the talent networks themselves over time. We’ve already seen rapidly shifting data type demand work in favor of incumbents, benefiting those with early knowledge of where the market is headed. The binding constraint It’s truly remarkable that we seem to have the recipe — pretraining + RL — to absorb most economically valuable work, despite being far from a lot of what we expected from “AGI”. The same way chess engines revealed we never needed general intelligence to solve chess, as we originally thought, we’ll soon realize that software, mathematics, and the vast majority of the economy (including physical, just running ~3 years behind!) are the same. If recursive self-improvement or some other algorithmic breakthrough arrives, that’s wonderful, but we really don’t have to wait for it. The binding constraint between here and an automated economy isn’t that, it’s data coverage: every app, workflow, edge case, process, etc. sitting in private stores or someone’s head. Ultimately, while we make tremendous strides in more efficient model architectures, and clusters like Stargate equip us with zettaflop-scale compute, we really aren’t making rapid progress collecting the data we lack. We’ll soon live in a world where we have the methods & compute to accelerate scientific progress or economic growth, but not the data. And we’re already there today: frontier models would surely be as good at accounting/many medical tasks/legal advice as they are at software engineering if we only had the same pretraining & RL coverage as we did for code. I really want to drill this in: The speed at which we automate the economy is going to be directly rate-limited by our ability to collect data about it. Worth noting that under this assumption, with data as defensible and directly proportional to economic & scientific progress, data should also be considered a national strategic asset like compute. Imagine what we’d do in a world where we had a Manhattan Project-effort for AI and needed to mobilize data collection as a limiting factor. We should be concerned about China, with greater state capacity and authoritarian economic control, being capable of mobilizing data collection at national scale, potentially compounding their economy and scientific output faster than us down the line. A Stargate for data I’m leaving my complete ideas for a future post, as this one is already far too long, so I’d really like to pose the question here. Stargate exists because we organized trillions of dollars, international strategy, gigawatts around compute as a fundamental ingredient. What would equivalent ambition look like for data? Obviously, scaling data collection, a heterogeneous mass of information across the economy, isn’t going to be as clear as scaling compute, as a homogenous infrastructural effort. A core division will be first, coverage — all uncaptured knowledge sitting across the economy/science/physical world and all that simply isn’t recorded — and, secondly, sheer volume in the domains we already train on: more hard math tasks, more high-quality web text, way more coding data, more legal drafts, etc. I have a post coming soon which breaks down my proposals. There’s a lot of room for creativity. Quickly, we’ll probably want to start with a deep census of what we have and what we’re missing, predict what the 2030 model will still be bad at and work backward to what we should be collecting today. You can probably license a large amount, leveraging high lab valuations to buy datasets or companies altogether. There’s an adversarial nature to a lot of this collection with firms, so there’s lots of engineering to do this correctly. We should go convince important companies to turn off deletion policies, even if we’re not buying from them yet. Data flywheels in consumer products will be massive. Confidential training, government legislation for grant-funded research, running companies at a loss for their data, etc. We’re headed towards hundreds of billions in expenditure, national prioritization, and major data limitation on the horizon. We have a great opportunity to think creatively about what a megaproject for data would look like: How do we, deliberately this time, construct the next internet’s worth of data? Footnotes: [1]: I’ll probably soon publish my much longer post explaining my position on data efficiency and why the value of this data is still pretty high in most worlds regardless of new algorithms. [2]: The “AGI freeroll” bet: heads you win, tails ASI flips the world upside down anyways. [3]: We already see a glint of validation of this point, given the data market is strongly tilting towards ultra-high-quality agentic data, rather than unskilled labeling — niche expert workflows, live environments, and evaluations requiring increasingly obscure talent & knowledge — yet shows increasing, not decreasing, revenues.

Are We Picking Up Laundry Yet? There's something you should know about robot demos: whatever you see the robot doing in the demo is the absolute best thing it can do. That's true of what I've shown you previously as well, but I’d like you to see what I see. This video is a continuous eval of a new model I’m working on. In order for the Stringman to be a useful home appliance that actually saves you time, it needs to be able to pick up toys in a messy playroom and put them in some kind of container. So far, what I have found is that I can train DiT models to grasp things if my training episodes consist purely of grasping things. I have tried episodes that consist of traveling to an object, picking it up, and taking it to its destination, but I found that the DiT model in LeRobot was mostly unsuccessful at learning cross-room traversals from the data I have collected. So, for the cross-room traversal in the demo I showed, I relied on another model and some fairly straightforward point-to-point motion. That other model is what I've been calling the "target model" because it picks out targets on the floor. It requires its own set of training data, though it could possibly be swapped out for off-the-shelf solutions like YOLO. For the past few weeks, I've been trying to answer the question of whether a more advanced DiT model can learn both target identification and cross-room traversal behavior. I believed the amount of training data was a limiting factor, so I have collected 588 episodes that consist of locating an object, picking it up, and taking it to its destination. In that dataset, I stuck to a small set of prompts, such as "put laundry in the hamper," "put toys in the toy box," "put a ball in the ball pit," or "put trash in the trash can." The other limiting factor was that the DiT model in LeRobot ignores all but the CLS token from the CLIP vision encoder. To improve the vision encoding and hopefully produce an embedding that contains more spatial data, I switched to the DINOv3 encoder and added a few layers of cross-attention that consume the spatial patches from DINOv3. I retrained a model on this, and while it showed some promising results, it is still mostly useless, as you can see in the attached video. For clothes, the model did learn to seek out and pick up certain articles of clothing, although it almost always goes for the same pair of gray pants. It did not learn to take the gray pants to the laundry hamper. It prefers to drop them in some random place in the room and then pick them up again. Conversely, for toys, the model cannot seem to locate them. If placed directly over a toy, practically with the toy between its fingers, it will grasp it, pick it up, and carry it off to its destination. Unfortunately, it learned to overfit to the particular bin locations in my playroom. So, in both cases, the model learned basic grasping and cross-room traversals, but never all of it in one usable package on any particular type of data. Does this make it a useful cleaning robot? No, not yet. If hybridized with the point-to-point motion and the target-finding model, it is somewhat useful on laundry, though the target model is weak and needs a human to add or remove a target far too often. There are currently six total deployed Stringman robots, scaling up to nine in the next week. Two owners have agreed to contribute data, so hopefully, if data volume is the limiting factor, we will slowly overcome it. Large datasets of bimanual manipulation are available, but it’s not clear how they could be leveraged to improve the behavior of an overhead cable robot. I have tried fine tuning VLAs that have been pretrained on bimanual manipulation and it seems they pretty much immediately forget 99% of their training. In the meantime, I'll continue to rely on the hybridized behavior and investigate other techniques, such as JEPA, RL, and creating simulated training data using Gaussian splat-based images. Running a Mujoco simulation of Stringman is possible, though it doesn’t have a primitive for strings, so you’d have to either use chains or use the overlapping bowls trick. There isn’t a strong technical reason it hasn’t been done yet, it’s just a time and money constraint.

Today we’re introducing KinetIQ Ascend: our reinforcement learning approach designed to reach 99.9% manipulation reliability at human speed and beyond. It enables our robots to learn from real production tasks and improve through trial and error: • Machine feeding: throughput +42% • Item handover: throughput +85%, success from 80% to 98% • Tote handling: throughput more than doubled, success from 78% to 99% More in the blog post below 👇

Excited to share T-Rex: Tactile-Reactive Dexterous Manipulation 🦖🤖 Touch is fundamental to human dexterity, yet most Vision-Language-Action (VLA) models either ignore tactile feedback or lack the ability to react to high-frequency contact signals. In this work, we tackle both the data and architectural challenges of tactile-reactive dexterous manipulation. 🦖 A 100-hour tactile-synchronized dexterous manipulation dataset with 7,700+ trajectories, 22 motor primitives, and 200+ everyday objects. 🦖 A tactile-reactive MoT architecture with spatial-temporal tactile encoding and asynchronous high-frequency tactile refinement. 🦖 A scalable training recipe combining 22,889 hours of human egocentric pretraining with tactile-grounded robot mid-training. Across 12 real-world contact-rich manipulation tasks, T-Rex achieves over 30% higher average success rate than the strongest baseline. We are fully open-sourcing the dataset, models, teleoperation stack, training code, and inference pipeline. 🌐 Project: tactile-rex.github.io/ 📄 Paper: arxiv.org/abs/2606.17055 💻 Code: github.com/ZhuoyangLiu2005/T… 🤗 Dataset: huggingface.co/datasets/zeka… 🧵 Thread ↓

A few examples of π0.7 in action Turn the shirt right-side out and fold it:

Jensen Huang is a titan and a teacher. He recently sat down with me to explain his vision of the future of technology and humanity. He’s calm, clear, and very funny. We touched on many topics, ranging from the $20T or more AI economy in five layers to changes in labor for the AI age. Here are some of my main takeaways: 1) The world is moving from retrieval to generation 2) Generation offers intelligence customized to the individual 3) Nvidia is making the "generators" of intelligence 4) We've seen this kind of revolution at least three times before with energy (Generators), Telecommunications (vacuum tubes / transistors?), and now intelligence (GPUs) 5) There is a five-layer cake of participation in this many-many trillion dollar revolution: Energy, Chips, Infra, Models, and Applications. 6) There are many ways to participate in this revolution, and everyone has a role 7) We'll be pushed to dream up new problems to solve with this unprecedented intelligence 8) In this new future, it's not just having the answer, it's having the right questions 8) The right questions will drive us toward our individual and collective human purpose 9) We move from the carpenters to the architects I believe this is the realistic future. Thanks to Jensen and the entire @nvidia team for the conversation and for letting us share! 00:00 Introduction 00:42 From Chatbots to Generative AI 03:35 Agentic AI That Does Work 05:26 Downstream Industry Impact 06:25 Computing Shifts From Retrieval to Generation 11:26 A Planet Cocooned by Intelligence 14:27 Inside the NVIDIA AI Factory 20:48 AI Five Layer Cake 21:58 Beyond Chatbots to Biology 23:54 Tokens and World Models 24:53 Trillions in Applications 27:13 Ditch the AI Doom 31:32 Jobs Tasks vs Purpose 38:40 Closing the Tech Divide

Jensen Huang is a titan and a teacher. He recently sat down with me to explain his vision of the future of technology and humanity. He’s calm, clear, and very funny. We touched on many topics, ranging from the $20T or more AI economy in five layers to changes in labor for the AI age. Here are some of my main takeaways: 1) The world is moving from retrieval to generation 2) Generation offers intelligence customized to the individual 3) Nvidia is making the "generators" of intelligence 4) We've seen this kind of revolution at least three times before with energy (Generators), Telecommunications (vacuum tubes / transistors?), and now intelligence (GPUs) 5) There is a five-layer cake of participation in this many-many trillion dollar revolution: Energy, Chips, Infra, Models, and Applications. 6) There are many ways to participate in this revolution, and everyone has a role 7) We'll be pushed to dream up new problems to solve with this unprecedented intelligence 8) In this new future, it's not just having the answer, it's having the right questions 8) The right questions will drive us toward our individual and collective human purpose 9) We move from the carpenters to the architects I believe this is the realistic future. Thanks to Jensen and the entire @nvidia team for the conversation and for letting us share! 00:00 Introduction 00:42 From Chatbots to Generative AI 03:35 Agentic AI That Does Work 05:26 Downstream Industry Impact 06:25 Computing Shifts From Retrieval to Generation 11:26 A Planet Cocooned by Intelligence 14:27 Inside the NVIDIA AI Factory 20:48 AI Five Layer Cake 21:58 Beyond Chatbots to Biology 23:54 Tokens and World Models 24:53 Trillions in Applications 27:13 Ditch the AI Doom 31:32 Jobs Tasks vs Purpose 38:40 Closing the Tech Divide

Today, we enable AutoResearch in the physical world for the first time! Introducing ENPIRE: we give 8 Codex agents a fleet of robots, an allocation of GPUs, and generous token budget. We set them free with a simple goal: solve the task as quickly as possible, keep the robots busy but stay safe, don't waste precious compute. Make no mistake. Then humans step aside and our watch begins. The robot fleet starts to come alive: they learn to look for visual clues, reset the scene, practice novel skills, tinker with control stack, read papers online, debate, reflect, get stuck, and try again directly on the hardware. All we did is to give Codex an API to the world of atoms, and the rest is emergence. ENPIRE is able to solve high-precision tasks like tying zip-ties, organizing fine pins, and installing GPUs all by itself. We also discovered a new type of "physical scaling": 8 robots exploring in parallel improves significantly faster than fewer ones. A part of our NVIDIA GEAR lab now self-improves tirelessly over night. We just read the reports in the morning. /goal: we all take a holiday and Jensen wouldn't even notice ;) We will be open-sourcing everything, so you can host your self-running robot lab at home too! Deep dive in the thread: