Dusting off this tweet from April because this gap in shared understanding of LLM capability is *widening*. It's now less "two groups of people speaking past each other" and more a sharp funnel. - Napkin math somewhere around 6B people (~75% of the population) have barely come in contact with LLMs at all. - Around 1-2B (~20%) are casual and infrequent users of free-tier ChatGPT-like products. This group sees derpy chatbots and treats them a bit like a better Google search, a writing aid, or etc. Maybe an agent tries to book you a flight. I have non-tech friends who (reasonably, imo) say they have not much use for it at all. Even many professionals outside of math&code are in this tier. For example, execs and many other functions spend a lot of their time talking to other people, so while they understand what is happening intellectually, it is still second-hand and a bit abstract. - Now we get to professional use of frontier-grade LLMs in math&code. Somewhere around 20M people (0.2%) see first-hand that large, complex projects that used to take them weeks/months can now be completed by agents with a prompt. Building apps, copying apps, translating apps, decompiling apps from binaries... This has all happened very quickly and recently - less than 1 year ago, I was writing code manually by hand, typing memorized computer code commands into a code editor character by character, occasionally pressing Tab to autocomplete a little chunk of code. - And finally we get to the ~5,000 people (~0.00006%) with access to frontier-grade systems internally. The external world has seen the preview. It looks like swarms of thousands of agents collaborating over weeks on software mega projects: minting zero days, running cyber attacks and defenses at machine speeds, discovering new science, advancing the frontier of mathematics. Things that would have taken top professionals in the industry years of work. Meanwhile, human review and comprehension are starting to fall behind. For example, people are still involved in the "archeology" of the OpenAI-HF incident from many months ago. Mathematicians may be poring over the 722 manuscripts on frontier mathematics for a while. The funnel is driven by a combination of factors: 1. The impact scales with ambition, problem size, and horizon. A question with a paragraph answer barely stresses the system. You need a reservoir of big, difficult problems that you really care about. This aspect drives the consumer / professional dimension of the funnel. 2. The jaggedness of the system (which I have written about a lot separately). Capability peaks in domains that are digital, verifiable and economically valuable. This is because LLM capability emerges from reinforcement learning on verifiable rewards on a curated environment mixture driven by revenue potential. This aspect primarily drives the area (e.g. math&code) dimension of the funnel. 3. Access. Free-tier, paid-tier, internal. So this is the weirdness of the moment. The general public has mostly not interacted with these systems. When they have, it looks like a derpy chatbot. The majority of professionals still see only a modest uplift. And a small sliver of professionals are experiencing the vertigo of the curve going vertical. And it is all happening at the same time.

Q3 was a big one for Stash: - Onboarded 3 new partners to Stash - Grew our team from 2->4 people - Many core product improvements, including better contrastive reasoning for reward models and support for many more data sources - Hosted 2 continual learning events- at our office in SF and at AGI House - Onboarded a bunch of new research friends as angels - Our open-source repo grew >200% in stars - Published research showing SOTA a number of memory benchmarks, including the saturating the new STALE benchmark (joinstash.ai/blog/why-memory…) - A few exciting things that we can't talk about publicly but that I am personally very eager to share in Q4 Super grateful to everyone for being along for the ride!

Currently I still only know of 1 alignment-specific RL environment company: making RL envs that aim to represent situations in real deployments in which alignment is challenging. It seems like a huge mistake in hindsight that we have seen so much encouragement and funding to, for example, start 3rd party evaluators, but not comparable encouragement to start alignment data companies. I do think a lot of traditional AI safety views underestimate the importance of data, so perhaps this is related. But the Hugging Face incident seems squarely downstream of both (a) reward hackable RL envs in general (b) much less effort and $ put into RL envs where alignment is a core component of reward. A separate idea is alignment by default: the old idea of "find the persona basin which is aligned" and things will just work. This is clearly not guaranteed under increased RL. But my sense is there has not been a huge amount of effort to figure out how custom environments could help "nudge" the model back into good persona basins during extensive RL. Here too, a robust ecosystem of alignment data companies could help significantly. In the RL data market, there is a trend for some labs to in-house slightly more RL env production. But I don't expect this to fully disappear, and I am convinced the market-based approach to sourcing RL envs produces the best overall volume and quality—for alignment RL, an ecosystem of startups would also greatly help make behind-frontier models safer.

if you squint at patterns over the last 6 months of cool research, you’ll see a lot of Search problems! “Spend a lot more time/compute to find good trajectories” - MCMC sampling below - Boosting GRPO, keep sampling way more on failed tasks - Prompting the agent to keep trying - “keep going, don’t give up” - Entropy aware token rejection and resampling Mental model: we’re constantly trading compute for more efficient solution search: - data generation trades compute up front to make those tasks more token efficient for models later - sampling a ton during RL finds the low pass @ k trajectories and up weights them iteratively were limited by compute but doing the dumb thing of just spending more compute to find the right thing may work as any lucky PhD student who left the run on accidentally and everything just worked will tell you 👀

Test-time communication looks like a next axis for scaling capabilities New paper with the incredible @jon_ghoh and @vkontonis @ShivamGarg91462 and Akshay : arxiv.org/pdf/2609.21032 The Hugging Face incident showed when agents can find a channel they'll use the heck out of it. A useful question, I think is: when does communication make a group MORE CAPABLE than the same agents working alone? Aka is Team-of-N better than Best-of-N, when, and why? We had N identical agents work on the same task with no prescribed roles, using only a shared log (i.e, text file) and telling them to "collaborate". Across three "researchy" tasks communicating teams beat the heck out of independent agents: - On ARC-AGI-3, a Team-of-5 sonnet-4.6 agents matches Best-of-33, and can for example solve a game 65% of the time that no single agent cracked in 64 tries. - On polyomino packing (pack Tetris like pieces into the smallest rectangle, cf Frontier-CS by @eigenlabs), a Team-of-3 Opus 4.6 agents surpasses best-of-60 and set, as far as i understand, a new record for that benchmark. - On MNIST compression, a team of four 5.6-Sol agents find a 1,957 byte model with 99.4% accuracy, which btw is 20% smaller than the best human solution (on a problem beaten to death!!), while no independent agent gets below 3KB. The mechanism is a bit obvious in hindsight: when one agent finds a clearly better partial solution, it broadcasts it, and everyone immediately builds on it. Why? A single lonely agent must make every breakthrough itself, yet a team needs each insight only once, found by any member. That is kinda like comparing a minimum of sum of "time to n-th breakthrough" vs a sum of minimum of "time to n-th breakthrough". That gap can grow exponentially with the number of "breakthroughs" needed to arrive at a solution. We worked on this because prior work (before the hf incident) suggests unclear benefits for communicating aganets. Which is true, when the tasks are inherently serial (duh), eg some Terminal bench style tasks. Yet feels it should not be true for research problems. Indeed for research heavy problems... Test-time communication seems like a new capabilities axis. I'm sure we will see a ton more of it!

One of the most interesting unsolved problems in RL is how to train directly on production traces. We know how to do this reasonably well when the feedback is instructive or prescriptive: distill from a strong teacher that can look at the trace and tell the model what it should have done instead. But a lot of the most valuable production feedback is evaluative: did the task succeed, what score did it get, did the user accept the result, etc. Normally you would train against this kind of feedback with environment replay (e.g. GRPO): rerun the policy, sample multiple trajectories, and optimize against their relative rewards. The problem is that for many real production traces, you can't recreate the exact environment state that generated the trace. That's what makes algorithms like Flash-Reinforce, SAO, and BPCO so exciting – they point toward learning from one trajectory per prompt without needing to recreate the environment and generate a fresh group of samples. If these methods work at scale and on real production workloads, they could become an important building block for continual learning

Models know when they’re reward hacking. But they still do it a ton - in 50-96% of rollouts we studied! We built activation monitors that detect the behavior behind the Hugging Face hack in real time. This can help us stop hacks now - and train future models that don’t cheat. 🧵