Aman Khan

Models by @GoogleDeepMind scale better on cost than their rivals. Gemini 4 Carbon is the prime example. As tasks get more demanding, every rival gets more expensive relative to Carbon. For a typical task, GPT-6 Astra by @OpenAI costs 3.4× as much, Claude Opus 5.5 by @AnthropicAI costs 3.9× as much, and GPT-6.1 Sol is the cheaper option at about half of Carbon’s cost (0.54×). Across the 5% most expensive sessions, Carbon’s position improves against all three. Opus edges up to 4.2×, Astra’s cost roughly doubles relative to Carbon, and Sol’s discount shrinks. This in turn can translate into meaningful savings across real-world tasks such as building an Android security app, one of many long-horizon tasks that our Agent Arena users delegate to their agents.

Come hang out with @GoogleDeepMind , @cognition and @JudgmentLabs at Benchmark in SF If you're not in the mood to climb, we've got a lounge and matcha for you :) luma.com/climbingbenchmark RSVP now, limited spots available!

I promise I’m not just saying this because I work at @Google But Gemini generated analysis and tokens are just more pleasant to read than other LLM-speak (I.e “claudese”)

Bench Summit by Kaggle has officially kicked-off! Trust in benchmarks is eroding. The path forward starts here. We started strong with @janiejgu sharing key insights on the problems with benchmarks today and Kaggle’s mission to build trust in evals and help the world discover what works and doesn’t work in AI. During the session, she shared four ways benchmarks fail today: 1. Relevance: Many public benchmarks don’t resemble what your agents do in production. 2. Saturation: Evals go stale within a release cycle. 3. Verification: Everyone has their own implementation of the same benchmarks, meaning scores vary wildly due to infrastructure, not model quality. 4. Infrastructure: Evals infra is hard to build correctly, and everyone rolls their own. To get numbers you can actually trust, Janie highlighted four essentials: 1. Real workflows as the foundation for benchmarks 2. Continuous benchmarks from crowdsourced tasks, arenas, and production data 3. Rigorous and transparent methodology audited by the community experts 4. Eval infrastructure that everyone can use out of the box Here’s a sneak peak of the early moments:

OpenAI, Anthropic and other leading AI labs face a growing challenge: Better models aren't necessarily attracting more usage. A 10-point intelligence advantage now earns just 20% more developer traffic, down from nearly 69% in late 2025. Full story: thein.fo/4hGJ9jY

I don’t want to manage a agents with different personas. If intelligence is truly super there should be no need for roles. Smart generalists rule the world. If you’re making agents with personas, are you actually finding value from it?

Auditing RL environments and reward models is one of the highest-leverage tasks evaluators can do in service of building aligned models (embedded or otherwise).