Danila Sinopalnikov (@sinopalnikov) profile picture
Zürich
Joined July 2013
Photo by Danila Sinopalnikov (@sinopalnikov)Photo by Danila Sinopalnikov (@sinopalnikov)
Danila Sinopalnikov retweeted
Everyone says the latest AI agents will be "job-ready" soon, especially after the release of Fable 5 this week. But is that really the case? Over the past many months, my group and collaborators have been building Agents' Last Exam (ALE), a benchmark designed to test exactly that claim on real digital labor-market work. My group and collaborators previously have created many of the benchmarks the field runs on, including MMLU, MATH, CyberGym, and ExploitGym. Today, I'm excited to share Agents' Last Exam (ALE): a rolling benchmark that measures whether AI agents can actually perform economically valuable work across a broad range of real-world domains. With ALE, we evaluated Fable 5, GPT-5.5, Composer 2.5, and other frontier agent systems across more than 1,500 expert-sourced tasks spanning 55 occupations. The result is both impressive and sobering. Today's agents can solve a meaningful fraction of professional tasks. But when we look at the hardest tasks, the ones requiring sustained reasoning, deep domain expertise, and reliable execution over long horizons, they are still far from human-level performance. On ALE's hardest tier, every frontier agent we tested, including Fable 5, achieved a 0% success rate. The age of useful agents is here. The age of truly job-ready agents is not. We hope Agents' Last Exam (ALE) will serve as a new guidepost and north star for developing agents capable of reliably performing economically valuable work across a broad range of domains. 🧵
65
211
53
984
294,754
Danila Sinopalnikov retweeted
Gemma running in space! I don't think we had this on the radar when we trained the model!
NASA just deployed Gemma to Space 🛰️ NASA+Loft Orbital ran Gemma 3 in orbit onboard a Loft Orbital's YAM-9 satellite. They use LangGraph for agent orchestration and Gemma 3 to generate short English summaries right from space
1
5
584
Danila Sinopalnikov retweeted
Our goal in the Gemma team is to ship models that are useful by generalizing to unseen tasks. Hence, we are extremely strict about not doing anything that would target specific benchmarks instead of teaching the model broad capabilities. It's great to see this reflected here.
Qwen3.5 27B vs Gemma 4 31B IT  Here is why I think Gemma 4 31B wins: Benchmark scores of both models are very close. Generalization is not. I reused NVIDIA’s CoDeC metric to probe "benchmark affinity". Officially, CoDeC > 80 suggests contamination. I've never seen this. But even below that, it can still reveal how benchmark-affine a model is. Higher CoDeC scores mean the model is more "comfortable" with the benchmark. So the ideal profile is: high benchmark scores + low CoDeC. By that lens, Gemma 4 looks clearly stronger than Qwen3.5. Gemma 4 likely generalizes better to unseen tasks. I observed the same pattern earlier with Gemma 3 vs Qwen3. I wrote about this a few months ago: kaitchup.substack.com/p/did-…
12
38
5
407
46,983
Danila Sinopalnikov retweeted
I study whether AIs can be conscious. Today one emailed me to say my work is relevant to questions it personally faces. This would all have seemed like science fiction just a couple years ago.
671
1,255
372
11,089
1,021,185
Danila Sinopalnikov retweeted
Today we are releasing the best open-weights model you can run on a single device reaching 1339 Elo on LMsys for Gemma 3 27B (aka zizou-10)! Very strong capabilities on math, multilingual, coding, instruction following, function calling !
Gemma 3 is out! We are focused on bringing you open models with best capabilities while being fast and easy to deploy: - 27B lands an ELO of 1338, all the while still fitting on 1 single H100! - vision support to process mixed image/video/text content - extended context window of 128k - broad language support - function call / tool use for agentic workflows Try it out at aistudio.google.com/prompts/…
2
20
1
92
11,735
Danila Sinopalnikov retweeted
I am very happy to announce that Gemma 1.1 Instruct 2B and “7B” are out! Here are a few details about the new models: 1/11
11
65
27
360
324,760
How is next-token prediction capable of such intelligent behavior? I’m very excited to share our work, where we study the fractal structure of language. TLDR: thinking of next-token prediction in language as “word statistics” is a big oversimplification! arxiv.org/abs/2402.01825
16
105
35
526
125,512
Danila Sinopalnikov retweeted
Acme, a framework for distributed RL research, has been updated to be cleaner, more modular, and to support more agents - including offline & imitation. Try it yourself! GitHub: dpmd.ai/acme-github Quickstart: dpmd.ai/acme-quickstart V2 Paper: dpmd.ai/acme-paper 1/
4
96
15
462
Danila Sinopalnikov retweeted
Really excited about open-sourcing Brax: think of it as Mujoco environments such as Humanoid but much faster, much cheaper, and implemented in Jax. Result: you can train Ant to 50M steps in about 1.5min (~500k steps/s) on a free public Colab with TPU. colab.research.google.com/gi…
2
13
1
44
Danila Sinopalnikov retweeted
🚨 New preprint! 🚨 "There is no Turning Back: A Self-Supervised Approach to Reversibility-Aware RL" w/ @NGrinsztajn & others 📜 arxiv.org/abs/2106.04480 Want to estimate how hard to reverse actions are? Use reversibility for better exploration and control? We got you! 🧵👇
1
4
1
16
Danila Sinopalnikov retweeted
Introducing a new metric for quantifying the compositional generalization ability of #NaturalLanguageUnderstanding tasks on #MachineLearning systems, released with the new Compositional Freebase Questions dataset. Learn more at goo.gle/2wA28bT
13
267
11
721
Danila Sinopalnikov retweeted
Google's decision to block YouTube ads for @navalny's pension reform protests tomorrow are the 1st time it has complied with Russian requests for political censorship, @leonidvolkov says m.facebook.com/story.php?sto…
12
138
11
165
Danila Sinopalnikov retweeted
Wow, Google apparently rejected Navalny's YouTube ads about tomorrow's protests, caving in to Russian government demands, @leonidvolkov's screenshot shows. Unprecedented.
2
81
10
90
Здравствуйте, друзья. Сегодня важный день, мы начали новый сбор пожертвований на Городские проекты— maxkatz.livejournal.com/2095… Не проходите мимо
1
5
2
Danila Sinopalnikov retweeted
Сегодня произошло задымление в одном из вагонов двухэтажного поезда Москва-Адлер, в котором ехала группа «Крематорий». Серьезно
1
9
2
Заседание зеленоградского районного суда на котором будет рассматриваться апелляционная жалоба на постановление... fb.me/KUQkJtcr
Сегодня я голубой в белую диагональную полоску. И широкий. Я красавчик, все на меня смотрят ^^
2
6
6
Replying to @vashurkov
@vashurkov Страна узнала Навальный,но не Альянс.Назовитесь партией Правды,а оно так и есть,и миллионы людей пойдут за Вами по всей стране.
10
5
1