@pwk
🇮🇹Account based in
About this account
- Account based in
- Italy
- Connected via
- Italy App Store
Account-level information from X, not a live location or the device used for a specific post.
25Y in startups/VC. CEO AutoBench, eZecute, SBC FoodTech, Agrifood-tech Italia. https://xitter.cf/t.co/K7SAx2ZQx5. Physics. 1/2 Polish, 1/2 S. African, born in Milan, grownup Roman.
Rome, Italy
Joined December 2007
- Tweets15.2K
- Following484
- Followers14.6K
- Likes3.1K
Pinned Tweet
Who is the true LLM King of 𝗔𝗴𝗲𝗻𝘁𝗶𝗰 𝗔𝗜? 👑
Today we announce a major breakthrough in agentic benchmarking: The first 𝗔𝘂𝘁𝗼𝗕𝗲𝗻𝗰𝗵 𝗔𝗴𝗲𝗻𝘁𝗶𝗰 Run is LIVE (with LLM-generated agentic virtual environments).
See how @AnthropicAI’s Claude 4.7, @GoogleCloud’s Gemini 3.1 Pro, @openAi's Gpt-5.4, and open-weight beasts like @Zai_org's GLM-5.1 stack up in real-world enterprise workflows.
We tested 30+ top models across 100s of complex agentic tasks (from parallel calls to adaptive replanning).
No static workflows. No gameable tests. Just pure orchestration under pressure.
👇1/10
I have some good news and bad news for @AnthropicAI and @OpenAI.
Let's start with the good news (on Monday...)
The good news…
Do LLM routers work? I have some good news for @AnthropicAI and @OpenAI: LLM routers suck.
Routers are systems that, given a prompt, choose among a set of LLMs which is the best to answer. The goal is to identify the model that guarantees a high quality response at the lowest cost possible. The problem they promise to solve is simple: why call Gpt Astra on a prompt that a low-cost open source model like @deepseek_ai can answer just as well? A nightmare scenario for the big labs like OpenAi and Anthropic.
Well, we applied the @AutobenchAI benchmarking methodology to 7 major routers. Result: always calling a mid-tier model like GPT-5.6 Luna beats every router we tested on both quality and cost. No exceptions.
Check out the quality-vs-cost Pareto board. The light shaded zone is the bare minimum prize: better than a random pick and cheaper than a random pick.
Luna is already sitting in it. Every router we tested is outside it.
It's even worse. The routers that look cheap just collapsed onto one flash model and landed on the random line. The ones that look strong just bought an expensive model.
1/14 👇
Do LLM routers work? I have some good news for @AnthropicAI and @OpenAI: LLM routers suck.
Routers are systems that, given a prompt, choose among a set of LLMs which is the best to answer. The goal is to identify the model that guarantees a high quality response at the lowest cost possible. The problem they promise to solve is simple: why call Gpt Astra on a prompt that a low-cost open source model like @deepseek_ai can answer just as well? A nightmare scenario for the big labs like OpenAi and Anthropic.
Well, we applied the @AutobenchAI benchmarking methodology to 7 major routers. Result: always calling a mid-tier model like GPT-5.6 Luna beats every router we tested on both quality and cost. No exceptions.
Check out the quality-vs-cost Pareto board. The light shaded zone is the bare minimum prize: better than a random pick and cheaper than a random pick.
Luna is already sitting in it. Every router we tested is outside it.
It's even worse. The routers that look cheap just collapsed onto one flash model and landed on the random line. The ones that look strong just bought an expensive model.
1/14 👇
That is the story on the board. The prize is real. Nobody is collecting it.
That is the good news for frontier-model businesses.
And, yet, there is no good news without bad news.
More on Thursday.
13/14
Who are we? AutoBench is the AI evaluation infrastructure for the agentic era.
We use multi-LLM evaluation (Collective-LLM-as-a-Judge) to measure quality, cost, and speed on live, open-ended prompts (the kind agents actually send).
Interested in what we do?
Reach out here, or on autobench.org, or on our HF community: huggingface.co/AutoBench
14/14 end🧵
<< Per alimentare l'ostilità verso i rifugiati ucraini in Italia, la squadra ha proposto di diffondere una fake su come "le ville sequestrate appartenenti ai russi saranno consegnate ai rifugiati ucraini, con il costo della loro manutenzione proveniente dai bilanci municipali". Inoltre, i blogger italiani dovevano essere pagati per produrre post sul danno causato dalle sanzioni contro la Russia. "La cosa principale è che non capiscono chi l'ha commissionato - in questo momento, pensano che sia un partito politico italiano", ha avvertito un dipendente dell'ASD.>>
È raro ma, ogni tanto, vedrai dei giornalisti dimostrare che la categoria ha ancora una funzione (e una dignità). E questa indagine di The Insider sull’organizzazione moscovita che confeziona e propala le fake russe (parola che usano esplicitamente) è da Pulitzer: nomi, documenti, episodi, ecc.
La domanda, a questo punto, è: chi sono i blogger italiani di cui sopra?
Read about how @JDVance and others fell for a lie about Volodymyr Zelensky concocted by agents of Russia's Presidential Administration. And how the Vice President's falling for it was collected in a file sent directly to the Kremlin. Inside Putin's "Doppelgänger" operation, based on leaked correspondence between the PA and Russia's Agency for Social Design: theins.press/en/inv/298359
France is financially screwed, but it still maintains a massive security asset (nuclear deterrence). Whoever wins the next presidential elections will be handed the same deal: extend France’s security and deterrence apparatus to the rest of Europe in exchange for a feasible path to financial sustainability. This is something even Marine Le Pen understands well (not the Melenchonian left - totally beyond their mental faculties)
Token deflation is the main cause for the trouble now facing @OpenAI and @AnthropicAI. Ironically though, if these two labs slow down or, worse, crash, market capture through consolidation and regulation (@GeminiApp, @xai, etc.), will inevitably push token price up again, opening up again opportunities for arbitrage