This is fascinating. As it turns out, when you stop making benchmarks public, open models perform significantly worse.
For me, Deepseek 4.1 Flash and GLM 5.3 have been an incredible revelation, these models are sooo good.
But when these labs/people start trying to sell me the idea that we have "Fable at home" and that "China caught up" I just can't help but roll my eyes.
I want there to be Opus 5.5 at home so badly, but the narrative spinning is so unhelpful. We've gotten to a point where we basically ignore almost every benchmark.
Really hope this is a new leaf where we can start to put a little more stock in the results.
Announcing Artificial Analysis Intelligence Index v5, featuring 2 new benchmarks and 3 upgrades. For the first time, the majority of the Index is built on private datasets
The Artificial Analysis Intelligence Index is our composite measure of model capabilities across tasks reflecting real-world usage today. v5 is our biggest upgrade since v4 launched in January. With v5, 60% of the Index is now based on evaluations with private test sets or held-out answers.
Alongside Intelligence Index v5, we’re launching the Artificial Analysis Progress Index v1, which tracks model capability over time on one continuous scale.
New to the Intelligence Index:
➤ FrontierCode 2.0: Cognition's newly-launched private benchmark of real-world coding tasks, graded on whether a repository maintainer would merge the agent's change
➤ Terminal-Bench-Science: 70 agentic scientific research tasks, run in a terminal environment
Upgraded:
➤ HLE-Diamond (replacing Humanity's Last Exam): a refined 1,000-question version of Humanity's Last Exam, split evenly between reasoning and knowledge questions
➤ AA-Omniscience v1.1: An updated dataset with saturated and public set questions removed, new questions added, and ambiguous or erroneous questions revised
➤ CritPt v2: a private, 40-question hard subset of CritPt v2, revised by the CritPt authors and run with a code-execution tool
Staying in the Index: AA-Briefcase, AutomationBench-AA, GDP.pdf and Terminal-Bench 4.0
Removed from the Index: GDPval-AA v2.1, SciCode, and AA-LCR v1.1
Key results:
➤ Claude Opus 5.5 (max) leads at 58, ahead of GPT-6 Astra (max, 57) and GPT-6.1 Sol (max, 55): Anthropic and OpenAI hold the top five models. Gemini 4 Argon (high) is next at 50
➤ The open weights gap widens from 3 points to 12: Kimi K3.1 Pro (weights not yet released) is the top open weights model at 46. Among released weights, DeepSeek V4.1 Flash 1017 (max) and GLM-5.3 (max) lead at 32 - 26 points behind the leader and below Claude Haiku 5.5 (max, 36)
➤ Cost per task varies by more than 13x among the leading models: of the 10 highest-scoring models shown, GPT-6.1 Terra (max) is the cheapest at $0.96 per Intelligence Index task and Claude Fable 5.1 (max) the most expensive at $13.04
Axitek retweeted
⠀⠀⠁⠀⠠⡀⠈⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⣀⡀⣀⡠⠽⠖⠀⠀⢠⡀⠆⠀⠀⠀⠀⢀⠀⠀⠀
⠛⠛⣿⣧⣄⠂⠀⠐⣸⣷⣄⣈⣤⣦⣿⣿⡇⠀⠀
⠙⠛⠛⢿⣿⣿⣿⣿⣿⣽⡟⢻⡿⡤⠡⠀⠀⠀⠀
⣄⠀⠀⠐⠛⠛⠿⣿⣿⣿⣧⣷⣋⡿⠛⠳⠦⣤⣤
⠀⠀⠀⡀⠀⠀⠀⢀⣉⡱⠿⠟⠀⠀⠀⠀⠀⠈⠉
⠀⠀⠀⠀⠀⠠⠀⠠⠘⠉⠀⠀⠀⠀⠀⠀⠀⠀⠀
Ancient desktop PCs from pre-pandemic days should be e-waste, no?
No. You can now run frontier open weight models like DeepSeek-v4.1-Flash at a reasonable speed (35tps decode, 1,200tps prefill) on desktops, not data centers.
Made possible with the Maya-XS model - thanks Project Maya!
github.com/mw00/project-maya
HW 2x3090+128gb
Axitek retweeted
OrcaRouter x Project Maya
Project Maya × @OrcaRouter 🐋
OrcaRouter's OrcaCyber DeepSeek-V4.1-Flash-1017, the abliterated model built for cybersecurity research, now runs on Maya.
Both versions, Mix-256G and Mix-128G, are in Maya's model picker. One command sets it up on your own GPUs: NVIDIA or AMD, Linux or Windows.
Prefer an API? Use it on OrcaRouter: orcarouter.ai/models?tier=cy…
Thanks to the OrcaRouter team for working with us on this.
Faster DeepSeek Flash on 2x3090 is here!
DeepSeek-Flash-1017:Maya-XS is almost identical in eval quality to the larger Maya-S quant, and runs much FASTER on a 2x3090+128gb desktop.
🔥 Decode: 32tps (up from 17tps on Maya-S)
🚀 Prefill: 1,400tps (up from 980 on Maya-S)
Thanks Project Maya!
github.com/mw00/project-maya
We have frontier-level intelligence at home now and I'm not even kidding.
DeepSeek-v4.1-Flash-1017:Maya-S, a 550B MOE model, running on a <$5k desktop PC from 2020.
🚀 Decode: 20tps
🔥 Prefill: 1,000tps
Thank you Project Maya!
github.com/mw00/project-maya
HW 2x3090+128gb+nvme
This is running on the same server I had set up for Qwen3.8-27b months ago
xitter.cf/axitek/status/21142481…
You can run a much SMARTER model on your hardware, with better results across the board.
Qwen3.8-27b vs DSF4.1-1017 (550b) - look at the eval difference between a 27b and a 550b model, running on the same 2x3090+128gb server.
Thanks Project Maya!
github.com/mw00/project-maya
You can run a much SMARTER model on your hardware, with better results across the board.
Qwen3.8-27b vs DSF4.1-1017 (550b) - look at the eval difference between a 27b and a 550b model, running on the same 2x3090+128gb server.
Thanks Project Maya!
github.com/mw00/project-maya
This would make a great Maya-S-uncensored quant wdyt @PeasantSmith
We're open-sourcing our OrcaCyber DeepSeek V4.1 Flash 1017 (Abliterated GGUF) weights — one of the strongest uncensored/abliterated DeepSeek V4.1 weights we've tested, with virtually no capability loss.
- 91.5% fewer refusals
- 92% MMLU vs. 91% original (slight improvement)
- Runs locally on 128GB or 256GB configurations
Run it on your own infrastructure. No cloud required.
Cyber Tier API: orcarouter.ai/models/orca/or…
Huggingface: huggingface.co/orcarouter/De…
It helps to have more VRAM - but don't discount RAM and NVME support.
Same model, DeepSeek-v4.1-Flash-1017:Maya-S on Maya:
🟠 Orange: 2x3090+128gb+nvme
🔵 Blue 7x3090+256gb+nvme
Speed difference is significant, but even 48gb vram is enough to run DSF.
github.com/mw00/project-maya



























