A Smooth Guy; Developer of SmoothScroll for macOS, Windows & Google Chrome.
Joined April 2010
- Tweets4.8K
- Following1.6K
- Followers1.2K
- Likes4.4K
Pinned Tweet
Looking further into LLM benchmark x-correlations:
- Top row: how each benchmark relates to human judgement (Arena Elo)
- Other rows: any benchmark pair & their relationship
- On the right: samples = # of models tested for each benchmark
thx: @chipro @maximelabonne @ldjconfirmed
Blaze (Balázs Galambosi) retweeted
Microsoft AI has released MAI-Transcribe-2, achieving 2.0% AA-WER (#2) at ~411x real time (#2) for $1.67 per 1,000 minutes, placing it on the accuracy-speed Pareto frontier at one of the lowest prices among high-accuracy models
MAI-Transcribe-2 is @MicrosoftAI’s latest speech transcription model, ranking 2nd overall on the Artificial Analysis Word Error Rate (AA-WER) leaderboard, behind Alibaba’s Fun-Realtime-ASR-preview (1.7% WER) and ahead of ElevenLabs Scribe v2 (2.2% WER) and Smallest AI Pulse Pro (2.4% WER). It is one of the fastest models benchmarked, processing audio at 411x real time, around 1.5x the speed of Smallest AI Pulse Pro and more than 7x the speed of Scribe v2.
At $1.67 per 1,000 minutes, MAI-Transcribe-2 costs less than half as much as high-accuracy peers including Scribe v2 ($3.67) and Smallest AI Pulse Pro ($4). The new model expands language support from 43 to 60 languages and adds speaker diarization, word-level timestamps, as well as clean and verbatim transcription styles.
See more details below ⬇️
Blaze (Balázs Galambosi) retweeted
Best-in-class PII Detection...running entirely on your device 🤯
perplexity.ai/hub/blog/pii-t…
Blaze (Balázs Galambosi) retweeted
You might have not understand what this means. If this is true, and DS-v4-Pro is open sourced, then gg Opus or even Fable 😅
This means that DS is able to offer Fable level model at 1/50 API price with a much higher cache hit rate compared to Anthropic, thanks to DS v4's ultra small sparse KV Cache.
The total cost might be 1/100 lower while the model capability is pretty much the same.
Bullish for storage as SSD KV Cache will become a standard. Also for AI industry as a whole - making AI affordable meaning more people will start to use AI which in turn increase the demand faster.
Now I'm seriously asking? What're next? Are we ready for a post-Fable-open-source world?
Blaze (Balázs Galambosi) retweeted
Fable quality. We will soon release a new quantized inference framework that will let you run this on a single GPU (B300) with ~50 tok/s in good quality. Local Fable LFG!
Sol/Codex picture emerging: Strong Capability (long horizon coding benchmarks), Strong Continuity (see hidden contex compaction), Weak Control / proportionality (see tweet below Thompson).
pi vs codex server-side encrypted compaction: teapawt.net/kunchenguid/status/207…
1
191
Blaze (Balázs Galambosi) retweeted
Very interesting, this is why you really need to benchmark agents, especially with reasoning models that use variable # of tokens. This is similar to what we saw in some applications with Sonnet (databricks.com/blog/benchmar…) and to arxiv.org/abs/2603.23971. It will surely depend by task though!
GPT-5.6 Sol and Luna are ahead of Terra at every point on the Intelligence vs Cost per Task chart. GPT-5.6 Luna stands out as a particularly cost efficient model
Charting the Artificial Analysis Intelligence Index shows the trade-off between intelligence and Cost per Intelligence Index Task. Across reasoning efforts, each GPT-5.6 model pushes past GPT-5.5 on the Pareto frontier (excluding non-reasoning).
However, Luna and Sol are always ahead of Terra. This means for any Terra effort level, there is a Luna or Sol effort level that is more intelligent at no extra cost, or as intelligent at lower cost.
Blaze (Balázs Galambosi) retweeted
For agentic coding, one can say:
- Unless you need Terra Ultra perf, it's always better to use a Luna model with higher effort setting (same or better performance but cheaper).
- Forget everything below Sol High, use Luna with higher effort settings here
- Forget Sol Extra High, use Terra Ultra here
- The extra cost of Sol Ultra is probably not worth it over Max
Blaze (Balázs Galambosi) retweeted
LFM2.5-230M running at 1400 tokens/s locally with WebGPU 🚀🚀🚀
Blaze (Balázs Galambosi) retweeted
The frontier gap in agentic frontend coding is closing fast.
On Code Arena: Frontend, @Zai_org's GLM series has followed a remarkable trajectory, climbing from GLM-4.6 at 1408 to GLM-5.2 (Max) at 1595 - surpassing Opus 4.8 and closing in on frontier model Claude Fable 5 at 1665.
GLM-5.2 (Max) is the lab's strongest coding model yet, narrowing the gap with frontier labs on real-world HTML/React tasks.
Huge congrats to the @Zai_org team on this open-source breakthrough!
Blaze (Balázs Galambosi) retweeted
Turns out adding 0 helps :)
Today we’re introducing Ternary Bonsai 🌳, a family of end-to-end 1.58-bit language models in 8B, 4B, and 1.7B sizes.
Ternary Bonsai 8B is within 5% of Qwen 3 8B at 9x lower memory.
Still tiny. Noticeably smarter
Today we’re announcing Ternary Bonsai: Top intelligence at 1.58 bits
Using ternary weights {-1, 0, +1}, we built a family of models that are 9x smaller than their 16-bit counterparts while outperforming most models in their respective parameter classes on standard benchmarks.
We’re open-sourcing the models under the Apache 2.0 license in three sizes: 8B (1.75 GB), 4B (0.86 GB), and 1.7B (0.37 GB).
Blaze (Balázs Galambosi) retweeted
In 2023 I made GPQA, and it saturated in about two years. Here, the benchmark I was working on probably saturated while we were making it. People have said this before but it bears repeating: AI capabilities are improving faster than our ability to measure them is.
Blaze (Balázs Galambosi) retweeted
Flexoki 2.0 introduces 88 new colors that feel like watercolor pigments on paper.
This is a continuation of my attempt to bring the feeling of analog color to digital emissive screens.
Flexoki 1.0 only provided the a range of values for the grayscale colors. What I have been working to solve since then is how to expand the palette to a full range of values for every color, without desaturating the pigment effect. I'm very happy with how this turned out.
Flexoki is open source under the MIT license, and already available for most text editors, terminals, and many other apps. Flexoki 2.0 makes it into a more capable color system for UIs and more complex projects.
Blaze (Balázs Galambosi) retweeted
GLM-5.1 by @Zai_org is now #3 in Code Arena - surpassing Gemini 3.1 and GPT-5.4, and now on par with Claude Sonnet 4.6.
The first frontier level open model to break into the top 3. It’s a major +90 point jump over GLM-5, and +100 over Kimi K2.5 Thinking.
Huge congrats to @Zai_org on pushing open model progress forward 🚀
Introducing GLM-5.1: The Next Level of Open Source
- Top-Tier Performance: #1 in open source and #3 globally across SWE-Bench Pro, Terminal-Bench, and NL2Repo.
- Built for Long-Horizon Tasks: Runs autonomously for 8 hours, refining strategies through thousands of iterations.
Blog: z.ai/blog/glm-5.1
Weights: huggingface.co/zai-org/GLM-5…
API: docs.z.ai/guides/llm/glm-5.1
Coding Plan: z.ai/subscribe
Coming to chat.z.ai in the next few days.
Blaze (Balázs Galambosi) retweeted
This figure from the Mythos system card is a very interesting ablation on the reported SWE-bench scores with regards to memorization. They increasingly filter for task instances where the LM solution is dissimilar to the human gold solution and recalculate the scores. 🧵
