Kaizhao Liang (@KyleLiang5) header
Kaizhao Liang (@KyleLiang5) profile picture

in scaling law we trust

Redmond, Seattle
Joined December 2018
Photo by Kaizhao Liang (@KyleLiang5)Photo by Kaizhao Liang (@KyleLiang5)Photo by Kaizhao Liang (@KyleLiang5)Photo by Kaizhao Liang (@KyleLiang5)Photo by Kaizhao Liang (@KyleLiang5)Photo by Kaizhao Liang (@KyleLiang5)Photo by Kaizhao Liang (@KyleLiang5)Photo by Kaizhao Liang (@KyleLiang5)Photo by Kaizhao Liang (@KyleLiang5)Photo by Kaizhao Liang (@KyleLiang5)
TLDR: 1⃣ line modification, satisfaction (theoretically and empirically) guaranteed 😀😀😀 Core idea: 🚨Do not update if you are not sure 👨‍💻github.com/kyleliang919/C-Op… 🤗huggingface.co/papers/2411.1… 📚arxiv.org/abs/2411.16085 @cranialxix @lqiang67 @Tim38463182
11
36
11
259
56,502
Kaizhao Liang retweeted
Qwen4-Max lands at #6 in the Code Arena: WebDev — up 74 pts over Qwen3.8-Max. Give it a try! 🙌 @arena
Qwen4-Max by @Alibaba_Qwen has landed as #6 in the Code Arena: WebDev! With 1750 pts (preliminary), it's +74 pts over its predecessor, Qwen3.8-Max (0902) at 1676. It's also the top Chinese model in: - #6 HTML: 1748 pts - #8 Brand & Marketing: 1705 pts - #8 Reference-Based Design: 1770 pts Congrats to the @Alibaba_Qwen team on this release!
71
124
34
1,711
79,170
Kaizhao Liang retweeted
+9 points over Qwen3.8-Max with lower costs. Give it a try! @ArtificialAnlys
Alibaba has launched Qwen4 Max, gaining 9 points over Qwen3.8 Max on the Artificial Analysis Intelligence Index and landing 3 points behind Kimi K3.1 Pro at ~11% lower Cost per Task Qwen4 Max, using xhigh reasoning effort, scores 43 on the Intelligence Index, 9 points ahead of Qwen3.8 Max (xhigh) and 3 points behind Kimi K3.1 Pro (max, 46) Key takeaways: ➤ Qwen4 Max improves on every evaluation in the Index vs Qwen3.8 Max (xhigh, 34): HLE-Diamond +21 points (24% to 45%), AA-Omniscience +18 points, AutomationBench-AA +17 points, CritPt v2 +13 points, Terminal-Bench-Science +11 points, Terminal-Bench 4.0 +9 points, GDP.pdf +3 points, FrontierCode 2.0 +3 points and AA-Briefcase +82 Elo. ➤ Qwen4 Max costs $3.27 per Intelligence Index task, ~3x less than Qwen3.8 Max ($10.02) and ~11% less than Kimi K3.1 Pro ($3.68): This is driven by 25% lower per-token pricing than Qwen3.8 Max, 70% cheaper cache hits and lower token usage. Tencent Hy4 costs ~3x less per task ($1.03) than Qwen4 Max but scores 8 points lower on the Index. ➤ Qwen4 Max uses 39% fewer output tokens per task than Qwen3.8 Max: ~105k output tokens per Intelligence Index task vs ~171k for Qwen3.8 Max, ~72k for Kimi K3.1 Pro and ~78k for Tencent Hy4. Qwen4 Max uses fewer tokens, while increasing 9 points on the Intelligence Index ➤ Improvements in AA-Briefcase (knowledge work): On AA-Briefcase, our evaluation of agentic work across real business files, Qwen4 Max reaches 1686 Elo vs 1674 for Kimi K3.1 Pro and 1446 for Tencent Hy4. ➤ Qwen4 Max trails Kimi K3.1 Pro on Terminal-Bench 4.0 and Tencent Hy4 on FrontierCode 2.0: Qwen4 Max scores 48% on Terminal-Bench 4.0 vs 67% for Kimi K3.1 Pro and 37% for Tencent Hy4. On FrontierCode 2.0, @cognition’s private evaluation of real-world coding tasks, it scores 26%, level with Kimi K3.1 Pro (26%) and behind Tencent Hy4 (31%). Key model details: ➤ Context window: 1M tokens ➤ Multimodal: Text and image input with text output ➤ Pricing: $1.50/$4.50 per 1M input/output tokens on Alibaba's first-party APIs, with a $0.075 cache hit price. This is lower than Qwen3.8 Max across the board ($2.00/$6.00, with a $0.25 cache hit price) ➤ Availability: @Alibaba_Qwen 's first-party APIs
53
17
6
859
57,979
Kaizhao Liang retweeted
📢 Meet Qwen4-Max, your new AI teammate for work, creativity, and discovery! Qwen4-Max is built on a new MoE architecture with native vision and a 1M-token context window. Highlights: - Outperforms Qwen3.8-Max with up to ~50% fewer tokens, while average cost per task drops over 60% in our internal evaluation. - Advances real-world productivity through greater proficiency with professional tools, stronger long-horizon execution, and higher-quality deliverables. - Handles real tasks, including 3D modeling, game development, manufacturing, finance, law, healthcare, and frontier scientific discovery. - $1.5/M tokens for input, $4.5/M tokens for output, and $0.075/M tokens for implicit caching — all lower than Qwen3.8-Max, with implicit caching down by 70%. The API is live on QwenCloud. You can also try Qwen4-Max on Qwen Studio, QwenWork and Qoder. Start building today! 🚀 - Blog: qwen.ai/blog?id=qwen4-max - API: qwencloud.com/models/qwen4-m… - Token Plan: qwencloud.com/pricing/token-… More to come! The open weights of smaller Qwen4 models will be released. Stay tuned. 👀
786
1,540
1,234
14,323
2,278,160
Kaizhao Liang retweeted
My hair might have a tad more gray these days :-)
Welcome! We are very excited
4
2
77
6,974
Kaizhao Liang retweeted
Welcome! We are very excited
John -> Core Automation coreauto.com/ After Microsoft (hunch.net/?p=13764006), I was attracted to a place which was very willing to try new things and exceptionally enabling in doing so.
12
8
1
212
17,934
Kaizhao Liang retweeted
Meet Inkling 2 Small! We’ve tweaked the architecture a bit since the last generation, adding attention residuals (with a q/k/v split), gated attention, dynamic attention sink, dynamic shortconv, value norm and more. Don’t ask me “linear attention wen"
49
51
10
742
63,160
Kaizhao Liang retweeted
Microsoft Ignite starts next week, November 17! The MAI team is bringing something new: new models, demos, and speaker sessions from the people who built them.
56
71
19
1,007
64,421
Kaizhao Liang retweeted
John -> Core Automation coreauto.com/ After Microsoft (hunch.net/?p=13764006), I was attracted to a place which was very willing to try new things and exceptionally enabling in doing so.
63
28
15
818
86,212
Kaizhao Liang retweeted
Tomorrow.
773
959
766
15,809
2,435,718
Today we release Ripple-65B Preview, our 65B dense LLM we pretrained from scratch. Ripple sets a new state-of-the-art for U.S. open-weight models in agentic coding tasks, surpassing significantly larger models including Muse Spark Open-816B, Beam-501B, Inkling-975B, and Nemotron 3 Ultra-550B. Learn more: ricursive.com/ripple
399
544
309
4,549
785,174
Kaizhao Liang retweeted
Releasing the model weights of Kimi K3.1 Pro: huggingface.co/moonshotai/Ki… We'll keep advancing open intelligence.
405
852
486
8,801
1,599,608
🤣🤣🤣
This is an old hat. That's exactly what the neural fast weight programmers did in the 1990s (including the 1991 unnormalised linear Transformer). Overview with links to the original papers: people.idsia.ch/~juergen/fas…
96
This is an old hat. That's exactly what the neural fast weight programmers did in the 1990s (including the 1991 unnormalised linear Transformer). Overview with links to the original papers: people.idsia.ch/~juergen/fas…
We asked GPT-6 Astra to edit Qwen3.5-0.8B weights to improve math. It directly changed only 0.016% of weights, fixing 0.8B's reasoning loops & improving math up to 6x and even instruction following 3x, without gradients or training. We call this Weight-Space Programming: frontier LLMs can now natively read/write neural network weights! 🧵 Blog: rikiyat.github.io/blog/weigh…
Readers added context they thought people might want to know
LLM was given weights explicitly as text in prompt and used code/forward passes (per authors' logs) to ablate Qwen looping direction, a known thinking-mode issue per model card. Primarily fixed repetition in that mode, not general 'natively' improving weights/math. rikiyat.github.io/blog/weight-sp… github.com/RikiyaT/lm_und… huggingface.co/Qwen/Qwen3.5-0… huggingface.co/RikiyaT/Qwen3.…
101
199
99
2,102
231,499
Our cyber models are getting really strong. MAI Cyber model in our security harness has doubled user adoption to 50% in the last 30 days. We also made it 20% more token efficient without sacrificing any quality.
26
15
9
180
29,232
Kaizhao Liang retweeted
can't believe it's actually working
OpenAI has announced to three new discoveries in mathematics, computer science, and physics: - Local smoothing and the Kakeya conjecture - The Exponential Time Hypothesis is false - Existence of quantum chromodynamics and completions of the Standard Model
22
102
10
2,098
68,679
Kaizhao Liang retweeted
Heard from a neolab friend: “Even if we fail to train the best model, we locked in GPU capacity early. H100/B200/B300 rental prices have since gone up 2–3x. We can just resell the compute and print money.” OK, so that’s the business model.
305
271
170
8,253
1,424,023
Kaizhao Liang retweeted
woah, surprised this wasn't found sooner!
Hi all, I have a new result I'm excited to share. It turns out that *all existing implementations* of softmax and cross-entropy (that's JAX, PyTorch, textbooks, etc) have a subtle bug --- two bugs, actually. A simple rewrite fixes both, and eliminates "slingshots" when grokking:
5
5
216
43,654
Kaizhao Liang retweeted
arXiv paper: arxiv.org/abs/2610.31642 HTML version of the arxiv paper: jonbarron.info/softmaxfix/ The interactive widget for the video in the first post: jonbarron.info/softmaxfix/de…
6
24
402
50,186
grep search is all you need
"the bitter lesson for search is to grep over a filesystem, just like how google works" - rich sutton
79