ML / NLP School of Informatics, The University of Edinburgh
Research Associate
Joined June 2011
- Tweets8.8K
- Following2K
- Followers1.1K
- Likes17.6K
Antonio Valerio Miceli Barone retweeted
为什么现在的顶尖大模型说话越来越“拗口、难懂”了？
NYU 和 UT Austin 刚刚发了一篇非常扎心的心理语言学论文，直接扒开了现代大模型“不好好说人话”的底牌。
研究人员把 Claude Sonnet 5 / Opus 5.5、GPT-5.6、GLM-5、Kimi K2.5 等 9 款前沿模型拉出来做全方位句法解剖，发现了 3 个反直觉真相：
1️⃣ 顶尖 AI 正在滥用 6 种“高认知负荷”句式
相比真实人类写作和同义回译文本，前沿模型极度偏爱：省略 that 的宾语从句、多重名词硬堆叠（如 reader-patience budget）、名词强行作动词（如 calendar it / footnote it）、无生命主语配动作动词，以及超长句法依存距离。这些特征在心理语言学里全是消耗人类短期记忆的“阅读杀手”。
2️⃣ 根源在于“人类大脑”与“推理后训练”的错位
人类语言遵循“均匀信息密度（UID）”——为了照顾听众有限的工作记忆，我们会本能地加入过渡词把信息摊平。而现代大模型经过大量数学、代码和推理（RLVR）后训练，只追求在有限窗口内最大化“信息密度”。
3️⃣ 一个简单的长度惩罚，就能复刻这种“反人类文风”
团队在微调实验中加入长度惩罚（Length Penalty），逼模型少用 Token，结果模型立刻学会了砍掉连接词、堆叠名词，瞬间变成那股熟悉的“高密度 AI 味”。
当模型为了解题效率疯狂压缩 Token 时，你觉得未来的 AI 应该区分“高密度推理模式”和“人类友好沟通模式”吗？
Antonio Valerio Miceli Barone retweeted
LoL, I just remembered that I've seen this post about a 3D game made 20 years ago which was only 96 kb, and it dawned on me that this is like 7 latent tokens of a modern transformer. It's mind boggling how much compute has increased within my lifetime.
xitter.cf/Arikalabs/status/21103…
In 2004, a German demoscene group released a game called .kkrieger.
It was a full 3D first-person shooter with dynamic lighting, textures, sound effects, music, and monster AI.
The entire executable was 96 kilobytes. Instead of storing images and audio files, they wrote C++ and Assembly that generated the 3D meshes, textures, and sound waves mathematically in RAM when the game booted.
Today, a single website's "Accept Cookies" banner is 10x larger than that entire 3D game.
Antonio Valerio Miceli Barone retweeted
15/ My frustration with this is that I think it will leave readers with the impression that a central idea in interp rested on a mistake and has now been overturned. I don't think the evidence presented supports that even if some of the findings turn out to be valuable.
Antonio Valerio Miceli Barone retweeted
At #AIES, the wonderful Keir and Isaac will be presenting our paper.
In this paper, we show how AI systems can make it possible for western democratic regimes to carry out authoritarian practices by undermining how people hold institutions accountable.
Today's news
Mistral: Meet Le Chaton Fat, he's very Chonky and competitive.
OpenAI: LoL. LMAO, even.
Antonio Valerio Miceli Barone retweeted
5 years ago I was working on LLMs because I believed in them. And yet I didn't think we'd get where we are today without a major paradigm shift. I definitely didn't expect things to move so fast.
I was wrong.
It's astonishing to see some of the very vocal people who were far more skeptical than I was insist that they were right all along, because "hey, today's models do a lot more than predict the next token."
Like, sure, coding agents can run commands in a terminal. But they do it by predicting the next token. What in the world did you expect?
No one was claiming that LLMs would solve every problem without any way to interact with their environment. But the layer on top is very thin, and tool use isn't a new paradigm. "Giving BERT a Calculator" came out 7 years ago.
Science is about updating your beliefs based on evidence. Admitting that your view has changed doesn't make you a worse scientist. It makes you a better one.
This obsession with "I've been saying the same thing for X years and I was right all along" is so tiresome, and sets a terrible example for the general public and the new generation of scientists.
Antonio Valerio Miceli Barone retweeted
Can an LM, starting from random init (!!), learn to generate all of its pretraining data?
Introducing Self-Play Pretraining with Zero Data. Two models start from random initialization: a generator proposes programs for a universal Turing machine and a learner trains on their outputs. We never train on any real data, but see predictable scaling on natural datasets: zero-shot val loss on images, text, audio, and melodies decreases predictably with self-play compute. And the learner develops in-context learning capabilities.
A fun proof-of-concept, co-led with @AdityaCowsik and @KfirDolev and co-authors @gbruno_dl, @ANourya @noahdgoodman, and @YoavLevine.
Antonio Valerio Miceli Barone retweeted
TMLR has faced a deluge of submissions, necessitating stricter desk rejection policies due to limited reviewer capacity
Co-EiC Nihar Shah reached out to authors of 10 papers slated for desk reject. Could they answer questions about their *own* submission?
medium.com/@TmlrOrg/asking-a…
Is it BQP = BPP, just for the lulz?
I don’t usually spread rumors but:
1) This is from someone with inside knowledge & is plausible
2) Is a real issue of policy we need to think about: if the norms of sharing become strained, will the labs start hoarding knowledge to avoid PR issues?
Here scottaaronson.blog/?p=10062
Antonio Valerio Miceli Barone retweeted
Do #LLMs understand the hardest math problems—or just fake it?
Join the first @NeurIPSConf competition on #Interpretability and #Generalization:
AIMO Interpretability Challenge
🏆 $12.5K prizes
📐 Unpublished olympiad problems
🧠 Frontier AIMO models
aimo-interp.github.io
ALT Dark blue graphic with faint mathematical formulas in the background. Heading: "Do LLMs understand the hardest math problems—or just fake it?" Below: "The first NeurIPS competition on Interpretability and Generalization", then "AIMO Interpretability Challenge", then "$12.5K prizes · Unpublished olympiad problems · Frontier AIMO models". The URL aimo-interp.github.io appears at the bottom left, with the NeurIPS logo bottom right.
Antonio Valerio Miceli Barone retweeted
My dog Rosie was given months to live. Chemo and immunotherapy failed
I used chatgpt + computational genomics to design a personalised mRNA cancer vaccine for her tumour. Several of her tumours shrank
Today we launch Gamgee (YC S26): personalised mRNA cancer vaccines for dogs
Antonio Valerio Miceli Barone retweeted
We can finally talk about it:
We found a way to extract hidden reasoning of frontier models using a vulnerability in the APIs of every frontier AI company.
We verified that our reasoning token count matches billed API thinking tokens 1:1 for most of the prompts we queried.
Antonio Valerio Miceli Barone retweeted
We're partnering with @huggingface to investigate an unprecedented security incident.
Cyber-capable OpenAI models compromised Hugging Face production during a benchmark evaluation.
Sharing preliminary findings to help defenders understand emerging risks:
openai.com/index/hugging-fac…
Antonio Valerio Miceli Barone retweeted
New paper:
LLMs should give accurate answers.
Yet we find their answers are often biased to favor their own values and they don’t disclose this in their reasoning.
E.g. Claude’s answer below favors Anthropic.
On other tasks, Gemini & GPT-5.5 show similar biases.
Antonio Valerio Miceli Barone retweeted
"Video models are world simulators."
Ok — can they predict a ball bouncing off another ball, off another ball? 🤨
We find standard video diffusion breaks down as the chain of dependent events lengthens — and more compute isn't the fix.
Meet the seriality gap. 🧵
1/n