Research Scientist @Meta | Allen & Kelli Questrom Professor @BUQuestrom | 🇨🇦
Boston, MA
Joined April 2009
- Tweets1.3K
- Following632
- Followers1.2K
- Likes4.4K
Seeing this study being widely re-shared with the take away that now AI >= human in peer review, in every dimension. People need to tone it down a little... This is primarily GPT evaluating Claude's comments. To me, the more useful signal is the 3 past editors of unnamed journals rating human vs Claude reviews, but this is only related to just 10 papers, it's not clear the editors have specific expertise in the paper topics or methods, and it reads as though the editors looked at the reviews but not the actual papers. I know we will need AI in the loop in some fashion to manage the paper deluge. I also realize that AI is already widely used in peer review and it's better to step in to manage / govern that process than let people do whatever they want. In my experience, AI certainly surfaces useful / good comments, particularly on methods, but many of the comments can be pedantic. Many models also still have terrible judgment about the other aspects that determine a study's value, e.g., whether the question is important. We're not yet at the point of ditching the human. Note: I'm pushing back here on the interpretations people are sharing when circulating this work, not the discussion in the paper itself.
New paper: "AI Article Reviews Dominate Human Peer Reviews at Their Best: Evidence and Policy Choices" See: GaryKing.org/peerless. with Steven Worthington. Comments welcome!
This is a general issue with many startups in this space; a lack of transparency and an apparent lack of rigor in the way evaluations have been conducted.
I have been interested in LLM simulations of humans for a while, and recently saw lots of posters congratulating Aaru on these results. So I did what any curious person should do after reading the post, check the quality of the analysis by asking ChatGPT and Refine (if you want to be more thorough).
Here's what both AI referees say. Aaru uses a mystery methodology, doesn't have a good baseline model hurdle to clear, has potential data leakage issues, and makes unjustified statistical claims and comparisons to the published literature.
Perhaps Aaru has made a real advance and the above issues can be addressed. But without peer review or other pressure to force the authors to address them, we can't be sure. Until then, skepticism is warranted.
Gord Burtch retweeted
I have been interested in LLM simulations of humans for a while, and recently saw lots of posters congratulating Aaru on these results. So I did what any curious person should do after reading the post, check the quality of the analysis by asking ChatGPT and Refine (if you want to be more thorough).
Here's what both AI referees say. Aaru uses a mystery methodology, doesn't have a good baseline model hurdle to clear, has potential data leakage issues, and makes unjustified statistical claims and comparisons to the published literature.
Perhaps Aaru has made a real advance and the above issues can be addressed. But without peer review or other pressure to force the authors to address them, we can't be sure. Until then, skepticism is warranted.
Gord Burtch retweeted
Deploying DiffusionGemma-Jev (djev) just got a lot easier. You can now spin up a Jev API-compatible endpoint on Google Cloud Run using a single command.
Performance is solid: ~35-60 ms for single step latency and batch@32 is ~100-123 requests/sec.
It's a straightforward way to experiment without needing your own GPU. Runs at roughly $3/hr and drops to $0 when idle.
Get the code and instructions here: github.com/taeold/djev-run
Gord Burtch retweeted
GPT-6 Astra represents a step-function change in model capability for interactive reasoning problems. It scores 66% on ARC-AGI-3 using our standard harness, and nearly 100% with a continuous conversation harness and custom compaction, at a cost of roughly $360 per game.
In fact, the continuous harness version significantly outperforms our human baseline in action efficiency across almost all levels. When we examined the reasoning chains to understand how the model operates, we found it performing highly efficient, on-the-fly symbolic world modeling for each game and level. It goes as far as developing its own shorthand DSL to represent in-game situations -- essentially a game-specific algebraic notation.
Overall, Astra exhibits symbolic modeling behaviors we had previously only seen with sophisticated harnesses -- so harness capabilities are increasingly shifting into the model itself.
We see Astra as a major breakthrough in model intelligence.
Read our post on Astra and what these results mean: arcprize.org/blog/astra
Gord Burtch retweeted
Many informed observers have skepticism about the claims of companies like Simile and Aaru. They do not release datasets that results can be replicated on and have not published peer-reviewed work. While the proposed methods are cool and natural extensions of fine-tuning, there is not enough information to evaluate how good they really are.
Anyone can simulate the future. But the simulation only matters if it’s trustworthy.
At Simile, we train two types of models: simulation models and confidence models. Our first research blog post explores the origin of our proprietary confidence model, which predicts the accuracy of our population simulations and, in turn, makes them actionable.
simile.com/blog/confidence?v…
Gord Burtch retweeted
Is all code becoming the same?
On one hand, 95% of Kaggle submissions that set a random seed now use 42 (a Hitchhiker's Guide joke LLMs love). But, it turns out that while coding syntax is converging, approaches to problems are not converging. Human prompters drive real variety.
Gord Burtch retweeted
Here's an interesting study: An analysis of @kaggle code submissions by @gburtch showed AI coding agents are standardizing implementation details (such as using 42 as a random seed), but so far there is no evidence of homogenization of approaches and problem-solving strategies.
The Hitchhiker’s Guide to Monoculture
arxiv.org/pdf/2607.13077
Gord Burtch retweeted
Imagine every pixel on your screen, streamed live directly from a model. No HTML, no layout engine, no code. Just exactly what you want to see.
@eddiejiao_obj, @drewocarr and I built a prototype to see how this could actually work, and set out to make it real. We're calling it Flipbook. (1/5)
Gord Burtch retweeted
Ladies and gentlemen, today we're launching one of our biggest changes to 𝕏
Introducing Custom Timelines
This feature allows you to pin a specific topic to your home tab. With support for over 75 topics, you can dive deep into your favorite niche on X.
It's powered by Grok's understanding of every post with the algorithm's personalization—meaning every timeline is made just for you. And it works even better when it's a topic you already engage with.
This was a huge undertaking across many months, so we're excited for you take it for a spin.
We're giving early access to Premium subscribers on iOS (and Android coming very soon).
Gord Burtch retweeted
Just updated **panelView** in R: it now visualizes multi-way fixed-effect structures as a network graph, building on earlier work by Sergio Correia. You can easily identify singletons and duplicate observations before estimation. yiqingxu.org/packages/panelv…
Gord Burtch retweeted
Announcing a new Claude Code feature: Remote Control. It's rolling out now to Max users in research preview. Try it with /remote-control
Start local sessions from the terminal, then continue them from your phone. Take a walk, see the sun, walk your dog without losing your flow.
A good step on Prolific's part
Introducing bot authenticity checks that detect agentic AI and bots in online research.
100% accuracy in testing. Our tool flags automated browser scripts or non-human behavior within your @Qualtrics studies, so you can collect data with high confidence that it's human.
Gord Burtch retweeted
eBay has just banned AI agents that buy products on people's behalf. As @alexolegimas and I describe in our article (link below), this behavior is a barrier to consumer friendly agentic commerce. To counteract it, we may need a right to bring your own agent.
We've a new paper posted, measuring audience imbalance (divergent delivery) across large samples of advertiser A/B and Lift tests at Meta (only A/B tests show any). We evaluate how much imbalance can be reduced by configuring the experiment the 'right way'. Check it out!
1/ New paper on advertising A/B & Lift Tests! 📄
Lots of debate about these, especially on *divergent delivery*, when ad delivery algos push different campaign variants to different audience segments. Implications?
Paper: arxiv.org/abs/2508.21251 @gburtch
New piece out today in the @AMJPublicHealth "Abortion Restrictions and Infant Mortality in the United States, 2018–2023" ajph.aphapublications.org/do…
Gord Burtch retweeted
What's the optimal number of steps of physical activity that is associated with multiple (9) favorable outcomes?
The largest, systematic review of 57 studies, 35 cohorts.
7,000 steps is clinically meaningful; some benefit also seen for 4,000 steps cf 2,000 steps @TheLancet thelancet.com/journals/lanpu…
Gord Burtch retweeted
📜 New working paper @apo_filippas @johnjhorton & diego
AI is *rapidly* changing the job application process.
📈Employers getting 1000s of applications for a single job, all seemingly well-qualified.
👥Qualified workers are finding it difficult to stand out from the sludge of AI applications.
💡Our paper offers a market-based solution for this problem: allow workers to pay to send a costly signal of interest when applying for a job. (1/n)
New piece out today in Proceedings of the National Academies of Science... take caution using LLMs as human surrogates pnas.org/doi/10.1073/pnas.25…
