Wayne Chi retweeted
Many are noticing that frontier LLMs have gotten a lot better at games since Feb 11, 2026
Wayne Chi retweeted
🎶How can we learn to follow along to a music performance in a score without direct supervision?🎶
Introducing FuSiLi (Fused Sinkhorn-Localized Similarity)!
A multimodal contrastive learning approach that learns precise local alignments using only global supervision.
🧵
Wayne Chi retweeted
More meaningful to understand true game dev capabilities of GPT-5.6 Sol on the GameDevBench eval than any slop minecraft clones you see on the timeline
Wayne Chi retweeted
Introducing 🎼🫧MulTTiPop, an evaluation dataset of multitrack pop music for automatic music transcription!
MulTTiPop contains 3.5 hours (572 segments) of aligned multitrack pop audio and MIDI, and can be previewed at gclef-cmu.org/multtipop
🧵1/7
Wayne Chi retweeted
If you are ICML, checkout cool work by @iamwaynechi
Wayne Chi retweeted
ICML Poster Alert: Design has become one of the most interesting directions for eval. Many cool gaming demos go viral, but what game dev skills do agents still struggle with?
Earlier this year, we released GameDevBench, which used loop engineering to evaluate agents capabilities to create games... with surprising updated results about the latest frontier models...
Check out our new ICML paper Tuesday in Seoul!
2:00-3:45 PM KST
Hall A #4203
Arena reached a $100M annual revenue run rate just 8 months after launching our evaluation product. We started as a research project at UC Berkeley with a simple mission: measure AI progress through real-world use. As AI shifts from chatbots to agents taking on longer, higher-stakes work, the problem matters more than ever.
Today, Arena measures real-world AI utility with a community of tens of millions. With Agent Arena, we’re evaluating long-running agents on complex, real-world tasks - how they use tools, adapt to feedback, recover from errors, and accomplish goals set by humans.
We are excited to keep deepening our work in agentic evaluations.
Here’s @ml_angelopoulos on what this milestone means and where we go from here:
Arena has crossed $100M in annualized revenue run rate, eight months after launching our evaluation product.
With our recent release of Agent Mode, millions of users on Arena are doing real work with agents, from coding to document analysis, in long-running, multi-turn sessions with hundreds of tool calls. Arena now evaluates objective criteria like task completion rates, hallucination rates, and more, far beyond our original human preference voting model. This expansion has taken us from a student project at Berkeley to one of the fastest growing companies in history. Go Bears! 🐻
Our core thesis is simple: to align AI with human values, we must directly measure its impact on people in the real world. Today's milestone is proof that Arena’s platform is the de-facto standard for post-deployment evaluation of AI.
Wayne Chi retweeted
Can an AI agent surface why an ML paper might be hard to reproduce — just by reading it, without running any code?
We build ReproRepo, a framework for auditing reproducibility with agents.
Across 1,149 recent papers, the best agent surfaced a semantically related, human-reported reproducibility blocker for ~90% of them. 🧵👇
Wayne Chi retweeted
Attention @arxiv authors: Our Code of Conduct states that by signing your name as an author of a paper, each author takes full responsibility for all its contents, irrespective of how the contents were generated. 1/
Wayne Chi retweeted
I am thrilled to share that I will be attending ✨ #NeurIPS2025 in San Diego this week! 🌊@chrisdonahuey, @mittu1204, and I will be presenting Music Arena, an open platform for scalable human preference evaluation of Text-to-Music models. 🎵⚔️
Try it out & vote! 👇
🌐 Platform: music-arena.org/📄 Paper: arxiv.org/abs/2507.20900💻 Code: github.com/gclef-cmu/music-a…
#GenerativeAI #MusicAI #TextToMusic #CreativeAI
Wayne Chi retweeted
Today's AI agents are optimized to complete tasks in one shot. But real-world tasks are iterative, with evolving goals that need collaboration with users.
We introduce collaborative effort scaling to evaluate how well agents work with people—not just complete tasks 🧵
Wayne Chi retweeted
🎵Music Arena ⚔️ was accepted to the NeurIPS 2025 Creativity Track, and we've released a big update to celebrate!
Includes new models from @SonautoAI and @elevenlabs. Also, Music Arena is now available as a 🤗 @huggingface space and dataset!
Wayne Chi retweeted
⚔️PR Arena is LIVE⚔️
We’re kicking off with 5 frontier LLMs tackling GitHub issue resolution.
Resolve issues for free & vote for your preferred fix!
👉Leaderboard & Setup Guide: prarena.web.app
Wayne Chi retweeted
We are excited to launch the ⚔️PR Arena⚔️ leaderboard!
Full results will be revealed after a certain milestone of community votes.
Fix your GitHub issues for free and vote for better fix!
👉Leaderboard & Setup Guide: prarena.web.app
Wayne Chi retweeted
Sharing our initial leaderboard and open data release for 🎶Music Arena⚔️!
Music is subjective and multi-dimensional. A key goal of Music Arena is to provide insights beyond binary preferences! 🧵
Wayne Chi retweeted
We’re featured in the new tech report on Mercury models! Check it out👇
Since our launch earlier this year, we are thrilled to witness the growing community around dLLMs.
The Mercury tech report from @InceptionAILabs
is now on @arxiv with more extensive evaluations: arxiv.org/abs/2506.17298
New model updates dropping later this week!
Wayne Chi retweeted
Excited to announce 🎵Magenta RealTime, the first open weights music generation model capable of real-time audio generation with real-time control.
👋 **Try Magenta RT on Colab TPUs**: colab.research.google.com/gi…
👀 Blog post: g.co/magenta/rt
🧵 below
Wayne Chi retweeted
New result: Qwen-2.5-Coder jumps from 13th to joint 1st place with fill-in-the-middle (FiM)! Congrats to @Alibaba_Qwen 🥳
Also check out @lmarena_ai 's new UI 🖥️✨
Wayne Chi retweeted
blog.ml.cmu.edu/2025/06/01/r…
In this in-depth coding tutorial, @GaoZhaolin and @g_k_swamy walk through the steps to train an LLM via RL from Human Feedback!



























