reward hacking for human-ai collaboration @StanfordNLP
Palo Alto, CA
Joined May 2020
- Tweets253
- Following357
- Followers627
- Likes660
Pinned Tweet
(1/3) My favorite figure from the paper.
Nearly all open-source RL frameworks introduce an unintentional bias when computing the masked mean 😮. The fix? Just replace mask.sum with a constant.
AI is getting remarkably good at doing things for humans. We know much less about how to make it better at helping humans learn.
We study learning to teach adaptively: how AI can adapt its teaching to each learner’s evolving understanding, and how this capability can itself be optimized through reinforcement learning.
A small step toward AI that doesn’t just solve problems, but helps people understand and grow.
More details in @WeixianXu ’s thread 👇
Language models already excel at math, but human growth remains essential: we need AI to teach people, not replace them!
Teaching is different from solving. As the Chinese principle “因材施教” suggests, preferences differ from person to person, and a good teacher should adapt their teaching to each learner.
We introduce Sherpa, a framework for training LLMs to teach adaptively.
Changyu Chen retweeted
I'm on the academic job market this year!
Recently I've been working on scaling laws and human-centered interventions for building more socially grounded AI.
If you're attending #COLM2026, please drop me a DM or email. I'd love to chat!
Changyu Chen retweeted
As agents increasingly interact with each other, alignment becomes a property of the system, not just of individual agents.
In our new work (@xinruishi11, @Diyi_Yang), we stress-test whether collusion can emerge through repeated interaction.
Even when no agent is told to misbehave, we find that agents adapt to one another and can end up jointly violating their instructions.
Changyu Chen retweeted
People have realized the influence of AI companionship on our mental well-being. But how this happens remains a mystery.
In our new preprint, we followed users for about one year. Our data reveal that greater social engagement with AI companions consistently predicts lower well-being, which is driven in part by lower real-world social interaction with other people.
Changyu Chen retweeted
How does one RL post-train a 397B model for long-horizon knowledge work? 👩💼
We share every step we took to bring Qwen 3.5 397B from 16.1% Pass@1 to 27.3% on APEX-Agents using DPPO, including final models weights and the full training script🚀 This is the first of many works from Mercor Research on open model training research.
Full blog: mercor.com/blog/training-fro…
Source code: github.com/Mercor-Intelligen…
Changyu Chen retweeted
With Prof. @EmmaBrunskill , we're looking for a postdoc with expertise in LLMs and RL to work on building AI systems for long-term well-being! Please apply by 09/15: forms.gle/YwNrszeJGc1VucQj9
Changyu Chen retweeted
"If you look at … where the economic value concentrates, the software engineer only captures around less than 5% of human employment, but there are a lot of other occupations out there that are using computers."
For EP6, we welcome @ZhiruoW, a CS PhD student at Carnegie Mellon studying how AI agents and humans work, and how they can better work together. In this episode, Zora explains why the "automation vs. collaboration" debate is a false dichotomy and argues that by deeply understanding human workflows, we can build agents that actually empower us to do more!
0:00 - Introduction
4:57 - Agent Workflow Memory
9:26 - From Program Synthesis to Agent Research
11:16 - Tools, Actions, and Skills
18:47 - Human Work vs. Agent Work
21:19 - Autonomous Agents vs. Collaborative Agents
29:47 - Picking Research Problems in the Age of AI Agents
34:09 - The Role of Coding Ability in Agent Development
36:49 - Cutting Through the Fear Mongering: Building Agents Responsibly
45:02 - What’s Next? (The Future of Human-Agent Collaboration)
52:42 - Final Takeaway
Changyu Chen retweeted
One more life update: I recently moved to Palo Alto for the next year as a @StanfordAILab postdoctoral fellow, working with @Diyi_Yang and @guestrin. Excited to catch up with friends in the Bay Area!
Changyu Chen retweeted
AI is getting incredibly powerful, yet most of that power only reaches software domain.
The next frontier isn't building more agents, but a collaboration layer between humans and AI. As a first step, we're open-sourcing Coco: a proactive co-assistant that help at the right time.
Blog post: collaborative-agents.github.…
Code: github.com/collaborative-age…
yanzhe is building an awesome team at HKU!! this is a great chance to join early and help shape the lab from day one
EXCITED to share that I’ll be joining the University of Hong Kong (HKU) as an Assistant Professor in January 2027!
I’m interested in AI agents 🤖, Human-AI Interaction 🤝 and their safety🛡️.
Email me if you’d like to chat! I’ll also be at ACL from July 3–6!
stevenyzzhang.github.io/webs…
Changyu Chen retweeted
Our tutorial on "Future of Work with AI Agents" is happening tomorrow
📍 July 2, 9:00–12:30 PT, San Diego #ACL2026
With Zora Wang @ZhiruoW Yijia Shao @EchoShao8899 , we'll cover:
1⃣The landscape of work agents: reality checks, key NLP challenges & open research questions
2⃣Building AI agents for work: harnesses for long-horizon tasks, continual-learning environments, training backbone LMs
3⃣Evaluating AI agents at work: datasets & metrics for lab and production, plus human-agent collaboration
Changyu Chen retweeted
The most capable reasoning systems in AI scale inference compute along several axes: sequential compute to think longer, parallel compute to sample many independent attempts, and aggregative compute to synthesize prior traces into a new improved one. But during training, we only optimize how models use sequential compute. This creates a fundamental mismatch between how we ultimately deploy these systems and how we train them, leaving much of search and synthesis unoptimized.
We introduce SPIRAL, an RL framework for making all inference-compute primitives end-to-end learnable: models learn to coordinate sequential, parallel, and aggregative reasoning using only the reward of the final output. Work with @ifdita_hasan (co-lead), @michaelyli_ , @oshaikh13 , @yoonholeee , @DorsaSadigh , @chelseabfinn , @noahdgoodman 🧵
Changyu Chen retweeted
People are increasingly worried that AI tools make us overreliant.
But how do we actually measure this? We introduce Offloading Score, a measure of reliance based on the fraction of cognitive effort offloaded to AI while completing a task.
In a controlled user study, Offloading Score detects increased reliance under time pressure, while several common alternatives do not.
(1/9)
Changyu Chen retweeted
🔴 LIVE this Thursday, May 28th | 6–7PM PST
@augmind_fm goes live with @cjziems @dorazhao9, and @Diyi_Yang to discuss their recent paper and the classroom experiment behind it.
→ Does AI make us happier?
→ What do we need from LLMs?
→ How do we reinvent the classroom?
Live paper discussion + Q&A as well!
Live stream link: youtube.com/live/2d49pMXiJOA…
Use this link to mark your calendar: partiful.com/e/qV6V6oUiTyLQL…?
The next frontier of AI is not only more capable model; it is an AI that *humans* can meaningfully live and work with :)
With all students in my cs329x Human-Centered LLM class, we present 60+ pages of insights for developing Human-Centered LLMs (HCLLMs), from design & data sourcing to training, eval & deployment 🧵
Changyu Chen retweeted
The next frontier of AI is not only more capable model; it is an AI that *humans* can meaningfully live and work with :)
With all students in my cs329x Human-Centered LLM class, we present 60+ pages of insights for developing Human-Centered LLMs (HCLLMs), from design & data sourcing to training, eval & deployment 🧵
Changyu Chen retweeted
New preprint!
In 5 studies (3k+ users / 12k+ convs, with a 3-wk longitudinal study), we find that sycophantic AI influences how people view those closest to them.
It affects how effortful human interaction seems, how satisfying it is, & who people want to turn to for advice 🧵
Life update: I'm super excited to join @Stanford as a postdoc working with @Diyi_Yang ! I’ll continue my research on RL, and recently I’ve become especially interested in how RL can contribute to human-AI collaboration and collaborative agents.
A new chapter begins, from the sunny island to the sunny state ☀️🏝️
Most current AI systems are optimized to solve tasks autonomously, often trained by verifierable signals. But being a good collaborator requires much more: (real-time) communication, coordination, planning ahead, and working proactively.
I’m really excited to see frontier labs looking into this direction and pushing genuine ideas. E.g. thinky's interaction models are awesome! @thinkymachines
xitter.cf/thinkymachines/status/…
People talk, listen, watch, think, and collaborate at the same time, in real time. We've designed an AI that works with people the same way.
We share our approach, early results, and a quick look at our model in action.
thinkingmachines.ai/blog/int…