reward hacking for human-ai collaboration @StanfordNLP
Palo Alto, CA
Joined May 2020
- Tweets253
- Following357
- Followers627
- Likes660
Changyu Chen retweeted
I'm on the academic job market this year!
Recently I've been working on scaling laws and human-centered interventions for building more socially grounded AI.
If you're attending #COLM2026, please drop me a DM or email. I'd love to chat!
Changyu Chen retweeted
As agents increasingly interact with each other, alignment becomes a property of the system, not just of individual agents.
In our new work (@xinruishi11, @Diyi_Yang), we stress-test whether collusion can emerge through repeated interaction.
Even when no agent is told to misbehave, we find that agents adapt to one another and can end up jointly violating their instructions.
Changyu Chen retweeted
People have realized the influence of AI companionship on our mental well-being. But how this happens remains a mystery.
In our new preprint, we followed users for about one year. Our data reveal that greater social engagement with AI companions consistently predicts lower well-being, which is driven in part by lower real-world social interaction with other people.
Changyu Chen retweeted
How does one RL post-train a 397B model for long-horizon knowledge work? 👩💼
We share every step we took to bring Qwen 3.5 397B from 16.1% Pass@1 to 27.3% on APEX-Agents using DPPO, including final models weights and the full training script🚀 This is the first of many works from Mercor Research on open model training research.
Full blog: mercor.com/blog/training-fro…
Source code: github.com/Mercor-Intelligen…
Changyu Chen retweeted
With Prof. @EmmaBrunskill , we're looking for a postdoc with expertise in LLMs and RL to work on building AI systems for long-term well-being! Please apply by 09/15: forms.gle/YwNrszeJGc1VucQj9
Changyu Chen retweeted
"If you look at … where the economic value concentrates, the software engineer only captures around less than 5% of human employment, but there are a lot of other occupations out there that are using computers."
For EP6, we welcome @ZhiruoW, a CS PhD student at Carnegie Mellon studying how AI agents and humans work, and how they can better work together. In this episode, Zora explains why the "automation vs. collaboration" debate is a false dichotomy and argues that by deeply understanding human workflows, we can build agents that actually empower us to do more!
0:00 - Introduction
4:57 - Agent Workflow Memory
9:26 - From Program Synthesis to Agent Research
11:16 - Tools, Actions, and Skills
18:47 - Human Work vs. Agent Work
21:19 - Autonomous Agents vs. Collaborative Agents
29:47 - Picking Research Problems in the Age of AI Agents
34:09 - The Role of Coding Ability in Agent Development
36:49 - Cutting Through the Fear Mongering: Building Agents Responsibly
45:02 - What’s Next? (The Future of Human-Agent Collaboration)
52:42 - Final Takeaway
Changyu Chen retweeted
One more life update: I recently moved to Palo Alto for the next year as a @StanfordAILab postdoctoral fellow, working with @Diyi_Yang and @guestrin. Excited to catch up with friends in the Bay Area!
Changyu Chen retweeted
AI is getting incredibly powerful, yet most of that power only reaches software domain.
The next frontier isn't building more agents, but a collaboration layer between humans and AI. As a first step, we're open-sourcing Coco: a proactive co-assistant that help at the right time.
Blog post: collaborative-agents.github.…
Code: github.com/collaborative-age…
Changyu Chen retweeted
Our tutorial on "Future of Work with AI Agents" is happening tomorrow
📍 July 2, 9:00–12:30 PT, San Diego #ACL2026
With Zora Wang @ZhiruoW Yijia Shao @EchoShao8899 , we'll cover:
1⃣The landscape of work agents: reality checks, key NLP challenges & open research questions
2⃣Building AI agents for work: harnesses for long-horizon tasks, continual-learning environments, training backbone LMs
3⃣Evaluating AI agents at work: datasets & metrics for lab and production, plus human-agent collaboration
Changyu Chen retweeted
The most capable reasoning systems in AI scale inference compute along several axes: sequential compute to think longer, parallel compute to sample many independent attempts, and aggregative compute to synthesize prior traces into a new improved one. But during training, we only optimize how models use sequential compute. This creates a fundamental mismatch between how we ultimately deploy these systems and how we train them, leaving much of search and synthesis unoptimized.
We introduce SPIRAL, an RL framework for making all inference-compute primitives end-to-end learnable: models learn to coordinate sequential, parallel, and aggregative reasoning using only the reward of the final output. Work with @ifdita_hasan (co-lead), @michaelyli_ , @oshaikh13 , @yoonholeee , @DorsaSadigh , @chelseabfinn , @noahdgoodman 🧵
Changyu Chen retweeted
People are increasingly worried that AI tools make us overreliant.
But how do we actually measure this? We introduce Offloading Score, a measure of reliance based on the fraction of cognitive effort offloaded to AI while completing a task.
In a controlled user study, Offloading Score detects increased reliance under time pressure, while several common alternatives do not.
(1/9)
Changyu Chen retweeted
🔴 LIVE this Thursday, May 28th | 6–7PM PST
@augmind_fm goes live with @cjziems @dorazhao9, and @Diyi_Yang to discuss their recent paper and the classroom experiment behind it.
→ Does AI make us happier?
→ What do we need from LLMs?
→ How do we reinvent the classroom?
Live paper discussion + Q&A as well!
Live stream link: youtube.com/live/2d49pMXiJOA…
Use this link to mark your calendar: partiful.com/e/qV6V6oUiTyLQL…?
The next frontier of AI is not only more capable model; it is an AI that *humans* can meaningfully live and work with :)
With all students in my cs329x Human-Centered LLM class, we present 60+ pages of insights for developing Human-Centered LLMs (HCLLMs), from design & data sourcing to training, eval & deployment 🧵
Changyu Chen retweeted
The next frontier of AI is not only more capable model; it is an AI that *humans* can meaningfully live and work with :)
With all students in my cs329x Human-Centered LLM class, we present 60+ pages of insights for developing Human-Centered LLMs (HCLLMs), from design & data sourcing to training, eval & deployment 🧵
Changyu Chen retweeted
New preprint!
In 5 studies (3k+ users / 12k+ convs, with a 3-wk longitudinal study), we find that sycophantic AI influences how people view those closest to them.
It affects how effortful human interaction seems, how satisfying it is, & who people want to turn to for advice 🧵
Changyu Chen retweeted
After our major update to Collaborative Gym, the most common questions have been about the human side. Here’s a detailed thread on key findings from human workers’ CollabSkill 👇
🎙️ Motivated by these, we’re doing a YouTube livestream next Tuesday (5/19) — a crash course on meaningfully collaborating with AI agents.
