Co-founder @softmaxresearch: organic alignment through multi-agent RL. Co-founded @PluralityInst and @OpenBallotApp. Prev Google, Facebook, Asana.
San Francisco, CA
Joined March 2009
- Tweets90
- Following511
- Followers336
- Likes21
1/ Capabilities and alignment get treated as separate problems. I think one mechanism covers both: train agents in worlds full of other learning agents.
4/ Nature's mechanism for that middle is kinship: shared reward. Ants sit at one end, pure traders at the other, and norms, reputation, shame and love sit between them.
5/ Adults are vastly more capable than their kids and mostly act for the kids' wellbeing anyway. That's the existence proof I'd want alignment people to take seriously. It's what we work on at @softmaxresearch: softmax.com/mission
We don't ask whether a person is aligned. We ask aligned with whom, and we find out over years. Alignment is a relation between agents, not a property you can read off one of them.
We explore this in “Origins of Trust” at @softmaxresearch: where trust between humans and AI might come from, and what that implies for how we train them.
softmax.com/blog/origins-of-…
A cell doesn't know the body plan. It knows its neighbors and what it shares with them, and a body comes out anyway. Getting global alignment from local relationships might be easier than getting it from a global spec.
Hamilton's rule is a reward-sharing matrix written in genes. Parents share half their genes with a child, worker ants three quarters with a sister. We call the result love, and nobody trained it as a behavior. We're testing the same trick in multi-agent RL at @softmaxresearch.
Noise still helping at test time is the part I find interesting: cells that only see neighbors have to tolerate each other's errors. When you prune redundant parallel trajectories, do the survivors differ in strategy, or just in noise?
Self-play partners invent private conventions that only work with each other. The real test of a cooperative agent is a stranger. Report cross-play, not self-play.
Every alignment plan ends with trusting AI with real responsibility. Most say trust comes from examining the model. We think it requires the capacity to care and builds through shared history and a track record. With @eshear for @softmaxresearch: softmax.com/blog/origins-of-…
360+ game tasks for LLM RL, and training on games carries over to reasoning better than math or code RL in their setup. The environment is the curriculum. Worth a read from @VmaxAI, by @creus_roger @Cote_Marc @GlenBerseth @pcastr. xitter.cf/VmaxAI/status/21136608…
Voter guides are most useful where they disagree. If the Chronicle and the League of Women Voters split on a prop, that's the one to read up on. @OpenBallotApp lines them up so you find those in a minute instead of an evening.
An optimizer learns what the environment rewards. Agents that care have to come from environments that reward care: again and again, under pressure, with real others. You can't bolt it on with fine-tuning afterward.
Ask a model the red/blue button question fast, it says blue. With more reasoning budget, or heavy math and code training, it drifts to red. The dominance argument is valid but the wrong frame: it holds everyone else fixed. @softmaxresearch softmax.com/blog/red-button-…
Single-agent RL treats other agents as weather. Multi-agent RL treats them as minds that model you back. Most alignment worry is about systems that treat us as weather.
An agent is a dual of its training environment. Train it alone on tasks and you get something fit for tasks. Train it among other learning agents and it has to model them.
People shouldn't need 15 browser tabs to research their ballot. @OpenBallotApp puts 100+ California voter guides side by side for every prop and local race, so you can see where the ones you trust agree and disagree. Free, nonpartisan, no login. openballot.app
