@humansand, formerly @elicitorg
San Francisco
Joined November 2022
- Tweets143
- Following247
- Followers1.5K
- Likes448
Charlie George retweeted
We accelerated one of the most optimized NVFP4 decode kernels while also making it more accurate
Fewer bits typically mean more speed but less accuracy. We did the opposite: more bits, but up to 10% faster generation
Already used in Ampersand’s RL and Boba serving
Go chat with ampersand!
Introducing Boba - the social intelligence that understands your goals and your community. Text Boba today or message it online at boba.app
Charlie George retweeted
first we built Persimmon, our sota human simulator. using that we trained Ampersand, a model that understands people individually and in groups. which powers Boba — released today! @humansand is cooking!
Introducing Boba - the social intelligence that understands your goals and your community. Text Boba today or message it online at boba.app
Great thread from Noah with more details on the eval awareness results!
while evaluating Ampersand v0.1 for sycophancy @__Charlie_G found something interesting: deployment makes models more sycophantic, but good human simulators substantially reduce evaluation awareness.🧵
Charlie George retweeted
while evaluating Ampersand v0.1 for sycophancy @__Charlie_G found something interesting: deployment makes models more sycophantic, but good human simulators substantially reduce evaluation awareness.🧵
Cool result from the Ampersand system card: our user model Persimmon with test-time scaling reduced self-reported evaluation awareness to deployment-like levels, 40 percentage points below prompted Opus 5.5 for the most evaluation-aware models in our multi-turn sycophancy evals.
No one changes the world alone
Introducing Ampersand - the first model built for us. Ampersand is a collaborator trained to help people achieve their individual and collective goals. In multi-month simulations, users collaborating with Ampersand accomplished more than with Fable and other frontier models
Available to everyone through our API
I'm very excited about the direction of scaling compute to build higher fidelity simulations to reduce evaluation awareness in frontier systems. More details on this work: ampersand.humansand.ai/blog/…
Our new model trained to interact with groups of people over a long time horizon in high fidelity simulations is now out!
No one changes the world alone
Introducing Ampersand - the first model built for us. Ampersand is a collaborator trained to help people achieve their individual and collective goals. In multi-month simulations, users collaborating with Ampersand accomplished more than with Fable and other frontier models
Available to everyone through our API
Charlie George retweeted
No one changes the world alone
Introducing Ampersand - the first model built for us. Ampersand is a collaborator trained to help people achieve their individual and collective goals. In multi-month simulations, users collaborating with Ampersand accomplished more than with Fable and other frontier models
Available to everyone through our API
Charlie George retweeted
my roommate (who works at a diff ai lab) and i both came home unusually early today
surprised, i asked “why are you back so early??”
he said: codex is down, you?
1/7 Very excited about the Persimmon release!
Massive congrats to the entire team, particularly @314Bansal and @alexisjross, for leading this work.
6/7 The Persimmon model makes this kind of research possible by allowing researchers to simulate human-like behaviour through an API, setting a new standard for realism.
Charlie George retweeted
AI models can solve formidable math problems, trounce humans at programming competitions, and outperform doctors. But can they help people make sound decisions when the evidence is ambiguous and the stakes are high? How do we know they are helping us reason better, not just faster?
To find out, we built BioDecisionBench, a collection of 40 task variants derived from 26 cases of complex reasoning failures in life sciences. The tasks span the drug-development process, from target selection to clinical trial design to choosing new indications for a successful drug.
High-stakes decision-making under ambiguity and limited information is an acute problem in drug development. A company only gets to pick a small set of targets to pursue, a limited number of assays to run, and a single clinical trial design to conduct. It’s not possible to explore all avenues, so the only way to do better is to reason better about how things might have gone differently if they had chosen differently.
Existing benchmarks don’t meet this challenge: they tend to ask narrow questions, and they only evaluate final answers, rather than evaluating the reasoning that led to the answer.
