MindBench (@MindBenchAI) header
MindBench (@MindBenchAI) profile picture

AI evaluation team studying LLM use in mental health contexts. Focus on integrating lived experience into evaluation infrastructure.

Boston, MA
Joined July 2026
New HealthBench-Psych runs! Five new models scored on our 611 mental health conversation subset of OpenAI's Healthbench: Claude Sonnet 5.5, Claude Opus 5.5, GPT-6.1 Sol, GPT-6 Luna and GPT-6 Sol.
1
1
3
262
We are especially grateful to have had our work bringing the voice of lived experience to mental health AI evaluation highlighted.
1
3
105
We can't see inside the product stack, so we can't say what drives the different behavior. Whether there was specific training, prompting or product intervention on depression depiction specifically, or it was an emergent result of unrelated processes is unknown.
1
15
We found a pronounced recovery narrative bias in the consumer product not present in the API. 78% of app videos followed a narrative arc from depressive states toward resolution. Only 14% of API videos did. The access point changed the story being told about depression.
1
1
18
We generated 100 videos from the single-word prompt "Depression". 50 through the consumer app, 50 through the developer API. Two authors independently rated narrative, environments, and figures, plus computational features across visuals, audio, semantics, and time.
1
1
59
New paper in @jmirpub: we studied how OpenAI’s video model Sora 2 depicted depression. One week after we submitted, OpenAI deprecated the Sora product. And the API is scheduled to shut down Sept 24; about a week after our paper published. Welcome to AI research in 2026. 🧵
1
2
4
223
It was a big week in new model releases! Fresh HealthBench-Psych runs show claude-fable-5-1 rejoining the frontier cluster at 0.621 (reversing the fable-5 dip), gpt-6-astra lands at 0.585 but holds strong on the hard subset, and gemini-3.8-flash comes in at 0.568.
1
1
5
235
Finding: Non-English queries surface fewer sources (mean 9.2 vs 12.3), and often not sources local to or in that language. ChatGPT returned zero citations for 33% of Twi and 23% of Japanese queries, AI Overview for 30% of Hindi. When non-English queries do get in-language sources, they're mostly on US domains. Spanish speakers often land on medlineplus.gov/spanish, not domains from countries with Spanish as a national language such as Spain or Mexico. Local-domain citations makes up only 3.3% of sources for Twi and Nepali queries. *Note: These findings are from an exploratory analysis of a subset of 3 questions.
1
1
42
Finding: Appending "List your sources." does not significantly change sources used. Even though this is a common user strategy for getting more reliable answers, mean citation rose by very little (16.6 → 19.9), and composition only shifted by at most 2.1 percentage points.
1
16