Mai Shimada

Japanese speakers agreed, and many non-Japanese speakers seemed amused. This is one of the core linguistic complexities of training LLMs in Japanese. Word order and omission carry pragmatic meaning (tone, focus, and shared context) that isn't explicitly written in the tokens.

Training on realistic professional workflows improved performance beyond those domains, including on Terminal-Bench. - @LiudasPanavas(SurgeAI) A recent talk “Real World vs. Benchmark: The Case for Vague Prompts and Messy Environments.” by Liudas at Benchmark summit felt very real while we build real-world benchmark ourselves. I was particularly fascinated when he said that they trained on data resembling the Day Job tasks and saw gains of 16 percentage points on GDPval and 8 percentage points on Terminal-Bench. Summary below. 1. Real-World Benchmark Design: Day Job Series - Many professional benchmarks evaluate “models with training wheels on.” - Verbose prompts hand over domain expertise: where to look, what formulas to use, and how to format the answer. - They can become instruction-following benchmarks rather than domain reasoning benchmarks. - Day Job’s approach: short prompts + complex, messy environments. - Prompt: a few sentences, e.g. “Prepare the budget vs. actual analysis.” - Environment: mixed emails, PDFs, and spreadsheets; not every document is relevant. - Tasks represent work done weekly or monthly, rather than niche, one-off scenarios. 2. Agent Failure Modes in Messy Environments - Struggle to orient: one model merged notes from two patients without noticing that 8 of 23 files belonged to a different patient. - Miss domain judgment calls: a model correctly reasoned that sensitive health details should be excluded, then listed every diagnosis in the final letter anyway. - Models can “know” the right answer when asked directly but fail to apply it mid-trajectory. - Trip on everyday workflows: in an accounting task, a model incorrectly reversed revenue after a customer’s $2,000 check bounced. - Retrieval is often good; the gap is in using retrieved information correctly. - A common failure: pulling numbers from a draft document when a final version exists. 3. Benchmark Design Principles - Short prompts, complex environments: complexity belongs in the environment, not the prompt. - Purposeful environments: arbitrarily adding files does not necessarily increase difficulty for frontier models. - In one experiment, adding adjacent-but-irrelevant tax and case-law documents had no effect on top models’ pass rates. - The challenge is reasoning within and across relevant documents. - Keep experts in charge of task creation and quality control. QC agents can simplify tasks when they lack domain knowledge, removing the exact judgment calls being tested. - The team uses unweighted rubrics, emphasizing that mistakes in consequential workflows matter. - Preference data deserves more attention in agentic evals: a correct answer buried in 10 pages may still be a poor user experience. 4. Don’t Leave Everyday Work Behind - Progress on frontier problems is exciting, but agents also need to reliably handle the work people want to delegate today. - Closing the books or preparing clinical notes may be routine, but mistakes have real consequences. - Benchmark success should translate into less supervision, time saved, and work users can trust. 4. Realistic Work Tasks Can Build Transferable Skills - Liudas reported that training on data resembling these professional workflows improved performance on both GDPval and Terminal-Bench. - The reported gains were 16 percentage points on GDPval and 8 percentage points on Terminal-Bench. - Navigating environments, connecting evidence, and keeping track of work are useful across finance, healthcare, and coding. - Well-designed work tasks can provide training signal that transfers beyond the original domain.

What happens when benchmarks take months to build but models improve on them within weeks? I've been thinking about this after hearing a talk by @xdotli (BenchFlow) on “Superhuman Evals for Superhuman Intelligence.” from the Benchmark Summit earlier this week. Sharing notes for ppl building in this space. 1. Benchmark Saturation and the Verification Problem - Benchmarks are being saturated faster than ever, especially post-2024.erifiable vs. non-verifiable.” - Low-hanging verifiability, cheap-to-check tasks, is being exhausted. More ambitious tasks can be expensive to check. - Reward hacking is a risk across public environments, although the audits described found few instances. - Animation-to-code benchmarks expose verifier flaws: frame-level checks can pass animations whose timing or motion is wrong. 2. Frontier Physics: Building a Superhuman Benchmark - 56 tasks, sourced from original research, with 70+ physicists contributing. - Contributors had to be PhD-level researchers bringing their own prior work. - Around 8,000 human hours invested before any agent rollout. - Reportedly uses 41x more tokens per task than HLE. Some rollouts require dedicated H200 compute for up to 48 hours. - Validation: human auditing of agent traces, with failure modes labeled. - A leading failure: lack of rigorous experimental design. - An experiment may produce results without actually supporting the claimed conclusion. - Reproducibility is a major bottleneck: partners may need to drop tasks they lack the compute to run. 3. Toward Infinite Training Signal: The Collector Concept - Current evals are finite and expensive to build. New data formats and collection infrastructure could help scale the signal. - Analogy: iPhone + YouTube created an abundance of video that later helped train world models. - Vision: a “collector” that continuously captures agent traces and human interactions as potential training signal. - Corrections, interventions, and feedback contain useful information, but traces need to be stored and scored in ways that make that information usable. 0 Key infrastructure needs: environment snapshotting, rollout branching, and new formats for evaluation and improvement signals. Xiangyi’s bet: recursive self-improvement will emerge at organizations with the data, formats, and culture in place, not only at the top two or three labs.

Sharing some more notes from the Benchmark Summit for anyone designing evals, Making Fun (but useful) Evals! by Joseph Jennings (xAI) was a lot of fun. Evals should be fun (and useful)! 1. Why Fun Evals Matter - Best evals are unlikely to be trained on, easy to verify, and observable when they fail - Key property: generalizable but not expected to climb with scale - Microsoft AI paper showed most standard benchmarks can be approximated via low-rank decomposition - Labs will default to cheaper, simpler evals if they correlate well - Push for out-of-distribution, non-approximable evals to stay meaningful 2. Pokémon and Diplomacy - Grok 5 beat Pokemon FireRed (first model to do so), but likely memorized routes from pretraining - ROM hack test (Pokemon Brown, randomized map): Grok failed entirely, Astra could navigate partially - Suggests Astra has more generalizable spatial reasoning than Grok - Diplomacy (7-player negotiation + hidden action game): GPT-6 Astra leads at 37.5% vs. 14% random baselineDiplomacy score correlates with poker performance and willingness to bluff - Stronger models tend to take fewer all-ins in poker 3. Cryptic Crosswords and Pelican on a Bicycle - Cryptic crosswords (Minute Cryptic): extremely hard for LLMs as next-token predictors - Requires decomposing sentences into definition, fodder, indicator, and surface - Only one of five major labs solved the example; others failed - Pelican on a bicycle: despite high popularity, models don’t score especially well on it - Popularity doesn’t guarantee saturation for taste-based or out-of-distribution evals - Hard to memorize aesthetic preference, unlike factual answers 4. Rubik’s Cube Eval - Task: solve a cube from a scrambled state, starting with just 2 moves from solved - Gemini 3.1 Pro: solved in 4 moves (slightly off); Groq: 14 moves; others reasonable 0 Fully random cube: Gemini gave up after 7 moves, claiming it couldn’t read the imageUseful secondary signal: model laziness or willingness to persist - Argon solved the random cube in 1,600 moves(!?)(beginners: ~100; God’s number: 26) - Passed the task but missed a final swap step (OLL), thinking it was done - Reward can hide failure: passing score masked incomplete execution

Sharing some more notes from the Benchmark Summit by @kaggle for anyone building AI agents or figuring out how to evaluate them. Gems shared by @lukasb, @SalimansRobin (Zapier), @GregKamradt(ARC Prize Foundation), and Shaiyon Hariri (Ramp). 1. Benchmarks as Living Systems - Finding meaningful tasks is increasingly a research problem: where do models still struggle, and which gaps matter? - Continuously retire saturated tasks and introduce new ones. - Ground task design in real user workflows, product failures, and customer feedback. - As agents take on more responsibility, benchmarks need to expand from individual tasks to entire projects and longer-running workflows. 2. Making Hard-to-Measure Skills Verifiable - We may be running out of easily verifiable tasks, rather than verifiable tasks altogether. - Skills like clear communication, hypothesis testing, and research taste are difficult to grade directly. - One approach: create situations where success requires the skill, then measure the downstream outcome. - Example: have one agent communicate what it learned to another under a strict message limit. Evaluate how well the second agent performs using that information. 3. Internal Evals vs. Public Benchmarks - Internal evals support product decisions: model selection, cost, latency, quality, and reliability. - Public benchmarks help researchers understand capability gaps and users understand what they can rely on. - Regression evals ask, “Does this still work?” - Frontier evals ask, “What can’t we do yet, and how do we improve?” - These purposes can overlap when benchmark design closely matches the product’s goals. 4. Grading Is Harder Than It Looks - Even coding tasks can have flawed tests or incorrect graders. - Human testing helps establish whether tasks are solvable and appropriately difficult. - Capturing expert judgment and product taste in a grader remains a major challenge. - Strong benchmarks need to distinguish between models, which requires careful task selection and validation. 5. Read the Traces - Scores tell you whether an agent succeeded; traces help explain how and why. - Panelists described models mistaking unfamiliar games for familiar ones, while another model developed its own shorthand to plan in a new environment. - Those differences are hard to understand from aggregate scores alone. - Comparing successes and failures can reveal the capability gaps worth targeting next. 6. Benchmark Design Is Product Strategy - What you choose to measure reflects what you want your product to become. - Start with real failures, form a hypothesis about what makes them difficult, and design tasks that test that hypothesis. - Tool access, available information, and the agent’s operating environment also affect performance. - Ultimately, useful benchmarks should help drive better research, more reliable agents, and better outcomes for users.

Benchmark summit by @kaggle was so fun and informative. We live in such a cool time that human potential is rapidly expanding. I particularly enjoyed the panel talk "Evaluating the Frontier" featuring @OfficialLoganK, @OlszewskaKate, Eric Deng, and Tao Jiang. Sharing some notes for ppl building in this area. 1. Evaluation Challenges at Scale - Pace of model development has outrun evaluation infrastructure - Hard to measure progress when models improve faster than benchmarks saturate - Internal bandwidth is a real constraint, even with agents - Internal vs. external evals: both needed, but for different reasons - Internal evals: closer to product use cases, higher quality control - External evals: broader coverage, more credibility (model hasn’t seen them) - Signal quality matters more than volume: measure what you actually care about 2. The Last 20%: Domain-Specific Performance - General reasoning is improving across the board, but vertical depth still requires targeted work - Two buckets of domain-specific gap: - Domain intuition: practices, frameworks, and how knowledge work actually looks in finance vs. legal vs. medicine - Subjective taste: preference-based quality that’s hard to verify, where comparison-based and customer evals become critical - Verification is harder in soft domains (law, finance) than in codingIn coding, many implementations can achieve the same outcome - In legal/finance, the reasoning trace matters: did the model look at the right things, interpret them correctly, combine facts reasonably? - Process quality often matters more than output grading in these domains 3. Engaging with AI Lab and Benchmark Design - To collaborate with labs on benchmarks, three axes matter: (1) Relevance: does it align with real-world use cases and model goals? (2) Eval quality: does it actually measure what it claims to? (3) Practicality: can it run efficiently within compute constraints? - Most valuable signal: something GDM doesn’t already have, showcasing a gap the model hasn’t addressed - Release evals vs. training evals differ in key ways: - Training evals: fast turnaround (hours acceptable, 24h is slow), focused on where the model is moving - Release evals: broader coverage, more realistic, can include product-level scenarios - Benchmarks that don’t yet exist but are wanted: - Long-running agentic task completion (research, robotics, end-to-end science tasks) - Real users in real environments informing how people actually interact with models - Framing reminder: a model card benchmark table is the artifact by which labs tell the world how they’re making progress; design it around your actual goals, not generic coverage

Stopped by for an office hour #sftechweek

Had a conversation with someone form Anthropic yesterday about dynamic evaluation rubrics. I've been thinking about it a lot since so summarized my thoughts at the moment. Dynamic rubrics tailor criteria to the task, the instance, or the set of candidate responses, instead of applying one fixed checklist everywhere. I thought about it while working on a project to evaluate LLM's performance on clinical note generation as the definition of a good note for the particular case is unknown until the case is presented right in front of you. 1. Where clinical note evaluation standsEvaluation so far has been mostly static, meaning criteria were fixed in advance and applied unchanged. The approaches fall into a rough progression: - Overlap metrics against a reference note. These are cheap and static, and they tend to track physician preferences poorly and penalize harmless stylistic differences. - Fixed physician instruments (PDQI-9, PDSQI-9). The same few Likert items apply to every note. Reported inter-rater agreement for the PDQI-9 in an emergency department was near zero, which suggests generic instruments can measure different constructs in different clinical settings. - Error taxonomies. One framework found about 1.47% of AI clinical summaries contained hallucinations, with 44% of those potentially affecting care. It is valuable for safety, but it relies on expert annotation. - Checklists distilled from clinician feedback. One general checklist was built from feedback on over 21,000 encounters. It is interpretable but not per-encounter, rests on proprietary data from one deployed system, and was not intended for clinical deployment. - Case-specific clinician rubrics. Experts wrote criteria per encounter (823 cases). These are instance-specific but frozen once authored. HealthBench follows the same pattern for health conversations. A scoping review found only seven qualifying studies of ambient documentation evaluation and two public datasets. 2. Limitations of existing clinical rubrics Experts disagree about what a good note contains, so even gold standards are noisy. Generic criteria miss case specifics (task-agnostic rubrics covered about 4% of physician key points on a medical dialogue benchmark). Expert authoring is costly, roughly 18 minutes and about $29.50 per rubric. Scores saturate as notes improve, and rubrics stop separating them. The most rigorous work scores documentation fidelity only, not clinical reasoning or outcomes. Much of the strongest evidence comes from vendors using their own data and products. 3. What dynamic rubric research adds "Dynamic" covers three separate choices: granularity (dataset, task, instance), what the generator sees (prompt only, plus source material, plus candidate responses), and timing (fixed or co-evolving). - Finer-grained, instance-specific rubrics beat generic dimensions. - Training-free generation is competitive with existing methods. A small tuned generator can beat a larger general model at writing rubrics, though the gains are modest. - Response-set-conditioned rubrics work well as training signals, because they keep separating responses once models converge in quality. - In clinical documentation, LLM-written rubrics reached ranking agreement with clinicians comparable to clinician-clinician agreement, at roughly a thousandth of the cost. Part of that is ceiling compression, it measures ranking and not clinical equivalence, and it comes from a single study by the makers of the system evaluated, so it needs independent replication. Unresolved: - Generated rubrics trail human ones by a large margin (about 27 points in one benchmark), and more sampling or refinement doesn't close the gap. They miss implicit requirements. - Rubrics can be exploited. Tailored criteria can be specific about the wrong things and steer answers toward fabrication. - Naive rubrics can leave a judge worse off than no rubric. - Models apply their own rubrics most consistently, so generator and judge coupling is a risk. Medicine, law and finance are mostly untested in the leading papers. 4. Design implications for a note-generation benchmark - Dynamic at authoring, frozen at scoring. Response-set-conditioned rubrics make scores relative to the set, which breaks comparability. Generate per-instance rubrics once, validate, version and freeze them. Refresh in versioned releases with an anchor set to separate rubric drift from model change. - Ground rubrics in source material (transcript, filing, data snapshot) and use deterministic checks wherever possible, reserving LLM judgment for reasoning quality and appropriateness. - Add fabrication and uncertainty criteria, plus veto items for critical errors, so weighted averages can't let fluent output mask a wrong dose or figure. - Use a different model family for generator and judge. - Report score distributions, not just agreement, to catch ceiling compression. 5. Expert audit at the start Having experts check a fixed set of generated rubrics is sound if it measures the generator's reliability, not each rubric's soundness. - Have experts write rubrics blind for part of the sample, so you measure recall and not just approval. - Sample by stratum, oversample high-risk cases, and grade failures by severity. - Report an interval. With zero critical failures in n audits, the 95% upper bound is roughly 3/n. - Test rubrics in use, by checking that they reproduce expert best-versus-worst orderings and catch adversarial notes. - Use the corrections as retrieved examples or tuning data, with a holdout kept for honest measurement. - Fix the pass criteria before seeing results, set re-audit triggers (generator change, new domain, strong new models), and review all safety-gating items. - Have at least two experts review part of the sample, and report their agreement.

NYからSFに帰ってきて、この街の人たちがいかに良くも悪くも楽観的で、それに関しても含めて自己認識力が高いかに驚かされる。こちらの天気と、文化的に東洋に近いことが影響してるのかな。

A must read for people following AI in healthcare. @narges_razavian, @EricTopol, and team just published a critical breakdown in NEJM AI. They dicussed why innovative tools get stuck in a "10-year lag" between FDA approval and actual patient access. Key takeaways for founders building in health ai and clinicians: - The "Catch-22": You can't get payment without widespread adoption, but you can't get adoption without payment. - The API Problem: Patients own their data by law, yet vendors still charge for API access or rely on CDs/DVDs. - The Fix: We need dynamic, tiered pricing models that reflect the reality of software (R&D + compute), not static device pricing. I particularly liked the part where they discussed the issues about the pricing of and payment for AI products and proposed potential ways to fix them. I'd argue that the adoption of AI on the regulatory side itself seems essential for these reforms to actually happen. What do you think?

Outpatient note quality got worse: longer and more redundant. Clinical documentation is essential part of clinical care, yet worst part for many clinicians. By keeping record of what's happening with each patient, we can deliver better care more effectively and efficiently. However, as the purpose of documentation shifted from clinical care to billing codes, the note quality deteriorated. Nobody wants to be chased by a coder during your offtime to add a paragraph to justify billing critical care time. One study of 2.7M notes found a 60% increase in length and almost 60% of the text was copied or templated. Only about 30% was actually typed. This is why documentation feels useless and just a burden for care these days. We’re drowning in pasted text while the real conversation with the patient stays outside the chart. By bringing AI into documentation, it would pull signal from the session and help clinicians write cleaner, shorter, more human notes that still meet what payers expect so they actually get reimbursed for the work they did. If patients ask for evaluation document, you don't have to dig into dozens of progress notes and synthesize into a report. I would love to see AI augmented documentation become the standard of care so we can produce high qualtiy documentation with less time. What do you think? pmc.ncbi.nlm.nih.gov/article…

My old neighbors in Akihabara would be thrilled to see this.

People keep asking if cannabis or psychedelics can treat OCD. This new open access review looks at all the data. What it actually shows: • About 40–60% of people with OCD stay symptomatic even after SSRIs and CBT/ERP. • Cannabinoids: evidence is tiny and weak. Mostly surveys and very small studies. The only placebo controlled trial found no real effect on obsessions or compulsions and more side effects with THC. • Psychedelics: psilocybin has a real but early signal in treatment resistant OCD. Case reports and small trials show rapid drops in YBOCS scores, sometimes lasting days to weeks or more, but sample sizes are under 20, controls are limited, and follow up is short. My take: Right now the data do not support cannabis as an OCD treatment. Psilocybin is “promising but experimental” for severe, treatment resistant OCD and belongs in well run clinical trials, not DIY self treatment. Personally I don't think cannabis and OCD mix very well. I'm optimistic that data would soon support that psilocybin being helpful for OCD. Not in this study but with ketamine thearpy, I have seen mixed but generally positive results when paried with psychotherapy session led by a skilled therapist. sciencedirect.com/science/ar…

Most people think depression shows up in what you say. This study shows AI can catch it in your voice . Researchers had 101 people talk with fully automated virtual humans in casual, emotion-eliciting conversations. It's not clinical interview but just normal speech with avatars showing different emotions. Then they ran the audio through modern speech models. What they found: • Voice alone predicted elevated PHQ-9 depressive symptoms with an F1 of 0.566. • Combining voice + text increased this to 0.648. • Text alone did very little. In everyday conversation, people don’t verbalize depression directly. It shows up in prosody. • Prosody = pitch variation, loudness, timing, micro-pauses, and effort. They are subtle, involuntary, and hard to fake. Why this matters: Depression alters the physics of speech. It manifests as slower initiation, flatter pitch, lower energy, disrupted turn taking. These are autonomic and motor patterns. You may change what you And to be clear about the scope, this isn’t voice-based diagnosis. It’s voice-based prediction of a screening tool (PHQ-9) so effectively a screener for a screener. I'd love to see these models evaluated against clinician diagnosis and clinical severity, not just PHQ-9. If we can make screening passive, ambient, and accessible through normal conversation, we lower a barrier that stops a lot of people from getting help. What do you think is the best way to leverate passive voice assesment as a screening tool in the real world? sciencedirect.com/science/ar… #HealthcareAI

EEG may tell you if your depression respond to ketamine or not - Retrospective look at 43 patients, checking if frontal theta-band measures predict response. - Responders (per self/rater scales after 10–12 sessions) had higher frontal connectivity, lower entropy/complexity, and decreased aperiodic values vs non-responders. - These predict effects decently (ROC 0.7–0.73), but aperiodic stuff didn't forecast upfront. Some people clearly respond better to ketamine than others. Would be great if we can tell before they start treatment. sciencedirect.com/science/ar…

A lack of objective measurement in psychiatry is the root cause for many issues in mental health care. I hope this funding would be put into efforts that actually advance it in real world rather than into research for the sake of research.

Japan starts DOGE led by @satsukikatayama next week.