We build the multilingual layer for English-first AI. Custom evals, benchmarks, and RL environments across 200+ languages. #BeyondTranslation
San Francisco
Joined April 2026
- Tweets51
- Following18
- Followers16
- Likes0
2/ Newer isn't always better: GPT-6 Sol regresses vs GPT-5.6 Sol on nearly every language.
Only Arabic ticks up (+0.1pp). English drops 4.4pp.
3/ Open weights: Kimi-K3 beats Opus 4.6 in Arabic and German. Command A+ trails the field, except in Arabic, where it beats GPT-5.2.
aurora.lilt.com/
1/
AURORA deep dive: Multilingual τ³-bench. Multi-turn customer support with tool use (airline, retail, telecom, banking) in German, Korean, Turkish and Persian.
Claude Opus 5.5 leads GPT-6 Sol by ~9pp, at $0.28 per task vs $0.59 for Opus 5.
3/
Both Opus models sometimes drift out of the user's language mid-conversation. Opus 5.5 does it 80% less than Opus 5.
4/
Cost per task: GPT-6 Sol $0.16 (vs $0.26 for GPT-5.6 Sol). Gemini 3.8 Flash matches Opus 5 at under 1/3 the price.
aurora.lilt.com/
1/
AURORA - Multilingual AI leaderboard deep dive: Multilingual Terminal-Bench. 324 coding tasks across 10 languages, designed and verified by native-language domain experts representing real challenges in software written for native-language users.
Claude Opus 5.5 takes #1, ~3pp ahead of Opus 5, with the biggest gains in Chinese and Korean. It's also ~45% cheaper per task: 20% lower price, ~30% fewer tokens.
3/
Gemini 3.8 Flash edges GPT-6 Sol by 2.5pp.
But median turns per task: 41 for Gemini, 5 for GPT-6 Sol, 6 for Opus 5.5. Per task, Gemini costs 7× GPT-6 Sol.
4/
OpenAI's last three models all land within 2pp of each other on multilingual coding. GPT-6 Sol's gain is price: half the cost of its predecessor.
5/ [optional: release-date scatter]
Open weights: GLM 5.3 Flash (fp8) beats GPT-5.4 by 19.3pp at 8% of the cost, and is closing in on Opus 4.7.
Leaderboard: aurora.lilt.com/
On the board:
➤ Multilingual Terminal-Bench: 324 coding tasks , 10 languages representing challenges in software written for native-language users
➤ Multilingual τ³-bench:airline, retail, telecom, banking customer support
➤ MultiChallenge: long-context instruction following
➤ Multilingual GAIA-v2-LILT: agentic tool use
Every real-world enterprise task designed and verified by native-speaking domain experts.
Harness, trial counts, reasoning settings and 95% CIs are published for every run.
Which model or language should we add next?
aurora.lilt.com/
1/ New benchmark: Terminal-Bench-LILT. 300 coding tasks, 10 languages, written by native-speaking authors, deterministic verifiers.
Best frontier model (GPT-5.5) resolves 63.1%. Many tasks: solved by zero of seven models. 🧵
8/ Composition: 133 easy, 86 medium, 81 hard, calibrated from 30 trial outcomes per task across all 7 models. Categories skew cultural (129) and text conversion (76) over pure i18n (52).
Open question we'd take help on: how much of the remaining 37% is tokenizer-level.
9/ Paper: arxiv.org/pdf/2608.28641 Dataset: github.com/lilt/terminal-ben… Write-up: lilt.com/blog/terminal-bench…
Full dataset is available on request.
Built with a distributed team of native-speaking authors, screened for CS/engineering backgrounds. Fewer than 10% of qualified candidates passed our LLM-breaking assessment.