Conor Finlay retweeted
Cohere Labs at @emnlpmeeting: CALIBER: Calibrating Confidence Before and After Reasoning in Language Models
CALIBER introduces a unified confidence calibration protocol distinguishing pre-reasoning estimates of solving the prompt from post-reasoning estimates of answer correctness, aligning each supervision target with its information state. This state-dependent approach significantly reduces Expected Calibration Error across in- and out-of-distribution benchmarks like BigMathDigits, GPQA, and TriviaQA.
📜 Read the paper: cohere.com/research/papers/c…
Congrats to authors @ConorFinlay3, Joshua Kurien, @TheyCallMeMr_ , @mziizm, @beyzaermis
Conor Finlay retweeted
@emnlpmeeting kicks off tomorrow and we're excited to share some work from @cohere and Cohere Labs at the conference in Budapest! ✈️
Check out our Main Track conference papers here 👇
Explore all of our research: cohere.com/research/papers
I'll be here to present our team's work on confidence estimation in reasoning models, come by our booth!
Cohere Labs and @cohere are proud to share some of our latest research next week at EMNLP 2026, in Budapest, Hungary! 🇭🇺
Very honored to be featured as part of this conference focused on cutting-edge deep learning research and its real-world applications.
Conor Finlay retweeted
Announcing new research on scaling multi-agent teams for long horizon software engineering tasks. 🤖🧩📈
Work led by @keenansamway, @mrdanieldsouza, Omar Darwiche Domingues, @mziizm, @yichern_tan, and @nouhadziri
Read the paper: cohere.link/tStRYJS
Check out the blog post: cohere.link/0TzgKoy
Conor Finlay retweeted
Have been taking different local open-weight LLMs for a test drive in different harnesses (Qwen-Code, Codex, Claude Code).
30B Mixture-of-Expert models are kind of a nice sweet spot and can solve challenging problems. And they get roughly 40 tok/sec on a Mac or DGX Spark, which is similar to GPT 5.5 in a Pro subscription and totally useable for everyday work.
More interesting is also the harness choice! Claude Code seems to be using 2x many tokens as Codex.
Gemma 4 E2B is here just for reference to show that the tasks can't be trivially solved by smaller models.
Just finishing a longer write-up about this and will share soon (likely tomorrow)!
Conor Finlay retweeted
Announcing new research on calibrating confidence in reasoning models 🎯⚖️, led by @ConorFinlay3, @joshuakurien_, @TheyCallMeMr_, @mziizm, and @beyzaermis.
📜Read the paper: arxiv.org/abs/2606.24281
Can an LLM know when not to trust its own answer? 🤔
Reasoning models are powerful but overconfident, and their verbalised confidence is generally unreliable.
We built CALIBER: an RL method that teaches a model to estimate its confidence twice, before and after reasoning.
A nice side effect: eliciting confidence before and after reasoning also shapes how the model reasons. You can watch it start uncertain, work through what it knows, and arrive at a confident answer. 👇
Read the full paper here: arxiv.org/abs/2606.24281
Work with @joshuakurien_, @TheyCallMeMr_, @mziizm, and @beyzaermis.
Conor Finlay retweeted
Excited to triple down on London! 🇬🇧
thetimes.com/business/techno…
