carlos (@closji) header
carlos (@closji) profile picture

i like ai, philosophy, and politics

San Francisco, CA
Joined May 2019
Photo by carlos (@closji)Photo by carlos (@closji)Photo by carlos (@closji)Photo by carlos (@closji)Photo by carlos (@closji)Photo by carlos (@closji)Photo by carlos (@closji)Photo by carlos (@closji)Photo by carlos (@closji)Photo by carlos (@closji)
I completed my postdoc at Princeton University 9 months ago but never wrote a goodbye post so: My two years at Princeton were the best 2 years of my life. I'm forever grateful to @karthik_r_n, @prfsanjeevarora & Ellen DiPippo for creating the incredible environment that enabled me to be take part in amazing research. I'm forever grateful to @closji & @jyangballin for meeting me on my fourth day on the job and then inviting me to join the software engineering benchmark they had developed. And to @KLieret who joined us just after and became a core part of the SWE-team. Thank you @ShunyuYao12 who met with me when I was making the decision; I enjoyed our walks so much I decided to join just so I could go on more walks with Shunyu. Thank you @_awettig my collaborator in both science and art, and European<->American linker. Thank you to my FAIR postdoc hosts @syhw and @sidawxyz. Thank you to *all* of my collaborators, including first authors: @MinyangTian1 SciCode and CritPt, @AbramovichTalor SWE-agent for Cyber Security, @OriYoran AssistantBench, @18jeffreyma SWE-fficiency, @a1zhang VideoGameBench, @MiniHui_zhu CritPt, and @ori_press my actual brother and author of AlgoTune Thank you to my senior collaborators: @Diyi_Yang & @lschmidt3 (through John), @haopeng_uiuc (through Minyang), @profvjreddi (through Jeff), @JonathanBerant (through Ori Yoran) and @MatthiasBethge (through Ori Press). Thank you to all the people at Princeton who made PLI what it is: @danqi_chen @cocosci_lab and all of the other faculty, postdocs, graduate and undergraduate students.
18
19
1
342
29,652
Replying to @princeton_nlp
@princeton_nlp graduated 11 (!) PhDs at the hooding ceremony yesterday. Their research has fundamentally shaped the global AI landscape in so many ways, and they have been a core part of building the NLP group and its collegial spirit. It's been a real privilege working with and getting to know them all over the past few years! @ZexuanZhong @AmeetDeshpande_ @SadhikaMalladi @_carlosejimenez @JensTuyls @VishvakM @__howardchen @xiamengzhou @gaotianyu1350 @danfriedman0 @_awettig w/ @danqi_chen @prfsanjeevarora
7
18
3
187
27,370
@_carlosejimenez the perfectionism that went into the swe llama has finally paid off
1
7
689
Doesn’t get much clearer than this
2,201
15,050
2,581
49,023
5,152,189
Updates to official SWE-bench leaderboard: GPT 5.2 high is #3 after Gemini for similar price point. GPT 5.2 medium reduces gap to Sonnet 4.5 and is much cheaper. All independently evaluated using identical mini-swe-agent setup. More info & full trajectories in 🧵
18
51
12
567
79,042
Presenting this Thursday Morning at #NeurIPS2025, reach out if you wanna chat and become friend 🙂
As we optimize model reasoning over verifiable objectives, how does this affect human understanding of said reasoning to achieve superior collaborative outcomes? In our new preprint, we investigate human-centric model reasoning for knowledge transfer 🧵:
1
11
1,159
Come see the SWE smith poster at Neurips, hall E #113 !
At NeurIPS! DMs are open Excited to reconnect + make new friends; Let's talk about LM evaluation, long running SWE-agents, and human-AI collaboration. Where you can find me 👇 More info in 🧵
8
1,241
Updates to the official SWE-bench leaderboard: Minimax M2 is best open source model (but expensive!). Deepseek v3.2 reasoning close behind, very cheap, but very slow. GLM 4.6 reaches good performance fast and cheap. All independently evaluated with mini-swe-agent. Details in 🧵
13
19
4
194
118,917
If you're curious to read how Opus 4.5 solves its tasks, we've just updated the leaderboard on swebench .com including links to full transcripts of the agent
Opus 4.5 reclaims the top of the official SWE-bench leaderboard with 74.4%, narrowly ahead of Gemini 3. Cheaper than Opus 4, but more expensive than Gemini. Takes less steps than Sonnet 4.5, but still run for >100 steps for optimal performance. Details in 🧵
1
6
1,175
Opus 4.5 reclaims the top of the official SWE-bench leaderboard with 74.4%, narrowly ahead of Gemini 3. Cheaper than Opus 4, but more expensive than Gemini. Takes less steps than Sonnet 4.5, but still run for >100 steps for optimal performance. Details in 🧵
6
8
2
85
16,794
New eval! Code duels for LMs ⚔️ Current evals test LMs on *tasks*: "fix this bug," "write a test" But we code to achieve *goals*: maximize revenue, cut costs, win users Meet CodeClash: LMs compete via their codebases across multi-round tournaments to achieve high-level goals
31
97
23
416
106,036
IMPersona from @BenShi34 et al #COLM2025 They trained LMs on participants' real chat logs. Then they brought in *each participant's friend* to do a personalized Turing test. The chatbot was surprisingly effective, passing as the friend 44% of the time. LMs can impersonate YOU🫵
1
5
2
28
8,686
One of my fave papers from COLM by @BenShi34! Really cool eval too (a friends and family Turing test!)
IMPersona from @BenShi34 et al #COLM2025 They trained LMs on participants' real chat logs. Then they brought in *each participant's friend* to do a personalized Turing test. The chatbot was surprisingly effective, passing as the friend 44% of the time. LMs can impersonate YOU🫵
1
3
33
5,929
We evaluated Anthropic's Sonnet 4.5 with our minimal agent. New record on SWE-bench verified: 70.6%! Same price/token as Sonnet 4, but takes more steps, ending up being more expensive. Cost analysis details & link to full trajectories in 🧵
3
13
6
83
53,198
Accepted to #NeurIPS2025! Big shoutout to our ~120 participants, who graciously allowed me to pester them daily with reminder emails, bug fixes, and troubleshooting queries 😓
As we optimize model reasoning over verifiable objectives, how does this affect human understanding of said reasoning to achieve superior collaborative outcomes? In our new preprint, we investigate human-centric model reasoning for knowledge transfer 🧵:
1
14
865
Deepseek v3.1 chat scores 53.8% on SWE-bench verified with mini-SWE-agent. Tends to take more steps to solve problems than others (flattens out after some 125 steps). As a result effective cost is somewhere near GPT-5 mini. Details in 🧵
8
22
4
153
24,066
What if your agent uses a different LM at every turn? We let mini-SWE-agent randomly switch between GPT-5 and Sonnet 4 and it scored higher on SWE-bench than with either model separately. Read more in the SWE-bench blog 🧵
18
20
13
268
31,917