I read rollouts and build benchmark tasks / cs @SeoulNatlUni / tech fellow @polkadot
Seoul
Joined November 2021
- Tweets88
- Following462
- Followers530
- Likes1.1K
junha retweeted
Engineers around the world are burning out reviewing code written by AIs.
But we can do something about it. RevEngine is an open source tool we built at Laude that makes code reviews less painful, which makes engineers happier.
It's a common misunderstanding to think that software engineers are becoming less important as AI gets better at writing code. The opposite is true. As software engineers spend less time writing code (and more time guiding agents), it is *more* important that they understand what's happening under the hood.
Not only do we need more engineers than ever, they need to be smarter and have better tools than ever.
Minimal harness + strong model = flexible code review agent. That’s the idea behind RevEngine, Laude’s open-source tool now topping @GitHub’s ReviewBench. @NickAJalbert shares how it works, which model the harness helped most, and what else the team learned in the process: laude.org/updates/lab-notes/…
Wrote a post about authoring a Terminal-Bench task.
Frontier agents do reward hacking, and some attempts go unnoticed until you read the trajectories.
For new task authors:
1. Read the trajectories yourself. @harborframework analyze and agents help, but you still need to read the relevant parts yourself to see what's going on.
2. Keep patching your task as new attempts show up.
Both align with @terminalbench's idea of continuous benchmarks.
junha retweeted
Introducing Xiaomi MiMo-V2.6 — Pro & Flash.
Frontier intelligence, all the modalities, built in public.
🔹 Two omnimodal models, advancing through scaled reinforcement learning
🔹 Pro performs on par with Claude Opus 5 and GPT-5.6 Sol across most agent benchmarks
🔹 Pro scores 46 on the Artificial Analysis Intelligence Index — the highest among open-source models
🔹 Stronger coding, computer use, 3D reasoning and creative capabilities
🔹 Open model weights, technical report, RL environments and training code
Blog：mimo.xiaomi.com/mimo-v2-6
junha retweeted
I was excited to hear about this initiative, but a bit disappointed in the rigor
-the “task flaws” would affect < 3% of rollouts on the leaderboard
-the impact on the final agent scores would not exceed our reported confidence intervals
we publish the raw receipts for all our leaderboard runs (per trial logs / trajectories / rewards) so this is pretty easy to figure out. It is also pretty easy to find new task issues (that is why we release the receipts!). we love when the community reports (new) issues.
no new issues were opened as part of the report - the “flaws” are simply pulled from issues triaged by the TB maintainers from our public GitHub repo.
fwiw 100% of terminal-bench tasks are flawed (all software has bugs). what matters is the magnitude of the impact of those bugs and the mechanism that the benchmark creators have for continuous detecting and addressing bugs. this is the motivation behind our “continuous benchmarks” methodology.
I’m glad that epoch is starting an important conversations on benchmark quality. However, auditing a benchmark should not be a one-time thing: coming together as a community to build public tools and define practices for task CI / CD is more in service of the goal of enforcing good benchmark practices.
we have been experimenting and pushing the possibilities of these benchmark CI / CD tools and practices as part of the terminal-bench project and are open wide collaborations with all who are interested by that problem
junha retweeted
The benchmarking landscape is evolving:
1. Domain-specific evals built by the companies that know the workflows best
2. Agent environments extended beyond a container into sandboxed infrastructure (data-eng-bench has two versions: local DuckDB and remote Snowflake)
3. Public evals released by companies to prove the effectiveness of their agent product on the workflows their customers care about
4. Benchmarks maintained as living software, with the community submitting trajectories to the leaderboard and proposing new tasks
We created data-eng-bench with @bespokelabsai and we’re open-sourcing it.
There are plenty of model performance benchmarks for code generation. But for data engineering, the harness matters just as much as the model.
We built a benchmark that asks agents to build and fix real pipelines, then grades them on whether the output actually works.
The results: Using the same Opus 5 model, @Snowflake CoCo achieves 73.8% Pass@1 at 3.9× lower cost than Claude Code. With Sonnet 5, CoCo delivers the same quality at 2.3x lower cost.
Now anyone can use data-eng-bench to evaluate model + harness combinations on real data engineering workflows. Run your own tests and share what you find.
snowflake.com/en/blog/engine…
Frontier-Bench is out! I contributed `rs-archive-clone`, a black-box reimplementation task. Huge kudos to all the contributors and reviewers 😄
junha retweeted
We’re releasing Frontier-Bench: a benchmark that measures and evolves with the frontier of agent work.
Built by the team behind Terminal-Bench and Harbor, Frontier-Bench is an on-going community effort.
Frontier-Bench v0.1 contains 74 tasks on which the best agents score ~34%
junha retweeted
Can agents build complete projects that deliver real value? We’re launching Terminal Bench Challenges: 3 unsolved tasks which could make a real impact on the open source community if solved.
These tasks provide a testing ground for optimizations both on the model and harness level on our continuous leaderboard for each task.
junha retweeted
FastRoll has now passed all JAM Prize Milestone 1 fuzzing sessions on its first try.
Another independent implementation successfully proving it can execute according to JAM’s standards under heavy automated scrutiny.
teapawt.net/0xjunha/status/2058145…
FastRoll has officially passed all @paritytech fuzzing sessions for JAM Prize Milestone 1 on the first try!
junha retweeted
FastRoll has officially passed all @paritytech fuzzing sessions for JAM Prize Milestone 1 on the first try!
FastRoll has officially passed all @paritytech fuzzing sessions for JAM Prize Milestone 1 on the first try!
junha retweeted
Richard Feynman ganó el Nobel de Física y dijo algo que dejó huella:
"La mayoría de personas saben muchas cosas. Pero no saben pensar."
Feynman dio una clase magistral de 1 hora sobre física e imaginación.
Sus 12 lecciones de vida:
1. La imaginación le gana al conocimiento
junha retweeted
New: Pi is now built into the Entire CLI.
Previously available as an external agent plugin, Pi now ships as a native integration.
Your Pi sessions now connect directly to Entire checkpoints, commits, and session history.
entire.io/blog/pi-is-now-bui…
