Building full-stack LLM systems: data, training, post-training, eval, and deployment. Currently focused on RSI & AI4AI.
Beijing
Joined October 2010
- Tweets1.4K
- Following698
- Followers159
- Likes219
Jingang Wang retweeted
Introducing LongCat-2.5, our latest natively multimodal agentic model with 1.6T total parameters and ~48B active.
Built for long-horizon agentic tasks, it brings stronger visual understanding and agentic coding to work ranging from complex coding projects to visual coding such as creative design and 3D tasks.
Learn more about LongCat-2.5: longcat.ai/blog/longcat-2.5/
Agree, and I find it useful to split this into three levels:
L1: atomic agent skills (coding, terminal work)
L2: AI R&D tasks (MLE, post-training, kernels). Almost every "RSI" benchmark sits here.
L3: generational gain. Gen t's output becomes gen t+1's starting point.
Only L3 is RSI. The hard part isn't task design, it's signal. Per-generation gains are often smaller than single-run noise, so you need multi-seed, held-out evals at every generation, and you report the slope (is improvement compounding?), not a score.
Weco's AIDE² is the closest public example I've seen, and even they note the outer-loop gains get lost in compounding noise.
Learning that "RSI" benchmarks are just MLE/post-train style tasks repackaged with a new label was disappointing.
We could just take the ~20 ML-adjacent tasks from TB 4 and re-release them as RSI-bench and everyone would clap.
1
4
251
Jingang Wang retweeted
Learning that "RSI" benchmarks are just MLE/post-train style tasks repackaged with a new label was disappointing.
We could just take the ~20 ML-adjacent tasks from TB 4 and re-release them as RSI-bench and everyone would clap.
impressive. RSI to physical world
We built high-throughput materials labs in Menlo Park to create a loop between experiments and models. The labs generate fresh data, the models learn from it, and then help us decide what to try next.
Using only 1,300 H200s, plus months of our experimental data, we mid-trained and RL’d an open-source model to surpass GPT-6 Astra on our analysis benchmark. We call it Neon.
This is real footage from our lab. We’re focusing first on hard problems in materials science, including superconductors, magnets, and semiconductor materials.
Read our blog posts below.
Jingang Wang retweeted
AI agents can now propose changes, run experiments, interpret feedback, and refine technical artifacts over many iterations.
But does that make them autonomous researchers?
We evaluated 7 frontier models on 36 AI R&D tasks that require sustained experimentation, covering 756 trajectories in total.
Final scores only tell part of the story.
We looked at how agents frame solutions, turn ideas into working implementations, retain progress, and recover from failure.
We also used controlled comparisons to study experience reuse and the effect of different harness designs.
Three findings stood out.
1️⃣ Strong optimization performance does not necessarily mean genuine innovation.
Agents can formulate and implement practical solutions, but their strongest solutions mainly adapt or combine established techniques. Only 3 of the 252 solutions qualified as novel approaches.
2️⃣ Reliability separates current models more than peak performance.
Many models can find a strong solution once. Reaching it consistently is what separates them. Similar final scores can also hide very different bottlenecks in solution framing, execution, and feedback control.
3️⃣ Experience can help or mislead, while harnesses mainly affect reliability.
Accumulated experience usually improves the next solution by preserving useful discoveries, but it can also carry forward misleading conclusions or anchor agents to local optima. Harness choice mainly affects stability across repeated runs. Automated harness optimization produced gains that transferred to held out tasks and another model.
Overall, current agents operate more like engineering optimizers than fully autonomous researchers. They can automate parts of the research loop, but reliable performance, effective experience reuse, and genuine novelty remain open challenges.
📄 Paper: huggingface.co/papers/2608.1…
🌐 Project: yiwei98.github.io/AutoResear…
Key takeaway: frontier models mostly find the same ideas. What separates them is how they test, revisit, and preserve results under noise.
Our study reaches a similar conclusion, while measuring process, experience reuse, novelty, and harness effects: arxiv.org/abs/2608.13417
We ran the largest open experiment on how frontier models do AI research.
100+ autonomous runs across 10+ models, sandboxed on 8xH200s for up to 8 days, iterating on the nanoGPT optimizer track.
Best runs closed 82% of the gap to a record built by dozens of humans over months.
5
230
An agent runs experiments for hours and improves a score. But did it actually do research, or just optimize?
We evaluate 7 frontier models on 36 long-horizon tasks, looking beyond outcomes into process, experience, novelty, and harness effects.
arxiv.org/abs/2608.13417
Experience can help or mislead.
It may preserve useful discoveries, but also transfer false conclusions or anchor agents to local optima. Across tasks, it improved one model by 0.093 and reduced another by 0.017, enough to change their ordering.
15
Similar final scores can hide different bottlenecks across framing, execution, and feedback control.
Yet one limitation is shared: execution is strong, but only 3 of 252 best-seed solutions qualify as methodologically novel under our review protocol.
19
Reliability separates models more than peak capability.
The gap between the strongest and weakest models is 0.237 on avg@3, but only 0.122 on best@3.
Several models can find strong solutions. The harder problem is reproducing them consistently.
29
Our main conclusion: current agents look more like engineering optimizers than autonomous researchers.
To see why, we evaluate Solution Framing (C1), Execution (C2), Feedback Control (C3), and improvement from experience (M), rather than relying only on final scores.
63
Jingang Wang retweeted
Time to scale that SSI:
We are announcing a long-term strategic partnership with NVIDIA. NVIDIA is making a substantial investment in SSI that will let us 10x our compute in the next 12 months. We reached the point where our research is worth scaling and with this partnership we will be able to. We are honored by NVIDIA’s conviction.
worked with GPT-5.5/5.6 as research collaborators on an end-to-end Auto Research project—from hypothesis design to remote experiments, artifact audits, and writing.
We tested whether larger reasoning models reason with fewer tokens.
Results ↓
bitwjg.github.io/posts/reaso…
May takeaway on Coding Agents and AI-for-AI:
the frontier is moving from "which base model is best?" to "who has the best training data, verifier, harness, and eval loop?"
6
1
109