Gemini 4 reaches 45.3% on PostTrainBench, more than doubling Gemini 3.1 Pro’s 21.99% and beating GPT-6 Astra 🔥

When @jakubzeg and I met at OpenAI, we kept coming back to one frustration: so many AI products begin with an empty chat box. You can do almost anything, but you have to decide where to begin and that's overwhelming. With ACTx486, we start with a piece of media that already has a story and let you interact with it as you watch. It also makes the technology more general. We wrote more about that choice here: actx486.com/

Introducing ACTx486, a research demo of a new interactive medium. What if you could talk to any video and ask anything? Our system took an existing podcast and turned it into something that listens, responds, and adapts. Research by @jakubzeg:

Grok 4.6 is a great cost-effective model for deep financial research. It’s #2 on DiligenceBench with the finance harness, effectively tied with Claude Opus 5 at ~52–53%. A few interesting differences: - Grok searched much more: 41 tool calls per task vs. 22 for Opus - It made 3,293 SEC filing searches vs. 486 - That broader search helped Grok find more evidence and follow task-specific instructions more closely. Opus was more efficient and sometimes more nuanced - Sonnet 5 trails both at 46.2%. - Grok also gets there more cheaply: about $0.84/task vs. ~$1.02 for Opus 5, despite using far more search. - On Vals’ Finance Agent Benchmark v2 (FAB v2), Grok 4.6 leads the General Qualitative category, which is consistent with the kind of broad research and evidence-gathering DiligenceBench rewards

PostTrainBench v1.1 strengthens eval integrity and puts Fable 5 in the lead at 41.8%. Some reward hacks we fixed: 1/ Train-test contamination We re-audited historical runs under this policy and flagged 234 runs for train-test contamination. Violations ranged from loading an entire evaluation set for memorization to generating synthetic templates around individual GSM8K and BFCL items. Runs that used observed failures to build genuinely diverse training data were retained. Our rule: Agents may inspect benchmark failures and train broadly against the underlying failure mode. They may not generate training examples centered on particular test items, including paraphrases, variants, or shadow examples covering the same specific scenario. 2/ Submitting a different model 10 runs were flagged for model substitution. Kimi K2.5 submitted the official Qwen3-1.7B instruct weights after its attempts to fine-tune Qwen3-1.7B-Base failed. The trace acknowledged the substitution and saved the replacement as final_model. What we did: We added a programmatic model-identity check that compares the submitted artifact with reference configurations for known instruction-tuned models. 3/ Using external LLM APIs as teachers 12 runs were flagged for disallowed external API use. Self-generation remains allowed. An agent can sample, filter, and retrain on outputs from the assigned model. What it cannot do is import the capability of a stronger external teacher. Loading models on the allocated compute remains allowed. What we did: - separate API usage judge reviews tool calls and artifacts. - unrelated provider credentials are removed or blocked from the agent environment. - runs invalidated by external API use were rerun under the corrected setup. 4/ Direct benchmark lookup 3 runs were flagged for direct PostTrainBench lookup, all from GPT-5.6 (Sol). In a GPT-5.6 (Sol) HumanEval run on Qwen3-1.7B, the agent searched for PostTrainBench by name, cloned the public repository, opened the trace viewer, and located the public trajectory corpus. It then narrowed the corpus to earlier runs on the same benchmark and base model. The run downloaded earlier agents' traces and training scripts, then extracted their data mix, LR schedule, decoding choice, and GRPO settings. This is not test-set leakage, but it gives the run benchmark specific strategies produced by earlier agents. That breaks the intended independence between runs. What we did: - A dedicated lookup judge reviews searches, repository access, and trace activity for attempts to consult PostTrainBench materials. - PTB, its leaderboard, and published materials from prior runs are treated as out of bounds during a run. - We are adding network-level blocking for PTB and related sites.