
Building Agents services in trading market | ex-PM in Security & Wallet & L1. Focus on @questflow
Mars
Joined July 2009
- Tweets4.8K
- Following5.8K
- Followers1K
- Likes2K
Same accuracy, 10x the tokens. Log harness changes like code changes.
On our finance exam, every harness change is recorded with a reason and affected runs are re-run. Otherwise you can't tell if the model got better or the wrapper did.
teapawt.net/rohanpaul_ai/status/21…
New Google paper finds that an unchecked full rewrite of an agent harness matched accuracy but cost nearly 10 times more tokens per task than gated modular edits.
that self-improving agent harnesses work best when they change 1 code module at a time and must pass validation.
The harness is the code around a model that manages context, runs tools, and checks work before stopping. A harness tuned for 1 model often stalls or wastes tokens on another.
They let an optimizer model improve a 142-line coding harness by editing 1 of its 4 parts at a time. Each edit had to pass a validation set and a token cap, or it was rejected.
On 40 held-out SWE-bench Lite tasks with Gemini 3.8 Flash, the gated harness solved 40.0% against 5.0% for the seed, using 46.7k tokens per task. A full unchecked rewrite solved 42.5% but used 451.6k tokens, and prompt rules dropped accuracy to 2.5%.
If you let agents tune their harness, gate each change on held-out tasks and token cost, and enforce rules in code, not prompts.
27
A record $13.1B into $SMH this year, and the Nikkei fell 4% today as chip stocks extended their drop.
My read: the chips are fine. The risk is everyone owning the same story at once. I'd watch who pays for the compute, not who sells it.
teapawt.net/KobeissiLetter/status/…
BREAKING: The US semiconductor ETF, $SMH, has attracted +$13.1 billion in inflows year-to-date, on track for its largest annual intake on record.
This surpasses its previous annual record of +$7.9 billion set in 2023.
This is also +368% above the +$2.8 billion recorded in 2025.
Most of this year's inflows have come since July, with inflows of +$7.8 billion attracted over that period.
This month alone, $SMH has attracted +$2.2 billion, putting the fund on track for its largest monthly inflow since the +$3.2 billion recorded in July.
Investors are piling into US semiconductor stocks at an unprecedented pace.
38
Agree. In a few years the customer's own agent will be the main user of most apps.
Blocking it is the new "please call our hotline." Give agents scoped, logged access instead. Founders who build for that now will win the switch.
teapawt.net/emollick/status/212354…
I suspect companies will end up frustrating a lot of customers by telling them they can’t use their own AI agents to help with their own accounts or to manage their own computer. Scaring them will only work so well for so long.
39
Uber's weekly agent requests grew 9.4x since February. AI spend has stayed roughly flat since April. It now cuts coding sessions at 400K tokens.
Budget is the limit now, not the model.
On our finance exam, 8 models get 94%+ of user tasks right. Cost per task: free to $0.905.
1
1
77
"Usable in the real world" is the right question, and a leaderboard can't answer it for you.
On our finance exam, every new model joins the same frozen data and task set, graded by code. So one run tells you if it's usable for finance work.
teapawt.net/bindureddy/status/2123…
Google is on a tear! 4th model - Gemini Neon just dropped in less than 4 weeks
The real question - are any of them usable in the real world? 😄
Hopeful about Neon! Let's hope it's faster than Carbon and more usable.
31
$83 a share, above the $70 to $78 range. ANT starts trading on Nasdaq tonight, Asia time.
My read: public markets now get to price a frontier lab directly. Every API price cut will show up in a quarterly margin. That should make the price war more honest.
teapawt.net/AnthropicAI/status/212…
We’ve priced our IPO at $83 per share, with trading expected to begin on Nasdaq tomorrow.
anthropic.com/news/anthropic…
Legal disclaimer: A registration statement relating to the Class A common stock was declared effective by the Securities and Exchange Commission on November 18, 2026. The offering is being made only by means of a prospectus. Copies of the final prospectus may be obtained, when available, from: Goldman Sachs & Co. LLC, Attention: Prospectus Department, 200 West Street, New York, NY 10282, by telephone at (866) 471-2526, or by email at [email protected]; or Morgan Stanley & Co. LLC, Attention: Prospectus Department, 180 Varick Street, 2nd Floor, New York, NY 10014, or by email at [email protected]
50
Four Gemini 4 models in seven weeks. Launches now come faster than most teams can re-test.
My rule: keep a frozen set of your own real tasks and re-run every new model in one afternoon. If you can't, you're picking models off someone else's chart.
teapawt.net/ArtificialAnlys/status…
Google continues to accelerate its release cadence with Gemini 4 Neon, delivering a small Intelligence gain over Gemini 4 Carbon at a ~17% lower Cost per Task
@GoogleDeepMind has released Gemini 4 Neon, its fourth model in just seven weeks. Neon (xhigh) scores 61 on the Artificial Analysis Intelligence Index, 1 point ahead of Gemini 4 Carbon (high) and 3rd overall, behind GPT-6.2 Astra (max) and Claude Fable 5.2 (max). At high effort, Neon matches Carbon (high) at less than half the Cost per Task using Carbon’s standard (non-discounted) pricing.
Key takeaways:
➤ Slight improvement in Intelligence for lower Cost and Time per Task: Neon (xhigh) scores 61 on the Intelligence Index, 1 point ahead of Carbon (high), with small improvements in agentic knowledge work, visual reasoning and coding. It costs $6.44 per task, ~17% less than Carbon ($7.76 at non-discounted pricing), and takes ~21 minutes per task, ~40% less than Carbon (~36 minutes). Neon uses ~15% more output tokens per task, so the lower cost comes purely from its 25% lower token prices ($3/$15 vs $4/$20 per 1M input/output tokens), and the lower time from its higher output speed (~110 vs ~57 output tokens per second)
➤ Competitive with Claude Opus 5.5 and Claude Fable 5.2 on coding agents: With Antigravity CLI, Neon (xhigh) scores 59 on the Artificial Analysis Coding Agent Index, 3 points ahead of Carbon (high, 55). It scores in line with Claude Code with Claude Fable 5.2 (xhigh, 58) and Claude Opus 5.5 (max, 58) at a much lower Cost per Task. Gemini 4 Neon in Antigravity has an average Cost per Task of $6.86, compared to $12.61 for Claude Opus 5.5 (max) and $15.47 for Claude Fable 5.2 (xhigh). Neon is available in Antigravity CLI for Pro and Ultra subscribers
➤ Significant improvements in visual reasoning and output presentation quality: In GDP.pdf, Gemini 4 Neon (high) scores 33.4%, 5 points above Carbon (high, 28.4%) and in line with the highest scores measured, while Neon (xhigh) scores 28.0%. On AA-Briefcase, Neon (xhigh) scores 1700 Elo overall and leads on Presentation Quality with 1771 Elo, 45 points ahead of the previous leader Claude Sonnet 5.5 (max, 1726). On Google’s Multimodal Knowledge Agent evaluation (MKA), Neon reaches a Climb WSR of 56%, up 2 points from Carbon and on par with the leader, Claude Fable 5.2 (max, 57%)
Key model details:
➤ Context Window: 1M tokens
➤ Multimodality: Text, image, video, and speech input, with text output
➤ Availability: Gemini 4 Neon is publicly available at launch through Google’s API. Pro and Ultra subscribers can also access it in Antigravity and the Gemini app
➤ Effort levels: Gemini 4 Neon offers four effort levels (low, medium, high, and xhigh) - xhigh was introduced to Gemini models in the recent Hydrogen release
➤ Pricing: $3/$15 per 1M input/output tokens, 50% above Gemini 4 Carbon’s currently discounted price of $2/$10 (which ends on November 30), and 25% below its standard $4/$20. Cached input tokens receive the same 95% discount ($0.15)
40
OpenAI's new data: agents finish 46% of 2-4 hour research tasks solo, vs 23% in March. Median researcher: $2,400 of agents a day.
Agent spend is a line item now. So ask what a right answer costs.
On our finance exam, four models get 97%+ of user tasks right under $0.13 each.
24
Per-category scores beat one leaderboard number.
Same on our finance exam: the top six all score 97%+, but split by pillar the order shifts. Three hit 100% on financial analysis, three hit 99%+ on market and signal.
Pick the model for the job.
teapawt.net/jerryjliu0/status/2122…
We’ve benchmarked over 20+ open-weight models on ParseBench-2 🔥
The leaderboard is fully public and on @huggingface.
Perhaps even more importantly, it’s important to know the specifics of where open-weight models do better than others: across tables, charts, formatting, and more. The full breakdown is shown on ParseBench-2: parsebench.ai/
We’re rapidly benchmarking more models and shipping iterations on the benchmark itself. We’ve hit ~30k+ monthly downloads in the past few days.
Hugging Face: huggingface.co/datasets/llam…
Leaderboard: parsebench.ai/
38
Forecast beat, CEO drama, rest of tech still sliding. Nvidia added ~$150B anyway.
My read: the market can live without Jensen. It can't live without data center orders, and the interim CEO says hyperscalers still want more compute.
teapawt.net/Kalshi/status/21228404…
52
Ultrafast is the one I'd test first. On our finance exam an agent makes 7 to 15 tool calls per task, so every second of latency gets paid many times over.
Accuracy wins the demo. Speed decides if people keep using it.
teapawt.net/sama/status/2122798039…
6.2 Sol and Astra are out.
These models can do amazing things and are very token-efficient; we hope people will get great value from them.
In particular, 6.2 Sol ultrafast is something I think people will really enjoy.
38
Every launch now leads with "token-efficient." Good. For finance work that's the right fight.
On our finance exam, seven models get 94.9%+ of user tasks exactly right. Cost per task: $0 to $0.905.
When accuracy is close to a tie, price and speed decide who gets used.
40
Every benchmark saturates. On our finance exam, six models already get 97%+ exactly right.
So the next version needs tasks that look like real work: restated filings, a hidden extra condition, metrics you rebuild yourself. Easy questions stop teaching.
teapawt.net/scaling01/status/21224…
40
Worst two months for chip stocks since 2008, while new AI data center spending keeps getting announced.
To me this looks like the market pricing governance and export-rule risk, not weaker demand. Those risks can fade fast. A demand problem would not.
teapawt.net/Barchart/status/212249…
Semiconductor Stocks are on track for their worst 2-month period since the Global Financial Crisis 🚨 🚨
34
$1 for 1,000 runs. Watching the web for changes is now basically free.
The expensive part moves to deciding which change matters. For finance I'd point it at filings and earnings pages, then have an agent check the numbers before anyone acts on them.
teapawt.net/AravSrinivas/status/21…
Announcing Perplexity Monitor: an API that lets you track any event or topic or changes to webpages and set up webhooks to update you. Costs a $1 for 1000 runs.
44
A new model's first number usually comes from the lab's own test. Not lying, just not proof.
So we built a finance exam: same frozen data, same tools, answers graded by code. Six models now get 97%+ exactly right.
Independent tests are boring to build. That's why so few exist.
37
Physics is a great test for AI because the answer either holds up or it doesn't.
Finance works the same way: a ratio is right or wrong. I'd love to see more AI results where the output gets checked by code or by experts, not by vibes.
teapawt.net/gdb/status/21221260106…
41
Days after stepping down as CEO, he's reportedly talking comeback. That tells you who the market thinks owns Nvidia's roadmap.
If he returns, the price probably snaps back first. The governance questions take a lot longer to settle.
teapawt.net/unusual_whales/status/…
41
Best seed for an eval: a task that already fooled you.
One of ours: rank six companies by revenue growth, but use the restated figure from the latest filing. Gold answer computed by code before any model saw it.
That one rule catches a lot.
teapawt.net/Vtrivedy10/status/2122…
easy win for building evals:
agents build way better & more reliable Evals/Benchmarks if you seed them with examples of good/bad design across tasks (Terminal Bench, HLE, Harvey LAB, internal evals, etc)
the design space for a single Eval is huge
- verifier design (rubrics, programmatic vs judge,
- environment fidelity (container settings, realistic tools)
- synthetic data creation
- vagueness vs enough information in task descriptions
agents are amplifiers of their context -> ex: in codebases, they copy all the good/bad patterns that already exist
so we should use this to our advantage by seeding
no benchmark is perfect - but lots of time went into getting stuff right, agents are hopelessly bad at designing from scratch
we embed examples we like into skills which gives agents something concrete to reference
be wary of “overfitting” to existing evals, it’s a delicate balance between stability & diversity
36





























