
man in ✨ fínãnce ✨🥴 love to hate nyc 🐀🥀 @sportfoliokings @bettorgpt
d*mes sq**re
Joined February 2021
- Tweets6.1K
- Following319
- Followers3.1K
- Likes34.4K
deepvaluebettor retweeted
Since developers aren’t coding anymore, their new job is to rotate secrets every single fucking day
both of these are true:
1. frontier models are at the cusp of fully automating knowledge work
2. we are nowhere near the reliability and cost-efficiency needed for a lot of enterprise workflows
it's a crazy time and I have a lot of cognitive dissonance about this.
we hope this work shows just how early we are, even on a task as "simple" as document extraction.
super proud of the team for pushing the frontier on synthetic data and evals on this one.
Introducing ExtractBenchPro: the most challenging benchmark for real-world document extraction.
We see a massive long tail of document complexity in terms of use cases brought on from our customers, that are completely unsolved from a price-performance standpoint. These include trial balances, well logs, canvass binders, site plan reviews, duty reconciliations, loss runs, and more.
Most existing extraction benchmarks are saturated by the frontier models (our cheapest modes now score 93%+ on ExtractBench). So there is enormous economic value in building a complex extraction benchmark that reflects this long-tail of knowledge work.
We spent a long time curating the most complex elements that we saw from real world extraction use cases:
✅ Document conventions: containing legends or rules where the rest of the document should be interpreted
✅ Density of internal jargon: specific definitions that must be cross-referenced within a doc
✅ Aggregation and reasoning: beyond returning exact values on a page, infer values derived from aggregating multiple sources of information
✅ Drawn relations: complex diagrams like procedures and circuits
✅ Charts inside documents: read precise points within a chart
✅ Long chart sheets: some data series like well logs cram all this data into an extremely long page
Within the industries where document extraction is relevant, even small mistakes can have enormous negative impacts. Therefore our headline evaluation metric is based on perfect documents: the proportion of docs with perfect (100%) value f1 score.
(Besides this we also measure value f1 and grounded f1)
It’s an extremely hard task for even the latest frontier models to solve, topping out at ~58% or so in Value F1 score for Codex with Astra - which is quite good but extremely expensive (almost $1 per page!) Our LlamaParse agentic/agentic plus modes are 20-40% and cover the left side of the Pareto frontier, at price ranges that are feasible for full production use.
ExtractBenchPro is available today as a website/technical blog, and also on Github and HuggingFace. Check it out!
Blog: llamaindex.ai/blog/introduci…
Website: extractbenchpro.ai/
Hugging Face: huggingface.co/datasets/llam…
Github (is ExtractBench): github.com/run-llama/Extract…
deepvaluebettor retweeted
Saw a guy call it cuck coding instead of vibe coding because you just sit in a chair and watch😭
deepvaluebettor retweeted
This is the least surprising graph in the world if you understand how gradient descent works.
deepvaluebettor retweeted
Obama is perfectly displaying the concept of hallucination with LLMs here.
He sounds so darn good and confident, while having absolutely no clue wtf he sayin.
🚨 OBAMA ON AI: "If we are thinking about AI just in terms of how do we cure cancer or get better energy, you can do that without having agentic AI and having it just roaming free in the internet. The reason you are doing that is because you have to market a product that people will pay money for.
That’s a misalignment between what our society needs and the commercial imperatives that these companies are facing, not because necessarily they’re trying to do bad things, but because they’ve got to justify these valuations.
So, that’s one more reason why it is really important for us to have a competent government and a serious bipartisan conversation around this issue, and we have to do it fast. And I would encourage voters to pay attention to this. If somebody does not have a serious plan for how to deal with this, then they’re not meeting the moment, and you should probably look for somebody else."
deepvaluebettor retweeted
Okay, I’ll take a hack at Deans argument without his conflict of interest.
What China is doing in AI is called dumping. They do it in literally every industry they enter. The goal is to kill all local competition by subsidizing their own industry so they can produce at a loss, then once all competitors are dead, they can charge profitable prices and control the market.
In steel and automobiles this is just bad. In AI it’s potentially fatal to our country.
Unfortunately, dumping is the a very common argument for rent seeking domestic firms who want protection, and Dean’s position here seems to be a Jones Act of sorts for AI. This has famously not saved our shipbuilding industry and I suspect it won’t save our AI labs.
Stopping open weights it’s virtually impossible. So our only other option is governments taking stakes in labs and subsidizing our own industry.
If you scream and yell about either Jones Acting AI or subsidizing GPUs then you should also explain how we will escape the dumping trap china is trying to push on us to destroy our labs and leave themselves with the only functioning AI industry in the world.
Some observations on Kimi:
1. It's a very good model! I don't think its performance can be explained away by distillation or anything like that. In agentic coding sessions, it seems pretty much on par with the best public models of Q1 2026. In my fairly limited use, it also seemed very token hungry. It's not obvious to me that this model is actually that cheap to run.
2. I am personally surprised the Chinese state continues to allow the open sourcing of models this good, given potential risks. To be clear, I *myself* might be fine with models presenting this level of marginal risk being open weight, but I am surprised that China is fine with it. I suspect the reason they are is 75% explained by strategic blindness/lack of AGI-pilledness (the CCP is very Yann Lecun-y in its views of AI). The other 25% or so is their lack of compute for customer inference (making China's open-weight strategy an unintended byproduct of US export controls) and the normal Chinese strategy of aggressive exports. For the companies, as opposed to the government, the decision to open source is partially ideological and partially because they are behind, and they know that very few people would pay for sub-frontier models from China.
3. Open-weight models are inherently decelerationist, and I'm continually surprised to see the so-called "accelerationists" so excited about open-weight models. I suspect the reason they are is that they know open-weight models are effectively ungovernable, and they simply like the overall cloak of ungovernability open-weight models create over the whole of AI. It's not a bad strategy; it reminds me of James Scott's recounting of the hill people in "the art of not being governed." Still, in the end, open-weight models deter further AI capex.
4. One probable outcome of an open-weight-model-dominant world is full AI communism, which is precisely what China proposes: rather than a market product, AI is a "public good" which will ultimately be provided by the state as a kind of "digital public infrastructure." This future strikes me as a dystopian hellscape, but I've never met an open-weight models advocate who doesn't ultimately concede this is where things end. You'd be surprised how many 'accelerationists' lobbied me, while I was in government, to support an eleven or twelve-figure federally funded data center so that startups could train models at a subsidy and then give them away for free. There was no other way for AI to progress, they said. Perhaps this is the logical end state of things. Nonetheless, I find myself surprised to see supposed accelerationists excited about such an outcome. I think many of them just don't know what they're doing. Many accelerationists do not view the creation and serving of frontier models as a legitimate business.
5. I would guess that the Trump Administration will at some point realize that their best strategy here would be to create large amounts of regulatory risk around the use of open-weight Chinese models. You don't need to "ban open source" (one of the dumber motifs of AI policy discussion). You just need to direct every agency to issue soft law that creates FUD. "A Federal Reserve Advisory Bulletin found that there may be backdoors in Chinese AI models." It needn't be that well justified. You just create enough regulatory risk that every regulated enterprise backs off. You probably don't want to create so much regulatory risk that you scare off the hyperscalers from serving Chinese models; this will just drive startups to sketchier providers. There's a happy middle ground here. I'd assume they will do some version of this.
6. It's probably true that open-weight models of this capability make the world a bit more dangerous, but not so much more that you'll really notice. At some point the models will be capable enough that you will notice. "A nonliving, invisible, dangerous, and infinitely self-replicating agent escaped from a Chinese lab," you say? Color me shocked.
deepvaluebettor retweeted
Replying to @Fityeth @Polymarket
spoiler alert: it will have zero effect on the sky
deepvaluebettor retweeted
vibe coding allows "ideas guys" to bring their ideas to reality, revealing that their idea kinda sucked
deepvaluebettor retweeted
just never forget that the devs who yap about AI being the best thing ever all day never really knew how to program well to begin with.
They are idiots telling you that the world is solved (charitable read) or worse, they are selling you something and are attempt to deceive you.
This is why we see the same cycle every time. Opus 4 = God's gift to humanity to OMG its utter trash within a few months. The same thing happens with every model release because these people don't even know what dev is. They are larpers.
I believe this is what Geohotz meant when he: "I’m calling it now, the adoption of AI agents into software development will be one of the most costly mistakes in the field’s history."































