@refresh_dev (YC P25), building RL training gyms for computer use | believer in communities
San Francisco
Joined February 2022
- Tweets1K
- Following2.3K
- Followers2.2K
- Likes19.4K
Christopher Settles retweeted
Can today’s LLMs generate rubrics that reliably evaluate tomorrow’s smarter models? Automated rubric generation can save enormous expert effort, but prior work didn’t evaluate responses from LLMs released after the rubric-generator model.
Our new paper investigates how to best scale test-time compute for this, developing a new rubric generation pipeline called Rubricon. Rather than just relying on LLM self-critique, Rubricon grounds rubric-refinement in how candidate rubrics actually score example responses.
Compared to 4 other synthetic rubric generation methods across 5 diverse benchmarks, Rubricon rubrics consistently score model responses more like human experts do [1/n]
Christopher Settles retweeted
I went through every data vendor that launched in the last 3 weeks. Almost all of them are lying about their revenue, their lab partnerships, and their backgrounds.
(skip to 1:04 if you want to skip context)
Too many founders and VCs have been fooled by these people. Stop believing everything you hear. Data is much harder than most people think.
my unfiltered thoughts on the data industry right now:
there are too many brokers in the data space. in the last 4 months this industry has become the new equivalent to dropshipping.
i have no issue with that. but if your entire moat is spam calling smb owners all day and begging them to license their data for a micro1 referral fee, you are not going to make it, and you are not the next mercor. most of you are not building anything at all.
to everyone reading this, and the hundreds of other posts on your timeline from 18 year old sf kids claiming $200m run rates please ignore them. these are the modern day elizabeth holmes’.
they're taking one day of contracted (not paid) revenue and multiplying it by 365. the worst i've seen is someone who made $100k off a micro1 referral fee in 6 hours and claimed they're at a $73m run rate (12 (daily working hours) / 6 x $100k x 365).
this is literal fraud, especially if you raise money off of this claim. you see this in a lot of yc companies in recent batches.
the only businesses i see surviving this are the giants that already exist (mercor, micro1, surge ai, turing) and the non consensus bets: consumer data collection apps like kled, robotics data collection companies like build ai and meckaai, etc.
tldr: don’t believe everything you read, and just brokering company data is not a venture scale business and should not be claimed as such.
The main bottleneck to rl environments is the amount of disk space you have available on your MacBook.
Christopher Settles retweeted
We are releasing Ultramarine 1 Flash, a frontier-class model built for knowledge work. It is open-source and achieves similar performance to Fable 5, while being 60x cheaper.
It was post-trained by a team of 5 for ~$300k all-in. Available on API now, with weights releasing in 2 weeks.
Christopher Settles retweeted
Huge day for us today: @resend is now billing through Autumn
This is a full circle moment. @johnyeo_ using the Resend API is actually what inspired us to build a devtool in the first place. They set the standard for product quality and user experience.
They moved to @autumnpricing because they needed a single, reliable engine to monetize all the new products they ship, and their growing enterprise motion meant a lot more custom contracts to manage.
Resend is the most downloaded email API in the world and growing like crazy. We’re super proud they chose us to meter, enforce and bill their highest-volume workloads.
Christopher Settles retweeted
Today we're launching WebCloneBench: a benchmark which measures a model's ability to explore, copy, build and test various web apps.
For each task, we give the model a reference site from one of Gmail, Google Sheets, Jira, DocuSign, Shopify Admin, Craigslist, Figma, Slack and Zendesk. It is instructed to build a fully feature-complete clone of every part of the application with an 8 hour time limit.
Another animation benchmark the same day! How awesome that we're all tying together.
@himanshustwts we should collab!
Today, I am excited to announce Animation Bench. Our first benchmark to evaluate frontier models on web animation reconstruction.
Frontend generation is a common commercial use case for coding agents. In addition to static layout it contains transitions and interactions triggered by dynamic behaviours. We evaluated four frontier models on 48 tasks from real websites across three axes:
1. Visual Similarity
2. Motion Consistency
3. Layout Correctness.
We find that motion consistency is often incomplete or missing.
1
5
1,112
Christopher Settles retweeted
Too many people claim that coding LLMs can handle 3D, but how exactly? We need more benchmarks on this.
Christopher Settles retweeted
New continual benchmark just dropped! Excited to chug away at this, the models are good in some aspects but we are still far away from models making high fidelity 3d movies.
Today at Refresh we're proud to release our first public benchmark, BBv1.
We talked to animation experts who did the 3D modeling for harry potter, and realized that so many of the capabilities of the LLMs were still lacking.
Verifiers include VLM as a judge, along with geometry!
If interested in 3D animation evals, DMs open!
It is said often
To all the data company founders, if you can’t do 18 hours of work 6-7 days a week in bursts when needed, you’re ngmi as a category leader.
This is what Mercorians are doing regularly, and what we did at Scale AI back when Scale was still relevant. This is also why folks either survive and thrive after 6 months or they leave after 1-3 when the shiny logo aura wears off and it dawns on them that the work is done with blood, sweat, and tears on top of being adaptable and needing to learn at warpspeed.
This is also why Scale AI is cooked is because I hear folks are doing 50 hr weeks now with good work life balance.
It’s the harsh and brutal honest truth is that for operationally driven products and businesses, the champs will simply outwork you.
But if you can survive and thrive, you get granted with the power of Beast Mode. Afterwards, everything else seems like a walk in the park.
1
4
826
Christopher Settles retweeted
We’re sharing a solution to the Navier-Stokes Millennium Prize Problem, one of the deepest problems at the frontier of mathematics.
The proof was produced by a group of agents, using an OpenAI next-generation model significantly more capable than GPT-6 Astra.
The problem concerns whether the description of smooth three-dimensional fluid motion modeled by the Navier-Stokes equations can break down. It has remained unresolved for roughly 90 years.
