this is also because we've most of our coding has moved to slack. feels like the future IDE and company manager is going to be something like this with tons of agents running inside
I was annoyed at losing track of threads in slack so I got claude to one shot a feature complete clone of slack that has much better thread management. it highlights threads that need my input. took only about 5 minutes of my time to build
Today we're launching WebCloneBench: a benchmark which measures a model's ability to explore, copy, build and test various web apps.
For each task, we give the model a reference site from one of Gmail, Google Sheets, Jira, DocuSign, Shopify Admin, Craigslist, Figma, Slack and Zendesk. It is instructed to build a fully feature-complete clone of every part of the application with an 8 hour time limit.
This is exactly what we're doing at Plato. Using a harness that takes recordings of a given website or app we can systematically copy, test, and verify clones then using those clones to create 1000s of RL coding tasks
my spidey senses are telling me the labs are, as we speak, using unguardrailed versions of their unreleased models to systematically reverse-engineer ALL of the internet’s software, apps, and games, then feeding the reconstructed source code straight into their training pipelines while simultaneously building guardrails to ladder-pull the rest of us
after another thread with building manager that was "too large for her computer to allow her to edit" my Dot was able to figure out and purchase the right insurance from corgi and now our office keys are available for pickup
I told my Dot to "handle everything for our new office move" and after several days of it still not finalized I realized he had been spamming everyone with every minor update and 100s of questions trying to figure out movers insurance
mathematicians now have the job of paving the way for post-agi life
We’re releasing a broad range of new mathematical results produced by an internal frontier model.
We’ve been consulting with the independent Advisory Group on Mathematics and Artificial Intelligence at the Institute for Advanced Study, and we have drawn on their advice and public recommendations to inform how we release these results.
github.com/openai/math
This benchmark is the accumulation of a lot of the work we've done in the past 12 months. This benchmark tests coding, qa, long horizon planning, and most importantly world exploration.
Excited to see models climb this!
Today we're launching WebCloneBench: a benchmark which measures a model's ability to explore, copy, build and test various web apps.
For each task, we give the model a reference site from one of Gmail, Google Sheets, Jira, DocuSign, Shopify Admin, Craigslist, Figma, Slack and Zendesk. It is instructed to build a fully feature-complete clone of every part of the application with an 8 hour time limit.
Rob Farlow retweeted
Who grades the graders?
LLM-as-judge on short answers is well studied. Agent judges are not: they open the files, run the code, and investigate before ruling on another agent's work.
Introducing Arbiter-Bench: 71 real agent runs that frontier agent judges get wrong.
Rob Farlow retweeted
America is banning AI in schools.
China is using AI to create geniuses.
Introducing Aristotle: The AI tutor that solves America’s broken education system. heyaristotle.com