Applied AI at OpenAI - Evals
Joined August 2020
- Tweets36
- Following76
- Followers35
- Likes26.3K
Robert Tinn retweeted
OpenAI models helped researchers—operating as part of our Patch the Planet initiative—discover critical security flaws in PHP, an open-source programming language used to run an estimated 70% of websites.
Over the past two months, we led a coalition of maintainers, researchers, and security providers to prepare fixes and protections ahead of disclosure. This duration was necessary because the fix was quite intricate and required a major change to the PHP engine. Every site using PHP can now safely upgrade their version to stay secure, in addition to applying mitigations from various security vendors.
Due to the severity of these flaws and PHP’s widespread use, we consider this bug to be one of the most impactful vulnerabilities in recent memory. It also marks a new phase, where AI can find security bugs with internet-scale impact.
This is an example of the Defender’s Window in action: to our knowledge, frontier AI was able to find this vulnerability ahead of attackers, giving defenders time to prepare.
openai.com/index/coordinated…
Robert Tinn retweeted
I used GPT-6 Astra to break an unsolved cipher to one of Napoleon's generals that had gone unread for 217 years.
What makes this impressive isn't actually the codebreaking, but that Astra completed the entire multi-modal workflow in ~6 hours from a single image and goal.
1/6🧵
Robert Tinn retweeted
The team has been working hard to make GPT-6 Astra the best model for frontend and agentic coding. Please try it out and share what you think!
Real-world results are in. There is a new #1 on Code Arena - GPT-6 Astra (Max)!
It also reshapes the Pareto frontier as the best-performing model at $40/Mtoken, which matches the latest Claude model pricing.
GPT-6 Astra by @OpenAI takes the top spot in Code Arena: WebDev with a score of 1797 pts. This opens up a solid +35pt lead over #2 Claude Fable 5.1 (Max) at 1762 pts and #3 Claude Opus 5 (Max) at 1688 pts.
This is a significant improvement from GPT-5.6 Sol (xHigh) at +180 pts, ranked at #13.
Category level votes still incoming, but already we see it at #1 in: Data & Analytics, Consumer Product, Content Creation Tools and #2 in Gaming and Simulations.
Stay tuned for other categories like Brand & Marketing, Reference-Based Design and Full Stack rankings.
Congrats to the @OpenAI team on this release!
Robert Tinn retweeted
GPT-6 Astra built this Manhattan world in Unreal Engine over the course of a week.
It was literally able to go street by street to make each one perfect.
Robert Tinn retweeted
We will give one banked reset for every day you don't have access to Astra on your paid ChatGPT plan, starting today. Team is moving mountains to give access as fast as we can.
First one will land in ~ 3 hours. There is still time to create your account if you don't have one.
Robert Tinn retweeted
We are starting to release GPT-6 Astra and we are doing it as carefully and quickly as possible. It was very important to us that we bring it to all Plus users and not only Pro, Business and Enterprise.
It will take a few days for the rollout to complete and behind the scenes many novel systems will operate at scale for the first time and we are bringing a lot of compute up.
It is pure magic.
openai.com/index/gpt-6-astra…
Robert Tinn retweeted
GPT-5.6 Sol is now $4/$20 per 1M tokens – the cheapest frontier model out there.
Just for fun, I dug up GPT-4's original API pricing back in 2023. It was $30/$60 per 1M tokens!! For 8k context at the time. How far we've all come.
Robert Tinn retweeted
Previewing Ultrafast mode for @OpenAI's GPT 5.6 Sol, powered by Cerebras.
GPT-5.6 Sol Ultrafast generates responses at up to 750 tokens per second. That's the full, GPT-5.6-Sol model -- up to 14× faster than the same model on Standard processing.
It speedran Humanity’s Last Exam in 11h 11m, nearly 7× faster than Claude Fable 5 with comparable accuracy.
Robert Tinn retweeted
After deployment, we applied GPT-5.6 Sol to advance the frontier of efficiency by making itself more efficient to run.
The results:
- 20% lower serving costs from production GPU kernel improvements.
- 15%+ better token-generation efficiency from improved speculative decoding.
Robert Tinn retweeted
JUST IN: GPT-5.6 Pro disproves the 30 y/o Dinitz–Garg–Goemans conjecture, a long-standing problem in mathematics — after being prompted to "do a breakthrough."
Robert Tinn retweeted
there are a lot of benchmarks that suggest 5.6 sol is the best model in the world right now, but the most reliable way to tell is that elon is obsessed with me again
Robert Tinn retweeted
Okay, I gave Opus 4.8 max effort a shot against GPT 5.5 xhigh on a medium-ish scope ticket for work.
Both models run in latest version of @cursor_ai. 1 plan/execute session each.
Results:
Opus 4.8: 16.5M tokens, $17.26
GPT 5.5: 5.9M tokens, $5.57
5.5 still the goat.
Robert Tinn retweeted
Today, we share a breakthrough on the planar unit distance problem, a famous open question first posed by Paul Erdős in 1946.
For nearly 80 years, mathematicians believed the best possible solutions looked roughly like square grids.
An OpenAI model has now disproved that belief, discovering an entirely new family of constructions that performs better.
This marks the first time AI has autonomously solved a prominent open problem central to a field of mathematics.
Robert Tinn retweeted
1,000+ of you signed up for tomorrow’s Codex Community meetup in London 🤯
To make the waitlist a bit more fun, we’re running a tiny competition 😁
Reply with something you’ve built with Codex, or a cool workflow you use it for. Bonus points for a product link or demo 👀
@romainhuet @reach_vb @Andy_AJT and I will pick the coolest ones!
luma.com/codex-ldn-1
It's one of the most exciting teams!
Hiring in London 🇬🇧 for my Applied AI for Startups team at @OpenAI. Apply if you want to help frontier startups build the future + shape OpenAI.
Current team is full of ex-founder/CTOs, some have AI PHDs, others have been research engineers, data scientists & ML Engineers.
Looking for people that are deeply technical, high agency, and love startups. Apply or reach out
Robert Tinn retweeted
GPT-5.5 is in a league of it's own when it comes to coding. This was first clear based on the CursorBench results, and now via Terminal-Bench 2.1.
Adoption in the API and Codex has been unlike anything we've seen at OpenAI -- if you haven't already done so, give it go!
Very excited to release Terminal-Bench 2.1!
Coding agents are among the most economically consequential deployments of LLMs to date. As agents improve, benchmark reliability matters more.
We audited TB2.0 and found and corrected issues in 28/89 tasks. 30% of the benchmark!
But the rankings survived, absolute scores moved up to 12pp!
