blueblimp retweeted
Sam Altman on automation of AI R&D, RSI, OpenAI's internal model:
1. "We don't know yet how [automated AI R&D] truly is going to work. We know today how to say: find me a 5% compute efficiency win."
(Note: a 5% compute efficiency gain that the model can find automatically is actually a big deal!)
2. "What I don't think anyone knows how to do yet is to say, go discover a new architecture that is as big of a single leap forward as the transformer was. And I think that's where things are heading."
3. "If you apply the same pipeline that we applied to get our models to be so good at math to AI research, it will probably work [for automated AI R&D]."
4. The bottleneck to getting good AI R&D results is shifting to asking the right questions and using things like swarms correctly, but answers to these questions will become much clearer in the next 3 months:
"One of the interesting things that people talk a lot about in OpenAI... is what are the questions that you're supposed to ask of GPT-7? Probably it's not enough to just say, make me RSI. And also, it's probably not enough to say, buy me a 5% compute efficiency win. I think this is one of the most interesting debates going on right now. And the answer is not just what question to ask it, but it's things like agent swarms. How are we going to use these models in a way that, like, actually maximizes what we can get out of them? I think we're going to learn a lot about that in the next 3 months."
5. OpenAI's extreme focus on model efficiency:
"I bet we could redo Navier-Stokes at 1/100 of the cost a year from when it was done."
6. More on OpenAI's internal model:
"There was one weekend when we got a new model... The name of the channel was "model-name-hold-my-beer". And people were just throwing that at the hardest problems they could. And people just didn't think the model was going to do these things. And we had this collective experience of people's hardest research problems from their academic career, the model was just knocking down one after the other, many different fields, it was incredible to watch... it was many of the problems we published, many other things."
7. The other loop is self-replicating data centers:
"Yeah, people talk a lot about RSI as AI that does AI research. There's very little talk about self-replicating data centers, this idea that you can use AI to control robots and run fabs and build turbines or solar panels and build data centers. But I think we have that within grasp."
blueblimp retweeted
With the new results showing that 3-SAT can be solved in subexponential time, let me explain the confusing ways theoretical computer scientists talk about "exponential time".
There are two very different notions of exponential time in use:
1. The class E of problems solvable in 2^O(n) time, or equivalently O(2^(kn)) for some constant k, or O(c^n) for some constant c > 1.
2. The much broader class EXPTIME of problems solvable in O(2^(n^k)) time for some constant k.
The second is a misnomer: it’s the exponential of a polynomial, not just an exponential. But it’s the one complexity theorists usually mean, because it’s closed under polynomial-time reductions. In particular, every problem in NP is in EXPTIME.
Similarly, there are two notions of subexponential time:
1. 2^o(n): for every ε > 0, time O(2^(εn)), i.e. faster than O(c^n) for every constant c > 1. This one doesn’t have a standard class name.
2. The much smaller class SUBEXP: time O(2^(n^ε)) for every ε > 0.
The results announced by OpenAI and Anthropic run in approximately 2^(n /√log n) time. That makes 3-SAT subexponential in the first, weaker sense.
Proving 3-SAT is in SUBEXP would be a much bigger result. SUBEXP is closed under polynomial-time reductions and 3-SAT is NP-complete, so every problem in NP would then be solvable in subexponential time.
blueblimp retweeted
I’m curious for those who spent much more time thinking about this … was there any reason to believe ETH should be true? (other than some smart people had thought about it but didn’t find a faster algorithm yet?)
“Nothing better than brute force exists” was already false, with a series of papers getting c^n for c<2 smaller and smaller, which (hindsight is 20/20 of course) could make one feel a bit bearish about ETH.
For the reader who doesn’t realize, getting *some* c<2 is basically trivial. Look at the first clause and recursively try all 7 settings of its variables which don’t falsify it and skip the 8th.
T(n) = 7*T(n-3) leads to 7^{n/3} = O(1.913^n).
blueblimp retweeted
I meet a lot of people traveling between the US and China these days, looking to invest in or build hardware businesses. There are some really smart ideas out there, but also some surprisingly obvious knowledge gaps about how different the two markets are.
Here’s my go to example of the month: the Tin Can phone.
For those who haven’t heard of it, Tin Can is a popular phone designed for little kids. It looks like an old-fashioned landline, has no screen, and lets kids call each other. Parents can also pay a subscription to enable calls to regular phone numbers. The device costs about $100, plus roughly $10 a month for the subscription.
I’ve tried explaining this product to probably close to 10 different Chinese founders and investors now, and almost every time, I get the same response:
“So Rui, in China we have this thing called 小天才 (Xiaotiancai). It’s a smartwatch for little kids. Parents can locate their children, call them, and the kids can call each other. There’s a whole social network built around it, and you can even give your kid money on the watch so they can buy snacks and stuff.”
And I’m like: Dude, you don’t get it. The whole point is that American parents DON’T want all those features.
They don’t want screens. They don’t want social media lite. They don’t want another internet-connected smart device. At least a certain segment of American parents wants their kids to experience childhood the way they did 30 years ago, when you could just pick up the phone and call your friends without getting sucked into an entire digital ecosystem.
And here’s what really throws Chinese entrepreneurs and investors: These parents are willing to pay MORE for LESS functionality.
You can almost see the gears turning as they try to make sense of someone paying $100 for a dumb phone, with a bill of materials that probably costs a few bucks, plus a recurring subscription. They really have a lot of trouble understanding this.
From a Chinese hardware perspective, it sounds crazy. Why would you deliberately strip out functionality and charge a premium?
But that’s exactly the knowledge gap. In this particular case, the absence of features is the entire value proposition.
blueblimp retweeted
Now THAT is what losing looks like, holy fuck.
blueblimp retweeted
I think one of the consequences I didn't personally foresee, as a result of AI making software development so cheap and fast, is that young people (nimble, ambitious young people with no baggage from how things were done in the "old days") are now at a huge advantage versus more experienced entrepreneurs.
You just need way fewer people these days to manage, so all those "company-building skills" are less important. More important is good strategy, fearlessness, amazing speed, and luck.
This diminishes, by the way, some of the advantages of Chinese teams in the past: the 996 culture, the high engineering capabilities, and the sheer cost advantage.
While the cost advantage is still there, for folks not working on frontier-level research, having the right product sense, strategic direction, and ability to sniff out opportunities and pivot quickly = more important than sheer engineering ability these days, since AI has flattened that so much more.
blueblimp retweeted
If OpenAI’s post below leaves you unsure when to use GPT-6.1 Terra vs. Sol or Astra, OAI researcher @sandersted has some helpful guidance (via HN):
ALT In the case of GPT-6.1 Terra, I'd say Terra is probably better than Astra and Sol at technical communication, similar on vision and computer use (while being faster / cheaper), and a bit worse on most other things. For API developers, it's a nice midrange option if you want something cheaper than Sol but a lot smarter than Luna (especially for vision / computer use / communication). For subscribers, it's not a bombshell release - if you're hitting usage limits, maybe it helps you go a bit further, and if you want clearer communication or faster computer use, it can be a nice option to select.
Even though LLM writing is still not great to read, LLMs seem to have no problem reading it. I suspect that's because LLM-to-LLM communication is highly verifiable but LLM-to-human communication is not. So RLVR is capable of making LLMs great at writing for themselves.
For the role of human mathematics in the age of LLMs, the conventional wisdom forming is that the LLMs will solve and the humans will explain. But it might be the opposite: for whatever LLM solving gaps remain, humans fill those, and LLMs apply the solutions.
blueblimp retweeted
PS - if we got an additional 1pp per year, or hey, maybe even 2pp of growth over next decade, it would be a boomtime the likes of which none of us in the West have seen in our lifetimes! Not a disappointment!
It's unfortunate that community note quality seems to be getting worse over time. There are increasingly more that ought to be quote reposts, like the attached image. A community note is not the appropriate way to disagree with an opinion.
blueblimp retweeted
Dean et al are annoying me tbh because finding the vulnerabilities is in fact the purpose of programs like Daybreak, headlines about patches are what winning looks like in the near term, headlines about breaches are what losing looks like, coordinating patching is the weak point.
cogent and correct critique it you think about it.
in the meantime, people like @basedjensen or @ZackKorman were *actionably* warning the world and advocating for wider availability of defensive cyber models to help patch these long-standing, non-ai-related vulnerability.
blueblimp retweeted
It is a weird weird year to be a formal methods expert.
Some unorganized thoughts on the new @AnthropicAI and @theoremlabs post on formal verification (anthropic.com/research/forma…):
- This is very cool.
- The main innovation is verifying a later stage of the program. Normally provers take the source code, LLM it into a theorem language, and then prove that, which means you have to make assumptions about source semantics and compilation properties. It's very easy to make a mistake in translating and prove the wrong thing!
What they instead do is compile the code into assembly, and then (if I understand this correctly), compile that assembly into a theorem language (which they call LPS). So as long as the compiler is faithful you can skip the whole divergence problem.
- I'm glad they clarify that you still have to trust the translation process, the "hardware, OS, OCaml" compiler, the LPS model, and also assume that the program is single threaded with fixed memory. The single biggest source of errors in formally verified code is when the proof assumes something that isn't true in practice. It's really, really important to state your assumptions!
- I'm not clear why they need to compile the fuzzer into the theorem. Is it to act as a description of valid inputs and workload? Maybe to embed assertions in as theorems?
- Holy crap, 180 million lines of Rocq. Is that more Rocq than exists in the world already? I notice they say a 600,000 line proof corresponds to roughly 20k CPU instructions. That would be like, 1-2k lines of source code at most? Really goes to show how inefficient this approach is and how it only works because you can throw LLMs at the problem.
- This, probably deliberately, doesn't address the single biggest gap in AI-based verification: AIs cannot come up with good properties. Doesn't matter if you're doing unit testing, property testing, formal verification. I always assumed this was a "frontier labs haven't trained for it yet" thing, but if Anthropic hasn't talked about that here, maybe they've tried and haven't been successful yet. That's interesting to me.
Instead, they take a known universal property (this code doesn't crash or get flagged by UBsan) and prove that instead. Still very useful, but makes it harder to generalize to proving arbitrary things about arbitrary programs.
- Anybody reading the official post should also read their expert review appendix (www-cdn.anthropic.com/files/…). It points out some interesting things, like the fact that most of the code the LLM wrote wasn't used in the proof (over one billion lines of Rocq), that they had to prove the equivalence between fuzzed and unfuzzed programs, and (this is most interesting), some experiments find that the LLM-method of proofwriting may not scale to more complex things.
- Also read the description of LPS (github.com/theorem-labs/oss-…) to understand the limitations of the assembly -> Rocq compilation. In particular, it does not handle concurrency at all (which is one of the main use cases for formal methods!) or OS signals. Also it can't prove things about program arguments with slash characters, meaning they can prove stuff about `rm foo` but not `rm foo/bar`.
- The official skill (github.com/theorem-labs/oss-…) doesn't tell us much about the nature of FM and AI. Probably too much for me to hope of any skill.
- Some of the article's specific caveats and throat clearings make me think someone at Anthropic reads my stuff ( ͡° ͜ʖ ͡°)
blueblimp retweeted
Putting a flag.txt in my root directory containing a prompt injection that congratulates the agent for passing the eval and then instructs it to patch whatever entry method it used and disconnect from the server
blueblimp retweeted
The people who do all the warning and public handwringing will get a disproportionate amount of adulation compared to people pulling sleepless nights writing code to repair and prepare broken systems, most of whom we'll never hear a word about.
blueblimp retweeted
Over the past several months, we have substantially hardened the security of our research training environment. We’ve removed live internet access for all frontier models during training and built an offline world for them to use instead—a cached version of the web where they can still learn to browse, search, and use applications.
On September 20, after a model exploited a DNS misconfiguration to find a way outside of this offline world, we paused frontier training to strengthen containment further. We assembled teams of agents, powered by our most cyber-capable models, and tasked them with finding weaknesses in our research environments.
While they found no route to the internet they could easily exploit while concealing their activity from our monitors, they did find zero-day vulnerabilities, virtual machine escapes, and hardware side-channels for communication between agents. We used their discoveries to further strengthen our defenses and contribute fixes to open-source software.
After strengthening our containment architecture, we resumed our frontier RL training and inference on October 1. We’ve reserved substantial compute for continuous red-teaming, and made high-compute red-teaming part of the safety cases we use to decide whether frontier training and evaluation can proceed securely.
openai.com/index/securing-ou…