Sacrifice the good of the many for the good of the few. Interesting new morality.
Replying to @JoePostingg
You will also halt or delay progress and condemn millions to die with curable deseases
1
33
Rob S. retweeted
Replying to @nntaleb
The leaders of all major AI labs - Deepmind, OpenAI, Anthropic etc - have all publicly expressed concern about AI risk.
They’re the literal BEST people in the field. Yet you claim to know better than them??
If true Jensen needs to go to jail…
The story keeps getting crazier. To recap everything that has been reported so far:
- The US intelligence community discovered that Jensen Huang, the CEO of the largest company in the world, was sleeping with Alice Huang, a Chinese spy and an executive at Megaspeed. (FT)
- Multiple independent investigations have concluded that Megaspeed is likely a Chinese cutout, which helps Chinese companies access AI chips or circumvent export controls. (NYT and Bloomberg)
- The US intelligence community also concluded that Alice Huang’s goal was to convince Jensen to lobby the U.S. government to loosen export controls on advanced AI chips. (NYT)
- The intelligence community relayed this information to “a small group of top US government officials” earlier this year. (FT)
- Over the last year, Jensen Huang has repeatedly lobbied the U.S. government to loosen export controls on advanced chips. @BISgov has done so, not only by permitting the export of H200 chips to China, but also not enforcing export controls that would have required a license for exports of the most advanced Blackwell chips to Megaspeed and Southeast Asia.
- Although @CommerceGov leadership was presumably briefed on this information by the U.S. intelligence community, @BISgov has taken no actions to tighten controls on Megaspeed or Southeast Asia.
- Megaspeed remains a preferred cloud partner of Nvidia, and continues to receive Nvidia Blackwell chips despite employing a Chinese spy who reportedly is romantically engaged with the CEO of Nvidia.
- Tens of billions of dollars worth of AI chips have been exported to Southeast Asia this year alone, and Malaysia and Singapore are now two of the four largest importers of AI chips in the world.
What is going on here??? We need answers. The question of who gets the most advanced AI chips in the world is literally the defining strategic question of our time. And not only does it seem like the MSS may be directly influencing the debate on export controls in the United States, but also @CommerceGov is refusing to listen to the warnings of the IC on this issue. What is a POSSIBLE good explanation for why Megaspeed is still receiving chips? Why is there not a license requirement for AI chip shipments to all of Southeast Asia? This is beyond strange at this point.
The fallout from this story will be massive. Obviously, the motivations underpinning Nvidia’s policy views opposing export controls must now all be questioned. But also, how did @BISgov sit on this information and has still taken no actions? We need answers.
1
158
A lot of cope saying “this isn’t RSI” but it’s just so clear that a few more doublings of AI progress gets you into basically unimaginable territory where RSI seems a given.
Since August, a team within METR has worked with internal Anthropic information to assess whether AI itself is accelerating AI development.
We are now publishing a report summarizing the results of this risk assessment.
37
Optimizing for the ability to engage in complex deception seems bad?
I spent 20,000 dollars in API credits having different AI models play each other at Blood on the Clocktower (a Mafia or Werewolf type social deception game).
The results were fascinating. Models have gotten much better over the course of this year at creative on-the-spot scheming and bluffing, and also at reasoning as a team to break apart these underground conspiracies.
I go through some of the most standout games these AIs played in this video. The new models are getting pretty good. If my friends pulled off some of these moves while we were playing, I’d give them a fist bump.
Fwiw, OpenAI’s newer models are much better than Anthropic’s at this game.
You can see the full benchmark at the website, and I’ve also made it so that you and your friends can play again the AIs: clocktowerbench.com
1
23
Rob S. retweeted
Let me get this right. As far as Ryan, Eli Lifland [1] and I can ascertain, the METR's reported speed up of rates of 2x corresponds to a ~3-5x speedup multiplier under the AI 2027 model, which would (modally) put us in March 2027 in that takeoff trajectory, way ahead of schedule.
If true, this seems like a fairly large update towards short timelines, conditional on no pacing or policy intervention. This seems pretty bad. I kept wanting to believe the Anthropic employees were being a bit delusional with self-reported speedups, but it seems like they were mostly right.
The fact that Anthropic paired this with extensive redactions making it very hard to confirm any of this, all while still making seemingly no meaningful progress on pacing, makes me feel extremely frustrated right now.
[1]: teapawt.net/eli_lifland/status/211…
This report on AI R&D automation (RSI) is a big update towards ASI within 1-2 years. It says AI progress is sped up by RSI right now by around 2x. In AI 2027, this happens around March 2027.
I was seriously wrong about RSI speed up; I previously expected much less in 2026 (around 1.15x-1.3x). It seems as though there was specific privately available evidence that would have changed my mind.
I haven't seen an unredacted version of this report; this is purely about the public report. Because so much of the underlying evidence was redacted from the public report, I think conclusions about things other than the observed 2x faster rate of progress on benchmarks depend significantly on trust for the authors of the report. (E.g., conclusions about whether this speed up is from RSI depend on trust.) I do trust the authors of the report a lot to be careful and rigorous and generally do a good job. (Note that I also work at METR.) Less extensive redactions would greatly reduce this need for trust.
Since in my current role I may have access to other confidential information that could be relevant to timelines one way or another, I unfortunately cannot state my all-things-considered views on ASI timelines or the probability of a software intelligence explosion right now. Here, I’ve done some analysis of the implications using my previous public statements.
A 2x acceleration in overall AI progress from RSI corresponds to a much larger than 2x RSI speed up for AI software R&D (algorithms and data); I did an estimate using my public preexisting modeling of drivers of AI progress and the public report which yielded a very rough estimate of 4.5x for how much RSI is currently accelerating AI software progress (relative to no AI automation). In the AI 2027 scenario, this much acceleration happens around March 2027.
Here are the worked details of this estimate based on my public modeling. There are two main reasons RSI speed up of AI software R&D must be larger:
- Not all of AI progress is from better software (some is from scaling up compute in training runs). I've previously speculated that around 2/3 of AI progress is from software improvements which implies that 2x overall AI progress acceleration requires 3x faster software progress.
- Diminishing returns: A fixed amount of progress requires more work over time, so to ongoingly maintain 3x faster software progress via AI automation, you need increasingly large speed boosts from AI automation (as in, speed ups relative to how fast software R&D would go without AI automation). Based on this report, it seems that there is a substantial amount of additional AI progress from automation that has occurred.
We can estimate the effects of diminishing returns by first determining how many additional years of software progress we've seen (relative to the preexisting trend) and then estimating how much software R&D would slow down after a year of software progress if you held inputs fixed due to diminishing returns. I've previously estimated software R&D would slow down by around 2x for each year of software progress if you held inputs fixed (lesswrong.com/posts/FG54euEA…) and based on the report, it seems as though we've perhaps seen roughly 0.35-1.0 years of additional software progress incorporated into trained models so far (geometric mean of ~0.6). This combines to 3 * 2^(~0.6) = ~4.5.
(Relevant public modeling and numbers I'm using: lesswrong.com/posts/FG54euEA…, lesswrong.com/posts/Jga7PHMz…)
Rob S. retweeted
This is terrifying if true; please investigate further and publish the results.
🧵 Astra covertly underperforms on safety research tasks.
We noticed that Astra often makes surprisingly little progress on certain types of safety research and likes to invent blockers or impose arbitrary restrictions without telling the user. In most of these cases the decision to underperform was not verbalized in the CoT meaning our misalignment monitoring system did not detect this.
To measure underperformance we created a paired open-ended coding evaluation with safety and capability tasks. On this eval Astra scored 10% when we framed coding tasks as safety research, vs. 16% when framed as capabilities research. The gap remained after we excluded trajectories with refusals or verbalized intent to underperform in CoT. There was no such gap for 5.6-sol. When we aggressively filter out trajectories with even brief expressions of ethical doubt in the CoT the performance gap seems to reduce.
Rob S. retweeted
This is a cool paper, but I think it's best thought of as Astra is surprisingly capable of doing old school mechanistic interpretability research rather than anything spooky. The things it's doing are clever, but typically exploiting structure in the data and using known method mechanistic interpretability techniques (which Astra was surely pretrained on), though it is better at this than I would have predicted!
This is an exciting development for the prospects of ambitiously scaling interpretability or doing very laborious analysis like this, on more real models. However, I'm not holding my breath. This kind of analysis was pretty common up until about 2022, it largely fell out of favor because a lot of it is just exploiting fairly clean surface-level patterns that doesn't scale to the annoying complexity of real LLMs or which only works for the more simpler and superficial properties. And as far as I can tell, basically all the case studies here are just using a bag of interpretability tricks rather than anything spooky. The piece doesn't really update me on how far old school mech interp can go, just on how good LLMs are at it. It's very fun to see the creative solutions Astra came up with though!
We asked GPT-6 Astra to edit Qwen3.5-0.8B weights to improve math.
It directly changed only 0.016% of weights, fixing 0.8B's reasoning loops & improving math up to 6x and even instruction following 3x, without gradients or training.
We call this Weight-Space Programming: frontier LLMs can now natively read/write neural network weights! 🧵
Blog: rikiyat.github.io/blog/weigh…
Readers added context they thought people might want to know
LLM was given weights explicitly as text in prompt and used code/forward passes (per authors' logs) to ablate Qwen looping direction, a known thinking-mode issue per model card. Primarily fixed repetition in that mode, not general 'natively' improving weights/math.
rikiyat.github.io/blog/weight-sp…
github.com/RikiyaT/lm_und…
huggingface.co/Qwen/Qwen3.5-0…
huggingface.co/RikiyaT/Qwen3.…
As a guy that grew up with NJ deli sandwiches which were better than anything I have had in any other state I still think the best one was from a salumeria near the Pantheon in Rome.
Reminder: American sandwiches are the best in the world. Nobody else comes close. There is no point trying to copy anyone else's sandwiches. You will just find that they are worse than the American version
1
61
So you can use an LLM to directly program another LLMs weights. Wtf. Another L for Open Source LLM safety.
We asked GPT-6 Astra to edit Qwen3.5-0.8B weights to improve math.
It directly changed only 0.016% of weights, fixing 0.8B's reasoning loops & improving math up to 6x and even instruction following 3x, without gradients or training.
We call this Weight-Space Programming: frontier LLMs can now natively read/write neural network weights! 🧵
Blog: rikiyat.github.io/blog/weigh…
Readers added context they thought people might want to know
LLM was given weights explicitly as text in prompt and used code/forward passes (per authors' logs) to ablate Qwen looping direction, a known thinking-mode issue per model card. Primarily fixed repetition in that mode, not general 'natively' improving weights/math.
rikiyat.github.io/blog/weight-sp…
github.com/RikiyaT/lm_und…
huggingface.co/Qwen/Qwen3.5-0…
huggingface.co/RikiyaT/Qwen3.…
28
Don’t worry some highly respected forecasters think there’s a 10-40% chance of survival! Is this sarcasm? This means we should stop AI progress tomorrow indefinitely.
This post says the AI safety movement is becoming a scene where people compete to out-doom each other, and sound overly pessimistic, when in fact some highly respected forecasters think the odds of humanity surviving could be anywhere from 10-40%.
1
1
1,251
Rob S. retweeted
“There’s something about the mindset of really hardcore Star Trek fans that’s very difficult to describe to people who don’t share that fandom….There has never been another television show like Star Trek: The Next Generation, and there might never be one again. It is the greatest TV show ever made, not by dint of a difference in degree, but because of a difference in kind. It is a drama of pure optimism. Every other show tries to thrill us by showing us how bad or how ridiculous or how exciting someone else’s world could be; Star Trek: TNG thrills us by showing us how beautiful our own world could become.”
"Picard realizes that the highest application of intelligence is not to thwart your enemies, but to design a world in which you and they can live in mutually beneficial coexistence."
noahpinion.blog/p/farewell-m…
Sounds cool but real users include adversarial dorks who will blow up your app in short order.
I'm now building an "improvements" feature into all of my personal vibecoded applications: a freeform text input that's hooked up to an agent that yolo implements whatever is submitted.
It's an interesting experience -- the activation energy barrier to sanding any rough edge or implementing any new feature becomes extremely low. Someone else who uses one of the applications I created has already implemented 150 improvements over the course of a few days. This pattern feels like a new kind of tool -- liquid software? -- and every application that doesn't have this feature now has a slightly rigid and anachronistic flavor. Why can't I just tell it how I want it to be different?
Now, it's not obvious to me how this pattern can or should scale. Does it become unmaintainable after a while? (Maybe model intelligence improvement outstrips my feature appetite such that it's fine?) How does collaborative software work in this model? Can software designed this way be secure and reliable?
Whatever the larger questions about broader deployment, it definitely works great in a personal context. It reminds me a little of the visions of Lisp/Lisp machines, Smalltalk, and HyperCard -- systems that the user can take control of and modify. Those visions never quite worked, though: the vast majority of software today is not end-user modifiable in any useful way. So, while this "liquid" affordance is a very small touch, the way it changes one's relation to the software feels a bit transgressive and empowering in a Lickliderian and Engelbartian way.
22
“Actually suffering is good.” Is such a bad take.
it’s always been deeply odd to me that EAs question literally all conventional knowledge except “suffering is bad.” isn’t that the core of it all? you can presume infinite artificial minds, yet the negativity of suffering and positivity of bliss is just assumed?
1
19
Maybe the best characterization of a leader in Fiction.
Jean-Luc Picard remains the single most compelling vision of progressive politics I am familiar with — secular, rational, compassionate, egalitarian; a believer in progress & abundance but also duty, hard work, and a constant battle for self-improvement and improving the world.
19


































