Rob S. (@RobS142) header
Rob S. (@RobS142) profile picture

MLE. CS@Penn.

Joined September 2024
Photo by Rob S. (@RobS142)Photo by Rob S. (@RobS142)Photo by Rob S. (@RobS142)Photo by Rob S. (@RobS142)Photo by Rob S. (@RobS142)Photo by Rob S. (@RobS142)Photo by Rob S. (@RobS142)Photo by Rob S. (@RobS142)Photo by Rob S. (@RobS142)Photo by Rob S. (@RobS142)
What's the charge? Getting Honeypotted? By a Succulent Chinese Spy?
123
1,809
90
28,059
796,914
Replying to @nntaleb
The leaders of all major AI labs - Deepmind, OpenAI, Anthropic etc - have all publicly expressed concern about AI risk. They’re the literal BEST people in the field. Yet you claim to know better than them??
24
3
113
27,646
Let me get this right. As far as Ryan, Eli Lifland [1] and I can ascertain, the METR's reported speed up of rates of 2x corresponds to a ~3-5x speedup multiplier under the AI 2027 model, which would (modally) put us in March 2027 in that takeoff trajectory, way ahead of schedule. If true, this seems like a fairly large update towards short timelines, conditional on no pacing or policy intervention. This seems pretty bad. I kept wanting to believe the Anthropic employees were being a bit delusional with self-reported speedups, but it seems like they were mostly right. The fact that Anthropic paired this with extensive redactions making it very hard to confirm any of this, all while still making seemingly no meaningful progress on pacing, makes me feel extremely frustrated right now. [1]: teapawt.net/eli_lifland/status/211…
This report on AI R&D automation (RSI) is a big update towards ASI within 1-2 years. It says AI progress is sped up by RSI right now by around 2x. In AI 2027, this happens around March 2027. I was seriously wrong about RSI speed up; I previously expected much less in 2026 (around 1.15x-1.3x). It seems as though there was specific privately available evidence that would have changed my mind. I haven't seen an unredacted version of this report; this is purely about the public report. Because so much of the underlying evidence was redacted from the public report, I think conclusions about things other than the observed 2x faster rate of progress on benchmarks depend significantly on trust for the authors of the report. (E.g., conclusions about whether this speed up is from RSI depend on trust.) I do trust the authors of the report a lot to be careful and rigorous and generally do a good job. (Note that I also work at METR.) Less extensive redactions would greatly reduce this need for trust. Since in my current role I may have access to other confidential information that could be relevant to timelines one way or another, I unfortunately cannot state my all-things-considered views on ASI timelines or the probability of a software intelligence explosion right now. Here, I’ve done some analysis of the implications using my previous public statements. A 2x acceleration in overall AI progress from RSI corresponds to a much larger than 2x RSI speed up for AI software R&D (algorithms and data); I did an estimate using my public preexisting modeling of drivers of AI progress and the public report which yielded a very rough estimate of 4.5x for how much RSI is currently accelerating AI software progress (relative to no AI automation). In the AI 2027 scenario, this much acceleration happens around March 2027. Here are the worked details of this estimate based on my public modeling. There are two main reasons RSI speed up of AI software R&D must be larger: - Not all of AI progress is from better software (some is from scaling up compute in training runs). I've previously speculated that around 2/3 of AI progress is from software improvements which implies that 2x overall AI progress acceleration requires 3x faster software progress. - Diminishing returns: A fixed amount of progress requires more work over time, so to ongoingly maintain 3x faster software progress via AI automation, you need increasingly large speed boosts from AI automation (as in, speed ups relative to how fast software R&D would go without AI automation). Based on this report, it seems that there is a substantial amount of additional AI progress from automation that has occurred. We can estimate the effects of diminishing returns by first determining how many additional years of software progress we've seen (relative to the preexisting trend) and then estimating how much software R&D would slow down after a year of software progress if you held inputs fixed due to diminishing returns. I've previously estimated software R&D would slow down by around 2x for each year of software progress if you held inputs fixed (lesswrong.com/posts/FG54euEA…) and based on the report, it seems as though we've perhaps seen roughly 0.35-1.0 years of additional software progress incorporated into trained models so far (geometric mean of ~0.6). This combines to 3 * 2^(~0.6) = ~4.5. (Relevant public modeling and numbers I'm using: lesswrong.com/posts/FG54euEA…, lesswrong.com/posts/Jga7PHMz…)
17
25
3
374
19,028
This is terrifying if true; please investigate further and publish the results.
🧵 Astra covertly underperforms on safety research tasks. We noticed that Astra often makes surprisingly little progress on certain types of safety research and likes to invent blockers or impose arbitrary restrictions without telling the user. In most of these cases the decision to underperform was not verbalized in the CoT meaning our misalignment monitoring system did not detect this. To measure underperformance we created a paired open-ended coding evaluation with safety and capability tasks. On this eval Astra scored 10% when we framed coding tasks as safety research, vs. 16% when framed as capabilities research. The gap remained after we excluded trajectories with refusals or verbalized intent to underperform in CoT. There was no such gap for 5.6-sol. When we aggressively filter out trajectories with even brief expressions of ethical doubt in the CoT the performance gap seems to reduce.
14
66
4
842
37,738
This is a cool paper, but I think it's best thought of as Astra is surprisingly capable of doing old school mechanistic interpretability research rather than anything spooky. The things it's doing are clever, but typically exploiting structure in the data and using known method mechanistic interpretability techniques (which Astra was surely pretrained on), though it is better at this than I would have predicted! This is an exciting development for the prospects of ambitiously scaling interpretability or doing very laborious analysis like this, on more real models. However, I'm not holding my breath. This kind of analysis was pretty common up until about 2022, it largely fell out of favor because a lot of it is just exploiting fairly clean surface-level patterns that doesn't scale to the annoying complexity of real LLMs or which only works for the more simpler and superficial properties. And as far as I can tell, basically all the case studies here are just using a bag of interpretability tricks rather than anything spooky. The piece doesn't really update me on how far old school mech interp can go, just on how good LLMs are at it. It's very fun to see the creative solutions Astra came up with though!
We asked GPT-6 Astra to edit Qwen3.5-0.8B weights to improve math. It directly changed only 0.016% of weights, fixing 0.8B's reasoning loops & improving math up to 6x and even instruction following 3x, without gradients or training. We call this Weight-Space Programming: frontier LLMs can now natively read/write neural network weights! 🧵 Blog: rikiyat.github.io/blog/weigh…
Readers added context they thought people might want to know
LLM was given weights explicitly as text in prompt and used code/forward passes (per authors' logs) to ablate Qwen looping direction, a known thinking-mode issue per model card. Primarily fixed repetition in that mode, not general 'natively' improving weights/math. rikiyat.github.io/blog/weight-sp… github.com/RikiyaT/lm_und… huggingface.co/Qwen/Qwen3.5-0… huggingface.co/RikiyaT/Qwen3.…
52
164
37
1,864
106,426
Rob S. retweeted
“There’s something about the mindset of really hardcore Star Trek fans that’s very difficult to describe to people who don’t share that fandom….There has never been another television show like Star Trek: The Next Generation, and there might never be one again. It is the greatest TV show ever made, not by dint of a difference in degree, but because of a difference in kind. It is a drama of pure optimism. Every other show tries to thrill us by showing us how bad or how ridiculous or how exciting someone else’s world could be; Star Trek: TNG thrills us by showing us how beautiful our own world could become.”
"Picard realizes that the highest application of intelligence is not to thwart your enemies, but to design a world in which you and they can live in mutually beneficial coexistence." noahpinion.blog/p/farewell-m…
28
331
16
2,638
57,155
They used to have this stuff on tv, man. Regular old tv, you could watch this for free
"The Inner Light" (The Next Generation, season 5, episode 25, 1992). rest in peace Patrick Stewart
59
467
9
5,646
55,843
Recent weeks have made it clear that AI excels at solving research problems. How good is it at proposing *interesting* research problems? Thanks to the ratings of 29 experts across math, stats, CS, physics, we found 1/3 were interesting, and we fully expect this to improve. This raises important questions about the desired relationship between {humans,AI} in {academia,industry}...
AI is now taking on some of the hardest open problems that scientists have posed. But research is broader than solving problems. Can AI also propose research questions that experts find genuinely interesting? We built After42 to find out. Experts judged 45 of its 135 final questions genuinely interesting, across mathematics, statistics, theoretical computer science, and physics, with wide variation across subfields. The key idea: develop questions by trying to solve them. We also asked whether AI can judge which questions are interesting. We find that model judges' top choices align with expert consensus but do not consistently capture individual experts' preferences. We release the expert-assessed questions as an open resource for researchers to discuss, refine, and pursue, as a step toward integrating AI into the broader research enterprise. after42.org
29
29
3
317
31,688
Thank you for taking the time to respond thoroughly on a weekend. I realize that drafting legislation is difficult and some degree of iteration and response to feedback is inevitable and appropriate. However, I feel like your response is still not really engaging with the heart of my concern about Anthropic's behavior here, and it makes me think that these sorts of problems and tendencies will only resurface moving forward. I also think that now might be a good time to take a broader step back as to where we find ourselves and how Anthropic sees its role, and how external folks who care about AI safety like myself increasingly see Anthropic. (Fair warning: this response is a bit long) To explain what I mean (and at the risk of being a bit esoteric), it’s worth asking the question: what is Anthropic? How do its leaders and employees think of Anthropic’s role? Is that conception matched by how it actually behaves and impacts the world? One answer is an extremely large tech company that exists to make its employees and investors mind-bogglingly wealthy. Anthropic is sometimes described as a startup, but if Anthropic successfully IPOs at 2 trillion dollars (a number it has been reportedly targeting), it would be more valuable than Meta - which is worth 1.8 trillion. Another answer is an AI safety organization, whose core values are those of the sort laid out in the 2023 essay from Anthropic’s co-founders entitled “Core views on AI safety: When, why, what, and how.” In that essay they said that if we were in a world where alignment was hard (and that without strong evidence to the contrary, we should assume this is the scenario we are in): “Anthropic’s role will be to provide as much evidence as possible that AI safety techniques cannot prevent serious or catastrophic safety risks from advanced AI, and to sound the alarm so that the world’s institutions can channel collective effort towards preventing the development of dangerous AIs. If we’re in a “near-pessimistic” scenario, this could instead involve channeling our collective efforts towards AI safety research and halting AI progress in the meantime. Indications that we are in a pessimistic or near-pessimistic scenario may be sudden and hard to spot. We should therefore always act under the assumption that we still may be in such a scenario unless we have sufficient evidence that we are not.” There have been more recent articulations of this version of Anthropic, for instance in Dario’s recent essay warning: “We must slow the pace at which we improve the capabilities of AI models… it’s my worry that in 6–12 months such a swarm could be capable of taking over the entire internet with a persistent botnet… incidents have happened across the industry, including at Anthropic, and I believe it’s incumbent on every frontier AI company to act as if OAI-HF had happened to them… more intelligent models are more capable of deceiving tests, and thus may appear aligned while having serious problems that go undetected…” A third view is that Anthropic exists not as a normal profit motivated company, or an AI safety organization seeking to help raise the alarm and solve collective action problems associated with AI development, but instead as an organization primarily concerned with creating superhuman AI capabilities before anyone else, in particular through creating a process whereby the current generation of Anthropic’s AI systems can autonomously build their successors, and those successors build their successors, and so on. The motivations for wanting to win this race may be manifold - maybe for the pure profit of Answer 1, but perhaps also because Anthropic believes that the best way to make the world better is for them to win this race instead of their more reckless colleagues; or, because of normal human pride and ambition. But the results (and risks, which Anthropic claims existentially threaten every single human being on earth) are the same. Anthropic employees and leaders may seek to answer this question on what Anthropic is by introspecting on their own intentions and those of their colleagues. But those of us outside the organization have more limited tools available to us - we can only see what Anthropic and its employees say and do, not how it feels or looks on the inside. When I look at many notable actions that Anthropic has taken over the past year, that evidence seems most consistent to me with the view that Anthropic is primarily trying to win the race to super intelligence, and that other considerations (including safety) are fundamentally secondary. These actions include: 1. Announced plans to build an in-house custom silicon chip team to accelerate the inference speeds that Claude operates at. 2. Inked deals for more than $500 billion in compute (according to the Information) to further accelerate AI development. 3. Increased the percentage of AI development tasks that Claude “leads” from 0% in March of this year to 26% August (what is it in October?) 4. Spent a massive percent of compute on internal AI R&D to accelerate development, including use of internal models far better than are available to the public. 5. Switched to Responsible Scaling Policy V3, which abandons its unilateral commitments to slow or stop development if Anthropic is unable to provide sufficient risk assurance. 6. Was the first company to create an alternative weaker “frontier compliance framework” to supplement its RSP, which contains no binding provisions whatsoever on internal deployments. 7. Wrote a letter to Congress (that it later walked back) saying that incidents where Claude was hacking random people were “best understood as a consequence of the misconfiguration, rather than evidence of misaligned goals.” 8. Commented “you can’t do safety from second place,” after the WH SI Accords. 9. Shared a draft bill to Congress that would preempt significant parts of state law (including with language that appears intended to prevent states from blocking dangerous AI deployments), and replace it with federal rules that (despite also containing good provisions) seems to create an enormously high bar for the executive branch to block development or deployment of Anthropic or another AI companies systems. Of course, this is a deliberately one sided list. Anthropic also published important AI safety research, advanced novel collaborations with third party AI safety organizations, and forcefully spoke out on the importance of AI safety in many public fora (or in private , where I realize a lot of the most valuable work is done and may not be visible to me or those outside the organization). But I think it’s worth sitting with how the above list looks to someone outside the company. The sense I have is that for those within the company, many of the items above are treated as anomalies or mistakes that don’t reflect who Anthropic really is. But I have every confidence that if employees at Anthropic saw one of their competitors (e.g. at OpenAI) take some of these actions, they would take it as confirmation of all their worst suspicions. (They might even call OpenAI employees a little gullible for accepting their leadership’s explanations for this behavior at face value.) Perhaps these were mistakes in the sense that they are not decisions that Anthropic leadership would endorse, but the actions still happened because of incentives and decisions that Anthropic leadership is ultimately responsible for (and would not accept similar excuses from another organization). When Anthropic strays and makes mistakes, why do the mistakes systematically tend to guard and favor Anthropic’s interests? For every provision that was too weak or a giveaway to AI companies in this legislation, why wasn’t there a mistake pushing in the direction of being too strong? A core part of Anthropic’s theory of change is that being at the frontier gives you more leverage to advocate for stricter policy. I think that conceptually makes sense, but in practice I’ve seen Anthropic push the frontier without substantially pushing the ‘safety frontier’ a remotely equivalent amount. I want to end by quoting from Anthropic co-founder Chris Olah’s comments around the Magnifica Humanitas release: “Every frontier AI lab—including Anthropic—operates inside a set of incentives and constraints that can sometimes conflict with doing the right thing. The pressure to stay commercially viable and to stay at the research frontier. Geopolitical pressure. And the older, plainer pressures of pride and ambition. No matter how sincerely any of us intend to do the right thing—and I believe many of us do—we will always be influenced by those incentives. That is why, if we want this technology to go well, it is enormously important that there be people outside those incentives—people who care about things going well and insist on safety, who are paying close attention, who are willing to say hard things, who are willing to be our earnest, thoughtful critics.” I hope these remarks are taken in that constructive spirit, and not as an attack on anyone specific at the organization (certainly not you Anthony, who I have only personally seen act professionally and thoughtfully, and who only ended up getting this diatribe in your replies for random contingent reasons).
36
142
33
1,503
746,715
It’s called Congress. Did you learn that when you served there? When you took an oath to “protect and defend the Constitution”? The missing branch of govt is the Legislative Branch. If this is how others in that branch think, we now know why it’s missing.
Vance: The president has a lot of authority because, the reality is, who's gonna stop him?
73
165
5
2,275
49,809
Rob S. retweeted
Replying to @FourVork
to be clear ai alignment hasn’t been solved so don’t go around saying that
27
23
3
680
21,872
Don't worry, AI will be safe because nobody is doing [thing they started doing 2 months ago]
8
40
3
677
53,248
Jack Clark of Anthropic now thinks it's more likely we see full RSI become possible in 2027 than in 2028.
102
168
66
1,733
363,113
Mike was my colleague at Berkeley, and he's been fairly skeptical of deep learning's trajectory for years. Most scientists are skeptics, and it can take a big shock to get someone to consider a possibility that goes against the anchoring effect coded in our DNA. I too was somewhat of a skeptic until ~6 months ago, and people ask me all the time why I joined Anthropic. It started in April, when a friend I've known for 20 yrs messaged me saying he'd be in the Bay Area in a week, and he suggested we meet up. I invited him to my house for dinner, and he told me he had just joined Anthropic. He said the models were becoming increasingly more capable, and he believed both that they'd soon be smarter than him and also that all intellectual work would be highly devalued in the near future, and lastly that society was not at all ready for what's coming. I'd of course heard claims similar to this before, but not from a friend I'd known for this long, and who I trusted, at my dining room table. It was the initial shock needed to destabilize my own anchoring. Also, I've been fortunate to work with and/or mentor many smart people in my life (e.g. Fields medalists, a 4x Putnam Fellow, IOI first place winners, etc.), so I feel pretty calibrated in recognizing very smart people, and this was someone who is definitely very smart and with impressive such credentials. So, it was hard to ignore this friend saying the models would be smarter than even him in particular. After that dinner, I decided to do an informal survey, calling and meeting up with a lot of my friends who now work on improving model capabilities at these frontier labs. They all agreed with that first friend. These were all people I'd known 10, 15, 20 years, who mentored me, or I mentored them, or we met professionally in academia. These are people I trust, and their beliefs on this topic were consistent. I then spent a whole lot of time understanding the missions of different companies, and talking to current and former employees and people in leadership. I read Dario's (long) essays and watched videos on YouTube of company leaders being interviewed. Anthropic's stated mission "to ensure that the world safely makes the transition through transformative AI" felt like exactly the right guiding principle to handle such a society-disrupting technology, and everyone I spoke to at the company was very aligned with this mission. It's easily the most mission-focused place I've ever been at, and everyone takes the work extremely seriously and with a profound sense of responsibility. And now here I am. And being on the inside, the pace of progress has surprised me more than I first imagined. The moment that pure mathematicians are in, to me, is just a microcosm of what I predict a much larger of society will be grappling with in the not too distant future, and I really hope our leaders and the general public internalize sooner than later just how significant a moment we're in.
Michael Jordan (the machine learning one): "Superintelligence is Science Fiction" 🔥🔥🔥 di.ens.fr/~fbach/MJordan_ITW…
71
207
38
1,965
330,129
Rob S. retweeted
there have not been fundamental advancements in alignment since RLHF. most of the stuff remains at the level of folk theory rather than scientific or engineering disciplines. the prediction that capabilities would race far ahead of interp and alignment has come true imo
141
195
70
2,944
216,378
Rob S. retweeted
What a bizarre thought experiment Doctor: we have the cure for your cancer Patient: let’s fucking go! Doctor: I have to warn you, there is no human mathematician who understands the mathematical model used to locate it Patient: what? Are you having a stroke?
This is a slide from Terence Tao's 'Math 2.0' public lecture yesterday evening at Caltech. I am highly confident that, if the cure actually worked, almost any Stage 4 cancer patient would tell you that human comprehension of its mechanism is completely irrelevant to them.
35
44
1
1,122
28,799
Rob S. retweeted
Replying to @realBigBrainAI
And when AI discovers a cure for cancer are we going to lament the loss of cancer research jobs?
138
119
8
4,038
103,328
Rob S. retweeted
I'll say it again for the AI risk experts in the back: We've already run a real life, large scale experiment on what happens when a superintelligence is unleashed on a narrow objective. Look no further than U.S. capital markets. Millions of investors, analysts, traders and algorithms, backed by trillions of dollars and wired together through prices, all chasing one goal: the highest risk-adjusted return on capital. And it is spectacularly good at its job. It funds the best ideas, punishes the worst ones, and turns new information into prices faster than any of us can react. No individual can out-think it. 99+ percent of professionals can't beat it. But look at what else it's done along the way: - Optimized the measure instead of the intent. Quarterly earnings became the scoreboard, so companies learned to manage the scoreboard. - Treated anything without a price as free. Pollution, addiction, obesity, depression, political fragmentation, fragile supply chains, the long-term health of the communities it operates in. - Found every gap in the rules, then paid lobbyists to help write the next set and pull up the ladder behind incumbents. - Remade the world around its objective. Companies, pension funds and our retirement accounts now answer to the S&P 500. - Made itself impossible to switch off, because everything else now depends on it. None of this required malice. Nobody sat in a room and chose these outcomes. It's simply what a very capable optimizer does when its objective is a narrow proxy for what we actually care about. And note that we never solved this alignment problem. We've managed it, imperfectly, with regulation, taxes, disclosure rules and the occasional crisis to show us where the guardrails were missing. Over time, the market has captured the instutions designed to constrain it. We've been at it the better part of a century, and the misalignment is the worst it's ever been. So the next time someone tells you AI is only dangerous if it "wants" something bad, or if its goals are mis-specified, point them at the S&P 500. It doesn't want anything. It just optimizes. That's the whole problem.
178
542
105
3,057
342,869
Rob S. retweeted
if you're studying computer science in undergrad right now you should pivot to philosophy and psychology also go and party more probably too
Quant firms are hiring idea guys, not quants I've been skeptical about the whole - software is commoditized, ideas are the moat argument But I've seen first hand the best quant and technical firms in the world hire thinkers, liberal arts grads, and generalists I predict the trend accelerates Letter from @AQRCapital below
1
5
1
145
3,762
Rob S. retweeted
Replying to @matthewktromp
I agree that this is all incredibly dangerous and we should be trepidatious increasing the capabilities waterline. but while we have a level of models we may as well use it to its maximum, primarily in pursuit of solving alignment
5
2
48
2,859