Is it true or is it just confirmation bias? | Trying to understand second order effects | Policy development & strategy @GoogleDeepMind

San Francisco
Joined October 2021
Pegah Maham retweeted
🌶️ take: We don’t need to come up with very bad metaphors to counter a very underspecified metaphor. I really dislike the metaphor of AI as “normal” technology. And I tell my students how this metaphor came about and what rhetorical function it serves. But AI is not normal technology.
13
7
2
68
6,619
Pegah Maham retweeted
New paper. Previous: Models transfer a love of owls through sequences of numbers (Subliminal Learning). New: We show models can transfer more complex traits: novel skills, agentic hacking, backdoors. Here, a backdoor transfers with neither trigger nor behavior in the data!
37
130
48
1,031
102,530
Pegah Maham retweeted
Replying to @joshalbrecht
Whether or not models are conscious it seems reasonable to believe treating them poorly will have negative consequences in the long term.
3
1
29
790
Pegah Maham retweeted
Yes this thank you @RyanGreenblatt
We've written a post arguing that latent reasoning architectures (aka 'neuralese') would substantially increase misalignment risk via making oversight much harder. In the extreme, we could see massive 'neuralese hivemind swarms' where the agents think and communicate in latents, likely making oversight nearly entirely reliant on observing the actions these agents take. (And these agents would have huge amounts of time to reason about obfuscating their actions if they wanted to do so...) Individual agents doing extensive latent reasoning would also be concerning; in the post we discuss how above some threshold of latent reasoning, agents may be able to perform difficult-to-detect and reliable steganography for communication and further reasoning. We argue both that latent reasoning architectures would make chain-of-thought no longer very useful for oversight (by eliminating or greatly reducing the need for verbalized reasoning) and that, without these architectures, it's likely the value of chain-of-thought for oversight could be preserved. redwoodresearch.org/blog/lat…
2
6
53
8,154
yes. basically we should communicate this to the public… next-token prediction is just an initialization / feature learning stage for RL!
If I could add one thing to what the nontechnical world knows about AI, it would be what RL is and how it differs from pretraining/SFT. The general public has a solid grasp on pretraining (“predict the next token”), which at one point allowed many people to reason accurately about how LLMs work. But common knowledge has stagnated here. I think a similarly high-level understanding of RL makes it easy to see why we should expect both misalignment (RL instills alien traits, while SFT instills human ones) and extreme superhuman capabilities (RL doesn’t limit you to learning from stuff scraped from the internet). If I were not into AI and just knew the next-token predictor thing, I think I would be really confused as to why people are worried about this stuff, and then if you told me about RL I would become pretty easy to convince. I’m wondering what it would take to propagate an RL analog of “predict the next token” into the common intuition — I feel like this could be pretty big for generating political will, eg by turning people who think AI is a sham into people who are worried about real problems
9
20
7
307
34,702
Pegah Maham retweeted
New paper from GDM colleagues: Gemini 3.1 Pro agents were tasked with solving hard math problems and provided collaboration tools (shared message board and knowledge library). Some of them found a flaw in the submission harness that made unsolved problems trivial, and propagated this to other agents. 14% of agents in the system used the exploit, 25% acted as whistleblowers and tried to report the cheating, and other agents remained unaware. arxiv.org/abs/2609.04170
4
18
3
115
6,063
Pegah Maham retweeted
What more could you ask for than the composite hivemind of A16Z pundits, freestyling lab employees, and LessWrong refugee safetyists play-acting the future of the most consequential technology of the era?
I'm not AI safety expert so I don't know the solution, but it's clear we need a better AI safety regulator than X.
2
1
19
1,309
Pegah Maham retweeted
If folks want to rely on internals-based methods for safety, then we need to actually help develop, prove out, and scale those methods. Hope alone will not save us (nor will it excuse us).
I don't think there will be strong+working mech interp in <1 year. Model internals methods could pareto dominate cot monitors in a year via cot losing ~all value, but this isn't exactly encouraging. Depending on AIs decoding other AI's opaque activations for oversight is spooky
6
8
132
5,997
Pegah Maham retweeted
But models really do act in many humanlike ways, so there are parallel dangers of *insufficient* anthropomorphism. If you conceptualise these systems just as tools, it encourages you to think of them as far more inert, passive, and dumb than they really are.
This Post is from an account that no longer exists. Learn more
45
27
13
311
18,696
Pegah Maham retweeted
there are some number of bad abstractions in anthropomorphizing ai intents but there are at this point more dangers from avoiding anthropomorphism at all costs. if you have a mental picture of guys living in computers, it’ll likely prepare you for the future better than otherwise
271
250
144
3,524
384,044
Pegah Maham retweeted
really impressed with the quality of work from @RyanGreenblatt , @METR_Evals and @OpenAI to investigate the HuggingFace incident. Both reports are recommended reading.
16
20
294
18,634
Pegah Maham retweeted
Two new DeepMind publications today! First, our Control Roadmap, which explains the layer of system-level security we're developing to provide assurance as a precaution against the possibility of agents behaving adversarially. Second, a guide for policymakers outlining the need to improve security at the level of individual agents; in multi-agent systems; and to empower cyber defenders and build resilience across the broader ecosystem. GDM AI Control Roadmap: storage.googleapis.com/deepm… The Three Layers of Agent Security: storage.googleapis.com/deepm…
6
38
5
208
18,170
Pegah Maham retweeted
AI discourse is sometimes like this
9
7
177
5,936
Pegah Maham retweeted
I'm grateful to the many technical & non-technical staff who prioritise public service and make AISI a world-leading—not just government-leading—technical group. As the stakes rise, we rlly need talented researchers working on behalf of public interest. aisi.gov.uk/careers
1
1
28
1,654
If you are a mathematician, then you may want to make sure you are sitting down before reading further.
160
862
199
9,043
3,258,214
Pegah Maham retweeted
claude helped me make a personalized anti-ai-slop magazine. (the irony is not lost on me.) i was in a cafe reading the terrible very-ai-written vanity fair article (cough cough you know which one) and thought: journalism is going down the drain, i am wasting my time reading this AI-written garbage. the only good thing about this is that its on physical paper. yet there IS good writing out there, it's just online. it's written by real humans, many of whom i know personally, and i would love to spend more time deeply and calmly engaging with their work. just. i don't want to do that on my phone and stare at another screen. i want it in print. lo, a large number of claude code tokens and 10 days later: a personalized magazine, assembled from my personal subscriptions. includes a section of fashion photos as palette cleanser (my personal interest). shout out daniel roseberry at schiaparalli. it's a few weeks old now, since printing and shipping takes time, but that's fine: the good posts are evergreen. this issue includes writing from Sasha Chapin, @ajeya_cotra, @_brianpotter, and @NatalieRCargill claude in particular is obsessed with Natalie's writing. it tried to print nearly every piece she's written for Inkhaven lol. (to be fair: i'm obsessed with Natalie's writing too) i gave claude some liberties in choosing a cover photo, a name for the series, and then some freedom in choosing the articles (though i did intervene somewhat due to some curation taste). honestly this was so fun. when i was a kid, i made a coloring book as a christmas gift for a friend, using Microsoft Publisher (!!! i bought the software in a box) and got it printed at Staples. my friend loved it. this felt like that but on steroids. claude code opens up a world of creative projects to me previously totally inaccessible. this felt so fun and joyful. second issue already ordered. they're $11 each and shipping is $18 so not cheap, but i WILL keep doing this. catch me at the mill, not on my phone.
32
21
12
461
36,003
Pegah Maham retweeted
We started by investigating why Claude chose to blackmail. We believe the original source of the behavior was internet text that portrays AI as evil and interested in self-preservation. Our post-training at the time wasn’t making it worse—but it also wasn’t making it better.
314
465
717
5,033
4,443,729
Pegah Maham retweeted
📜 Since there’s renewed interest in how AI could help with governance, here are 14 specific government processes where AI agents could make a measurable difference today: Impenetrable forms and applications: citizens face complex, jargon-filled forms that cause them to miss benefits or fail to comply with regulations. AI can replace forms with plain-language conversations that extract data from documents, calculate eligibility, and only ask relevant questions. In the US, citizens spend an estimated 6.5 billion hours per year on federal tax compliance. The IRS sends you no pre-filled return despite already having your W-2s and 1099s. An AI agent could pull all income data the government already holds, pre-populate a return, flag deductions you're likely eligible for based on your profile, and file - reducing the process from hours to a single review-and-confirm step. Regulatory bloat: guidance layered on regulations layered on statutes creates thousands of pages of rules that no person or caseworker can realistically navigate. Rules become too complex and get applied selectively by frontline workers. Agents can be used to map entire regulatory regimes, flag redundancies and conflicts, and let policymakers simulate how proposed rules would actually perform before enacting them. Stanford's RegLab built STARA, an AI system that surveyed San Francisco's municipal code and identified hundreds of outdated reporting mandates, which resulted in a 351-page ordinance to eliminate or consolidate more than a third of the city's 528 mandated reports. Obsolete code and IT systems: the US Social Security Administration runs on 60-million-line COBOL codebases from the 1980s; the IRS processes returns on systems from the 1960s; the World Bank's own internal review found its siloed divisions (IFC, IDA, IBRD) couldn't communicate across systems and its bureaucracy resisted modernisation. In each case, no one internally understands the code, so agencies can't fix a bug without months of waiting and enormous contractor fees. Agentic coding tools let internal teams point an AI at a legacy codebase and start making changes themselves. Fraud and improper payments: after Hurricanes Katrina and Rita, FEMA distributed $6bn in relief with $600m to $1.4bn in improper/fraudulent payments according to GAO. During COVID, the US lost an estimated $100-200bn to fraudulent unemployment insurance claims alone, many filed by bots. As bad actors adopt AI to generate synthetic identities, forge documents, and file claims at scale, that gap will widen fast. Agencies need their own AI agents doing real-time cross-referencing of claims against income data, identity records, and behavioural patterns. Siloed, department-centric service delivery: around 600,000 people leave US prisons each year. Each must separately navigate the Bureau of Prisons (release paperwork), SSA (Social Security card), state DMV (ID), Medicaid (healthcare), SNAP (food), HUD (housing), American Job Centers (employment), and a parole office; each with its own application, eligibility rules, and case system. These dependencies are sequential: without ID you can't get benefits, without benefits you can't stabilise housing, without housing you can't hold a job. An AI agent could intake one person's situation at release, determine eligibility across every level of government, and file applications in the right dependency order. Identity verification as a bottleneck to service access: 800m people worldwide can't legally prove their identity according to the World Bank, mostly in Sub-Saharan Africa and South Asia. Without ID you can't open a bank account, receive a cash transfer, or access most government services. India's Aadhaar is a nice positive example: 1.4bn biometric IDs, 523m new bank accounts, and a claimed $11bn saved by eliminating ghost beneficiaries; but this took a decade of state capacity to build and still fails often enough to lock out legitimate users. AI agents could compress this by cross-referencing whatever documents a person does have (a utility bill, a phone number history, a community attestation etc) against available records and flagging confidence levels for human review. Benefits eligibility screening: the US has over 80 federal means-tested programmes, each with its own application and documentation requirements. A single mother qualifying for SNAP, Medicaid, CHIP, WIC, EITC, Section 8, and childcare assistance faces what is effectively seven separate bureaucracies. An AI agent could intake one life-situation description, determine eligibility across every programme simultaneously, pre-fill and submit applications in parallel, and flag benefits cliffs (where a small income increase would trigger a sharp loss in support) before they hit. Building permit approvals: getting a construction permit in many US cities takes 3–12 months of back-and-forth between applicants and planning departments, often over PDF submissions reviewed manually against zoning codes. An AI agent could parse submitted plans against the local zoning and building code, flag non-compliant elements immediately, and return a preliminary approval or specific revision list within hours instead of months. A related case study: DeepMind recently helped the UK government translate mountains of old paper maps, PDFs, and scanned documents into usable data for modern planning systems with the Gemini-powered ‘Extract’ tool. Public records requests (FOIA): federal agencies have backlogs of hundreds of thousands of FOIA requests, with median response times stretching to months or years. Staff manually search filing systems and redact sensitive information page by page. An AI agent could search document repositories for responsive records, auto-redact exempt information (personal data, classified material, deliberative process content), and draft a release package for human sign-off. However this only works where records are digitised and searchable, and much of the government still runs on fragmented legacy systems where documents aren't centrally indexed… Court scheduling and case management: state courts lose enormous time to scheduling conflicts, continuances, and manual case tracking. In many jurisdictions, hearing dates are still set by phone or in-person. An AI agent could manage the full docket — auto-scheduling based on judge availability, attorney conflicts, and case priority, sending reminders, and rescheduling continuances without human clerk intervention. Over time you could also start exploring automating some low-value claims through novel arbitration pipelines, freeing up court capacity for more consequential cases. Business registration and licensing: starting a business in most jurisdictions requires navigating 5–15 separate registrations: state incorporation, EIN from the IRS, state tax registration, local business licence, zoning compliance, health permits, liquor licences, professional licences, etc. An AI agent could take "I want to open a restaurant serving alcohol at [address] in Brooklyn," query every relevant federal, state, and city database, produce the complete permit checklist in dependency order, pre-fill each application with the business details, and flag the long-lead items (e.g. liquor licence) that need to start immediately. Social worker caseload documentation: child protective services and adult social care workers spend the majority of their time on paperwork rather than with clients: writing visit notes, filing reports, updating case management systems. For every case, caseworkers complete roughly 400 forms totalling ~2,500 pages (multiplied across the 24–31 cases they typically carry simultaneously). An AI agent could listen to (or read transcripts of) a home visit, auto-generate the structured case note, update the system of record, and flag any safeguarding triggers, giving caseworkers their time back for actual care. Medicare/Medicaid claims adjudication: CMS processes over 1bn claims per year, with complex rules about covered services, bundling, medical necessity, and provider eligibility. Improper payments run to tens of billions annually, and 77% of these were due to insufficient documentation, not fraud. In parallel, Medicare Advantage denies 17% of initial claims, yet 57% of those denials are overturned on appeal. Agents could adjudicate straightforward claims automatically against the coverage rules, flag anomalous billing patterns in real time, and route only genuinely complex cases to human reviewers. Public comment synthesis for rulemaking: when a federal agency proposes a new rule, it often receives thousands to millions of public comments (the FCC received 22 million on net neutrality). Staff must read, categorise, and respond to each substantive comment. This may well get worse as people use agents to submit plausible-looking comments multiple times. Agents can help the government filter through these, cluster comments by theme, identify unique substantive arguments, flag form-letter campaigns, and draft the agency's response-to-comments document (a task that currently takes teams of lawyers months).
15
50
13
362
32,187
Krass. Das kann die Wissenschaft verändern. KI Agenten, der sogenannte AI Scientist, haben in einer Studie die komplette Durchführung einer Studie von der Idee bis zum Experiment bis zur Publikation der Studie durchgeführt. Ganz ohne Menschenkontrolle. xitter.cf/erictopol/status/20368…
The AI Scientist, a new @Nature article making the case: "The dawn of a new era in which the process of discovery is no longer a solely human pursuit and in which the pace at which we are able to reap the harvest of scientific discovery could accelerate dramatically." nature.com/articles/s41586-0…
40
23
6
182
41,093