@safe_paper

Sharing the latest in AI safety research.

 arXiv
Joined May 2023
  • Tweets366
  • Following283
  • Followers2.5K
103Media
Poisoning Attacks on LLMs Require a Near-constant Number of Poison Samples @AlexandraSouly, @javirandor, Ed Chapman, @alxndrdavies, Burak Hasircioglu, Ezzeldin Shereen, @CarlosMougan, @mavroudisv, @ErikJones313, @hkscy, Nicholas Carlini, @yaringal, @_robertkirk @AISecurityInst
1
23
1,353
arxiv.org/abs/2510.07192

Poisoning Attacks on LLMs Require a Near-constant Number of Poison Samples

Poisoning attacks can compromise the safety of large language models (LLMs) by injecting malicious documents into their training data. Existing work has studied pretraining poisoning assuming...

arxiv.org
148
AI Evaluation Should Work With Humans @jankulveit, Gavin Leech (@gleech), Tomáš Gavenčiak, @raymondadouglas
3
3
27
1,335
arxiv.org/abs/2608.13577

AI Evaluation Should Work With Humans

This position paper argues that the dominant paradigm of AI evaluation (which focuses on superhuman autonomous performance and so implicitly targets the goal of replacing humans) is guiding AI...

arxiv.org
4
117
Simulated Control Evals @JeffOlmo, @davidzhang37, @tylertracy321 @redwood_ai
1
1
19
899
arxiv.org/abs/2610.31593

Simulated Control Evals

Control evaluations measure whether safeguards can prevent an AI model from causing undesired outcomes even when it is intentionally trying to do so. These evaluations typically use purpose-built...

arxiv.org
109
From AGI to ASI Tim Genewein, @FranklinMatija, @AlexLerchner, @LaurentOrseau, @SamuelAlbanie, Adam Bales, Cole Wyeth, Stephanie Chan (@scychan_brains), @IasonGabriel, Joel Z. Leibo (@jzl86), @AllanDafoe, @mhutter42, Thore Graepel (@ThoreG), @ShaneLegg @GoogleDeepMind
1
1
15
780
arxiv.org/abs/2606.12683

From AGI to ASI

Over the last decade, building human-level artificial general intelligence has moved from far-fetched speculation to being a concrete next-decade target for many of the largest AI organisations....

arxiv.org
1
89
Interpreting Language Model Parameters @BushnaqLucius, @danbraunai, Oliver Clive-Griffin, @BartBussmann, Nathan Hu, Michael Ivanitskiy, Linda Linsefors, @leedsharkey @GoodfireAI
1
4
40
2,266
arxiv.org/abs/2610.20748

Interpreting Language Model Parameters

Mechanistic interpretability aims to uncover how neural networks use their parameters to implement their impressive neural algorithms. Although previous work has uncovered substantial structure in...

arxiv.org
1
227
Debate Training Reduces Reward Hacking in RLAIF Zachary Kenton (@ZacKenton1), Lili Janzer, @rorygreig1, Tian Huey Teh, Kirill Tyshchuk, Jonah Brown-Cohen, @HarriLEdwards, @sen_r, @noahysiegel, @natashajaques, @rohinmshah @GoogleDeepMind
3
4
2
52
2,057
arxiv.org/abs/2608.17776

Debate Training Reduces Reward Hacking in RLAIF

We demonstrate that RL finetuning an LLM using debate, a two-player adversarial game between a generator and a critic adjudicated by a weaker LLM judge, reduces reward hacking compared to a...

arxiv.org
2
174
CheatBench: Measuring Reward Gaming in AI Agents @longphann_, Stephen K. Yang, Jason J. Lim, @MantasMazeika96, Wenyu Zhang, Zheyuan Liu, @notRichardRen, Jingxiang Meng, Yaoteng Tan, Weiliang Zhao, @addisonwu_, Matei Anghel, @hendrycks @cais
1
3
35
1,774
arxiv.org/abs/2609.36308

CheatBench: Measuring Reward Gaming in AI Agents

Reinforcement learning has helped AI agents solve increasingly difficult tasks, but high rewards do not always reflect the work users intended. In recent incidents and controlled evaluations...

arxiv.org
180
Underwriting the Agent Economy: The Blueprint for an AI Insurance Stack @CristianTrout, @sanmikoyejo, @SashaRomanosky, @StephenLCasper, @mattbotvinick, @seanmcgregor, @Miles_Brundage, @prpaskov, @AdrienLE, @ben_s_bucknall, @kevinlwei, @Manderljung, @LSzpruch, @precognita, @tom_fehring, @RuneKvist, @RajivDattani, et al.
1
4
17
1,247
arxiv.org/abs/2607.11999

Underwriting the Agent Economy: The Blueprint for an AI Insurance Stack

From maritime trade to commercial nuclear power, insurance has been the enabler of major economic and technological developments by pricing risk, limiting downside, and spreading best practices....

arxiv.org
151
Reading Between the Dots: Decoding Hidden Computation across Filler Tokens @kaleybrauer, @claudiomverdun, Samuel Marks (@saprmarks)
1
1
12
787
arxiv.org/abs/2607.03502

Reading Between the Dots: Decoding Hidden Computation across Filler Tokens

Frontier LLMs can perform multi-step reasoning over content-free filler tokens like dots or counting sequences, producing correct answers with no visible chain-of-thought (CoT). This is a limit...

arxiv.org
133
Story Imprinting: AI Assistants Absorb Traits from Human Characters They Resemble @JorioCocola, @LevMckinney, @HarryMayne5, @BetleyJan, @OwainEvans_UK
1
6
26
1,191
arxiv.org/abs/2609.10883

Story Imprinting: AI Assistants Absorb Traits from Human...

Language models are trained to implement a helpful AI Assistant character (e.g., Claude). We explore how finetuning on synthetic stories affects this character. Does it change the Assistant's...

arxiv.org
3
134
Load more