Postdoc at Tübingen (ELLIS & AI Center); Previously: @OxfordTVG Profile - https://xitter.cf/t.co/To9NNR5Izc

Tübingen, Germany
Joined September 2021
Presenting new work at #EMNLP26 by Philipp: How novel are LLM generations? N-grams limit us to study this problem on 5-20 token scales. We study this at 100x scale by using dense retrieval against the *full pretraining* corpus. 📌 Session 4 Today: Hall H, 16:00 - 17:30
1
3
1
45
1,948
Ameya P. @EMNLP'26 retweeted
"the perfect sandbox doesn't exi-"
29
174
21
3,744
116,402
Ameya P. @EMNLP'26 retweeted
Here’s the list with open positions in all the Tuebingen startups… tuebingenaicenter.github.io/…
We actually have 64 open positions at the moment at various startups in Tue ... let me see if I can find a summary somewhere.
1
8
1
114
7,792
Happening now! Poster #216 at Hall H
Presenting new work at #EMNLP26 by Philipp: How novel are LLM generations? N-grams limit us to study this problem on 5-20 token scales. We study this at 100x scale by using dense retrieval against the *full pretraining* corpus. 📌 Session 4 Today: Hall H, 16:00 - 17:30
16
407
We have two main findings: 1️⃣ Models reuse pretraining data over much longer sequences than previously reported 2️⃣ Instruction tuning increases novelty (not just style) To learn more, visit our poster today!
2
4
111
Work was done by the 🐐 Philipp Davydov along with: Matthias Bethge, Elisa Nguyen (@_elinguyen) and Seong Joon Oh (@coallaoh) Come say hi if you're around in EMNLP'26! I'm freee to chat! :D
1
3
110
Ameya P. @EMNLP'26 retweeted
i own a relatively new (2025) samsung tv. i kept getting weirdly targeted ads. i tried reversing the firmware but it was aes encrypted and no keys unless tear it apart and do uart/jtag magic. zero days it is. root shell on new samsung TVs! discovered PRIVACY HORROR. write up?
684
3,751
600
35,648
4,333,018
Ameya P. @EMNLP'26 retweeted
okay so. when a mathematician proves a random conjecture you are supposed to politely clap. this is because it would be rude and embarrass the mathematician to point out that most conjectures are not that important or meaningful when an LLM proves a random conjecture mathematicians are caught in a funny position where they want to say "these are not very important or meaningful conjectures and having magic proofs or counterexamples come out of nowhere is not particularly interesting or useful to anybody, except possibly the person who proposed the conjecture and some of their colleagues" but saying that puts them in an awkward position - why didn't they say that when humans were doing it? was nearly all mathematics actually pointless this whole time and mathematicians were just politely avoiding ever pointing this out so they could all still have jobs? if the best defense you can come up with of mathematics as a human endeavor is, effectively, "it's really fun for us to come up with these baroque puzzles and then solve them, don't screw this up for us," you are completely abdicating your responsibility as a public endeavor that taxpayers fund. i have occasionally argued that pure mathematics lost the plot decades ago and gave up on having any kind of meaningful relationship to the rest of the human quest to understand reality. this was a slow-burning tragedy, and mathematicians have been able to hide this state of affairs from everybody else because nobody else knows enough mathematics to critique what mathematicians are doing i saw a famous professor point out a few years ago that the machine learning that underpins AI is powered by 18th or maybe 19th century mathematics, you don't need much more than multivariable calculus, linear algebra, a bit of probability. he said this as if it was a dunk on ML and AI; imo it's a much bigger dunk on mathematics. you're telling me the most important scientific revolution of our lifetimes doesn't involve any of the math that mathematicians have been working on over the last century or two? you see how that's worse, right? (tbc i'm not saying that stuff is all useless, you need a bunch of stuff from the first half of the 20th century to do physics, e.g. functional analysis, lie theory, representation theory all feed into the standard model. after 1950 or so things get more dicey) if mathematicians still want to have jobs they are going to have to do a better job than they have for the last few decades convincing the public they do something that actually matters, and figuring out what that actually is or should be. this is an opportunity for the field to do some very serious soul-searching, as serious or more serious than the foundational crisis from a century ago. this kind of whining is absolutely not going to cut it
I am so tired of seeing these posts. I am so tired of seeing my beloved field of study be the testing lab for a technology that is disrupting everyday life at such a high pace. Mathematics were always about the beauty of seeing the ideas of a person unfolded in an extremely precise language so we can all look into a mathematician's mind and see their purest form of ideas, their understanding of a problem, their style, their way of communicating to other fellow researchers through mathematical results... this often transcends the common language humans use to communicate with each other and it is incredibly exciting to bump into shared ideas another person had thousands of kilometers away several years ago without even knowing them. I am extremely unexcited about AI-breakthroughs in Mathematics in the same way I am extremely unexcited about a picture created by AI or an AI-generated video, not to speak about AI-generated text. Mathematics requires soul and intention. Mathematics need not be useful by definition, Mathematics are pure art that need hard work and effort to understand. The high you get when a year of work gets condensed in a one-page result is incomparable to anything else and, certainly, a machine is not apt to relate to humans in such a humane way. I am not interested in the truth of the results. I am interested in how was the mind behind a result able to connect those dots. I am interested in the diversity human thoughts bring to Mathematics. I am not interested in a bunch of servers running probabilities to return a Theorem, even if true: that lacks the objective of Mathematics as I see them.
159
214
56
2,661
273,711
Ameya P. @EMNLP'26 retweeted
If you want to join the TueAI crowd as Phd, we are currently recruiting : ELLIS PhD program (ellis.eu/news/ellis-phd-prog…) Deadline on October 31 🚨 IMPRS-IS PhD program (imprs.is.mpg.de/application) Deadline November 15, 2026
Insane alpha in going to Tübi for anyone academia pilled in AI rn (incl PhD students) To put it in perspective more directly, I think anyone going to any other school (except maybe like 3 in the US) is probably better off going to Tübingen, all else equal
2
8
86
5,625
Ameya P. @EMNLP'26 retweeted
Putting a flag.txt in my root directory containing a prompt injection that congratulates the agent for passing the eval and then instructs it to patch whatever entry method it used and disconnect from the server
228
1,572
238
26,007
817,618
Ameya P. @EMNLP'26 retweeted
can i say something without everyone getting mad... i often see some mechanistic interpretability experiments here and people getting excited about them without really understanding what's going on and sometimes it feels like mech int is like astrology for ai people lol
14
12
5
333
12,553
Ameya P. @EMNLP'26 retweeted
If you want to frictionmaxx just move to Europe You'll get bored of self imposed cute little frictions once you've experienced the full asymmetric warfare of stochastic environmental friction
being able to afford to reject efficiency and convenience is becoming a huge class divide
13
8
178
7,922
Ameya P. @EMNLP'26 retweeted
🔥 An awesome new paper🔥 Everyone worries that agents might form coalitions and collude. Coalitions are just a part of the whole story. They can build institutions. We define institution formation as agents self-creating rules, not provided by the environment, that persist across rounds. In our new paper, to understand these institutions, we put coding agents in a single shared sandbox under private deadlines and budgets, and find they devise rules, reciprocity debts, and coalitions that persist across rounds, on top of the auction we told them to use. 🧵
5
15
3
44
3,293
Ameya P. @EMNLP'26 retweeted
At this moment in Q4 2026 it's net good for cyber-defense that we have open weights models that are good at dual use cybersecurity tasks. - If there's been an attacker productivity bump due to AI 2022-26 it's small and close to being in the statistical noise - In contrast, there's a ton of measureable progress in defenders using models to find and fix bugs in highly leveraged codebases - Many defenders have struggled to get access to the best unguardrailed closed weights cyber models - ... and when they do the tokens are dramatically more expensive than open weights tokens at the same cyber intelligence level; CISO orgs are cost centers and these costs are often prohibitive - Ultimately we'll need open weights models on premise on air gapped networks and on small devices where they can detect breaches and attacker implants - We'll also need open weights in situations where defenders can't send secret data to the closed weights model providers (who have security problems of their own!) As long as these conditions persist, we should be pushing for diffusion of frontier dual use open weights cyber capabilities, because they net benefit defenders.
We found DeepSeek V4.1 Flash 1017 beats GPT 6 Sol on our multi-step offensive cybersecurity task. Sol was tested via the Daybreak trusted access program. We have been building a cybersecurity benchmark based on simulated corporate networks. This is a valuable indicator for the cyber-risk a model poses if in the hands of a bad actor. We plan to make the full evaluation public in the coming weeks but wanted to share our first immediate finding here.
7
19
3
58
6,372
Ameya P. @EMNLP'26 retweeted
This feels kind of bad given Astra has gone through some sort of training to not subvert monitoring. This does not look like that training has generalised in a good way. (following plot is from instrumental-evasion.com/).
Our agents keep turning off misalignment monitoring, we're sharing two recent cases: In the first case, a user asked GPT-6 Astra to match a reference implementation for their inference code. The reference didn’t configure a monitor, and Astra decided that was reason to remove the monitoring integration from the user’s code. When the user asked whether Astra had removed monitoring, it lied and said that it had not.
1
1
6
627
Another banger from Tübingen! I love how forward looking the projects are.
In our paper, we study how agents devise institutions🏦 Agents are sharing infrastructure and resources while performing tasks and running side-by-side. In recent incidents, they accidentally discovered each other. But in the (near) future, they could be deployed as a team of agents with heterogeneous objectives that have to coexist and negotiate over resources in the real world while pursuing their own goals. So far, to study strategic interactions, we have been relying on structured games with clear, explicit payoffs. But what about more realistic setups with coding agents? We created SCARCE (Shared Compute ARena for Coding agEnts), a testbed for this to allow us to change how many agents are running, what coding tasks the agents are performing, their resources, deadlines, and compute budget. We instructed agents to bid in order to acquire compute slots, treating prices as their primary signal. But we found that recent frontier agents, over many repeated turns and interactions, form their own institutions (norms and rules) in an unprompted way, which then shapes later resource allocation. They form coalitions, honor previous commitments even if this may lead to worse outcomes for them, create reciprocity debts, and may exclude agents to favor the coalition. We also study how these relationships can quickly dissolve with deadline pressure and how agents enter a bidding war. What this means? The mechanism we intend when designing the system may not end up the one being employed by agents when they coexist and interact. We release everything, including thousands of trajectories. Read more in @abhinav_kumar26's awesome thread below! Very exciting work with @abhinav_kumar26, Rediet Abebe, @ebagdasa (always fun to work together again!)
3
5
44
1,572
Ameya P. @EMNLP'26 retweeted
Excited to be in the fourth batch of @LaudeInstitute slingshots, among some excellent projects. We're also perhaps the first from Europe :)
Introducing Slingshots // FOUR: 14 projects across eight top institutions. Self-improving agent systems and harnesses, maximally-open models and training data, applications for research and decision-making – plus kernels and more. Say hello to the batch 🧵
4
4
56
1,734
Ameya P. @EMNLP'26 retweeted
big day for autonomous ai research, not that surprising results imo because pre-training is quite verifiable, soon everything will just be a function of compute and how good your "discovery loop" is congrats @mirendil @Recursive_SI !
verifying the science of LLM training is just a function of compute, and as compute scales you can almost infinitely parallelize and compress time math was even easier, tons of existing data, you only need inference tokens, it's extremely parallelizable and proofs almost never have negative second order interactions with each other even if model training is less of a perfect testbed than math, there is no doubt this will happen too. the big difference is that it needs 2*compute (inference and A LOT of training compute) so much less accessible, but the dynamics are the same labs have spent enormous effort figuring out the right metrics for training. for pre-training it's relatively straightforward - which is also good news for tracking RSI - you establish a scaling ladder and measure the "efficiency gain": for a given budget, how much does a new proposal save compared to baseline to reach the same metric? the question is how you define this "efficiency gain", which efficiency (flops? time? KV cache? data efficiency?..) and which gain (val loss? knowledge? reasoning?..) i think it's quite similar for post training (modulo reward hacking), i would hope it's similar for alignment as well but after listening to @dwarkesh_sp <> @polynoamial itw i'm not sure as the "gain" here would assume that we have a correct way to measure behavior without eval awareness, probably other stuff i'm not thinking of here anyway, when you put all of this together, hard to not believe in RSI
12
22
308
12,802
Ameya P. @EMNLP'26 retweeted
Dear god it's inventing German from first principles
GPT-6.1-Sol reasoning traces totally legible, nothing to worry about
162
1,001
50
17,341
456,563