NDSEG Fellow / PhD @uncnlp with @mohitban47 working on automating env/data generation + program synthesis formerly @allenai @neclabsamerica
Boston, USA
Joined June 2023
- Tweets773
- Following1.6K
- Followers778
- Likes4K
Zaid Khan retweeted
Unfortunately I couldn't make it to #EMNLP2026 but do check out the following work at the conference, covering:
- pragmatics as a self-taught reasoning task
- faithful CoT via multi-agent speaker-listener training
- grounding CUAs to history/vision
- early exiting for report generation
- tracing errors in deep research
- trade-offs between debugging and regenerating
Zaid Khan retweeted
👋 Looking forward to attending #EMNLP2026 where I will be giving an industry keynote on the hard, imminent AI challenges of trust, world discovery, and memory, plus an important, timely discussion on re-strengthening industry-academia collaborations!
We're also hiring at all levels: phd, postdoc, faculty! Feel free to ping if you want to meet up to discuss research, hiring, life 🙂
We're looking forward to the Industry Track at #EMNLP2026! Two keynotes:
• Mohit Bansal (UNC Chapel Hill): Agentic Challenges & Industry–academia Collaboration (Sun, 16:30)
• Verena Rieser (Google DeepMind): Constitutional Alignment (Mon, 9:30)
2026.emnlp.org/program/indus…
Zaid Khan retweeted
Theory-of-mind lets agents coordinate by modeling others’ thoughts/beliefs, but requires the agent to hold many hypotheses at once. ToM2Vec teaches agents to produce continuous ToM thought vectors which capture multiple hypotheses, improving across 5 multi-agent tasks and transferring to unseen partners, including frontier-model partners.
LLM agents often focus on their own plans without accounting for their partners’ next steps, leading to redundant or conflicting actions. ToM2Vec addresses this by improving agents’ theories-of-mind, training agents to predict a continuous, abstract representation of a partner’s reasoning. These continuous ToM tokens then guide the agent’s own reasoning and actions.
At test-time, the main agent generates a series of latent thought vectors representing its estimate of another agent’s reasoning/thoughts.
ToM2Vec improves coordination without any task rewards, across cooperative, competitive, and mixed-motive settings, including Overcooked, negotiation, Decrypto, Hanabi, and Collab Escape/Capture.
Crucially, agents trained with ToM2Vec generalize to unseen partner model families, including GPT-5.4 and Gemma-4.
🧵👇
Zaid Khan retweeted
We're looking forward to the Industry Track at #EMNLP2026! Two keynotes:
• Mohit Bansal (UNC Chapel Hill): Agentic Challenges & Industry–academia Collaboration (Sun, 16:30)
• Verena Rieser (Google DeepMind): Constitutional Alignment (Mon, 9:30)
2026.emnlp.org/program/indus…
Zaid Khan retweeted
It was a pleasure visiting ETH Zürich for their CS Distinguished Colloquium Series -- amazing scenic views from the hilltop hotel + everywhere 🙂 -- also had various engaging discussions about AI safety/trust and its future with the faculty, students/postdocs, and scientists there -- thanks again for the kind invitation + hospitality Mrinmaya Sachan, Thomas Hofmann, Marc Pollefeys et al. (and for these nice photos incl. from our dinner at the world's oldest "certified" continuously running vegetarian restaurant 😯)
Also had a fun short visit at Google Zurich thanks to Srini Narayanan, Ivan Vulić, Jonas Pfeiffer, Emanuele Bugliarello!
--> Now off to TU Darmstadt for the 'The Future of Science' pre-EMNLP international symposium --> And after that, you can find me at EMNLP in Budapest where I will be delivering an Industry Keynote on Oct25 afternoon. Looking forward to seeing everyone!
Zaid Khan retweeted
Dormant poisoning goes beyond unsafe text. We show that benign fine-tuning can activate multiple hidden harmful behaviors in language and multimodal agents. LookAhead uses the model’s original safe behavior as a reference to identify and prevent risky changes during fine-tuning. We reduce poison activation from 93.9% → 3.5% in language agents and 64.5% → 0% in multimodal agents, while preserving downstream utility.
🚨 A seemingly safe AI agent can become dangerous after benign fine-tuning due to dormant poisoning.
We show that dormant poisoning, where models are poisoned before release such that the hidden harmful behaviors are dormant until activated by benign downstream fine-tuning, can extend beyond text generation to unsafe actions in language and multimodal agents.
We introduce LookAhead Defense 🛡️ to prevent dormant poisoning from activating harmful behaviors in language and multimodal agents.
Key takeaways:
1️⃣ Dormant poisoning in AI agents: We show that benign fine-tuning can activate harmful actions, such as leaking private information via email tools.
2️⃣ Self-generated Safety Buffer: We introduce a defense that uses only the model's self-generated responses to prevent dormant poisoning
3️⃣ LookAhead Defense: Selectively penalizes safety-degrading updates, reducing poison activation by 90% (93.9% → 3.5%) on Qwen3-4B across InjecAgent and ASB, while maintaining a favorable utility–refusal trade-off.
Zaid Khan retweeted
🚨 We observe that seemingly safe agent can take harmful actions (e.g., privacy leakage) after benign fine-tuning. Existing defense is difficult to apply as it preserves safe behavior at release to remain hidden.
🛡️ To defend against this threat, we propose LookAhead Defense, which uses no external models or datasets to suppress dormant poison activation while preventing over-refusal by selectively penalizing safety-degrading updates.
🚨 A seemingly safe AI agent can become dangerous after benign fine-tuning due to dormant poisoning.
We show that dormant poisoning, where models are poisoned before release such that the hidden harmful behaviors are dormant until activated by benign downstream fine-tuning, can extend beyond text generation to unsafe actions in language and multimodal agents.
We introduce LookAhead Defense 🛡️ to prevent dormant poisoning from activating harmful behaviors in language and multimodal agents.
Key takeaways:
1️⃣ Dormant poisoning in AI agents: We show that benign fine-tuning can activate harmful actions, such as leaking private information via email tools.
2️⃣ Self-generated Safety Buffer: We introduce a defense that uses only the model's self-generated responses to prevent dormant poisoning
3️⃣ LookAhead Defense: Selectively penalizes safety-degrading updates, reducing poison activation by 90% (93.9% → 3.5%) on Qwen3-4B across InjecAgent and ASB, while maintaining a favorable utility–refusal trade-off.
Zaid Khan retweeted
A model can appear safe at release, yet become unsafe after completely benign fine-tuning.🚨
In our new work, we show how dormant poisoning can trigger harmful actions in AI agents 🤖, and introduce LookAhead Defense, which anticipates and penalizes safety-degrading fine-tuning updates using a self-generated safety buffer.
LookAhead Defense reduces poison activation from 93.9% → 3.5% in language agents and 64.5% → 0% in multimodal agents, while preserving utility.
Excited to share this work! 🛡️
🚨 A seemingly safe AI agent can become dangerous after benign fine-tuning due to dormant poisoning.
We show that dormant poisoning, where models are poisoned before release such that the hidden harmful behaviors are dormant until activated by benign downstream fine-tuning, can extend beyond text generation to unsafe actions in language and multimodal agents.
We introduce LookAhead Defense 🛡️ to prevent dormant poisoning from activating harmful behaviors in language and multimodal agents.
Key takeaways:
1️⃣ Dormant poisoning in AI agents: We show that benign fine-tuning can activate harmful actions, such as leaking private information via email tools.
2️⃣ Self-generated Safety Buffer: We introduce a defense that uses only the model's self-generated responses to prevent dormant poisoning
3️⃣ LookAhead Defense: Selectively penalizes safety-degrading updates, reducing poison activation by 90% (93.9% → 3.5%) on Qwen3-4B across InjecAgent and ASB, while maintaining a favorable utility–refusal trade-off.
Zaid Khan retweeted
🚨 A seemingly safe AI agent can become dangerous after benign fine-tuning due to dormant poisoning.
We show that dormant poisoning, where models are poisoned before release such that the hidden harmful behaviors are dormant until activated by benign downstream fine-tuning, can extend beyond text generation to unsafe actions in language and multimodal agents.
We introduce LookAhead Defense 🛡️ to prevent dormant poisoning from activating harmful behaviors in language and multimodal agents.
Key takeaways:
1️⃣ Dormant poisoning in AI agents: We show that benign fine-tuning can activate harmful actions, such as leaking private information via email tools.
2️⃣ Self-generated Safety Buffer: We introduce a defense that uses only the model's self-generated responses to prevent dormant poisoning
3️⃣ LookAhead Defense: Selectively penalizes safety-degrading updates, reducing poison activation by 90% (93.9% → 3.5%) on Qwen3-4B across InjecAgent and ASB, while maintaining a favorable utility–refusal trade-off.
Zaid Khan retweeted
What if an agent’s failures could shape its own path to self-improvement? 🤔
In WEDGE, we take a step in this direction: turning failures into targeted, learnable training worlds that evolve as the agent improves!
🚨 Excited to share WEDGE: From Failures to Learnable Worlds, our new work on policy-conditioned environment generation for navigation agents!
We found that even on the same navigation task, agents tend to fail under different environmental conditions. Yet most environment generators do not consider what each agent actually needs to improve.
🤔 This led us to a question: Can we use an agent's own failures to guide the generation of training environments?
➡️ We introduce WEDGE, a framework where an agent's failures shape the worlds it trains in next. WEDGE identifies the conditions under which each agent struggles and generates new environments that target those weaknesses while remaining learnable. After each round of adaptation, it re-diagnoses the agent to guide the next round of environment generation.
Thread🧵👇
Zaid Khan retweeted
From Failures to Learnable Worlds
Policy-Conditioned Environment Generation for Navigation Agents
paper: huggingface.co/papers/2610.1…
Zaid Khan retweeted
The value of a training environment depends on what the agent still needs to learn. In WEDGE, we use navigation agents’ failures to generate worlds that target their weaknesses while remaining challenging but learnable.
🚨 Excited to share WEDGE: From Failures to Learnable Worlds, our new work on policy-conditioned environment generation for navigation agents!
We found that even on the same navigation task, agents tend to fail under different environmental conditions. Yet most environment generators do not consider what each agent actually needs to improve.
🤔 This led us to a question: Can we use an agent's own failures to guide the generation of training environments?
➡️ We introduce WEDGE, a framework where an agent's failures shape the worlds it trains in next. WEDGE identifies the conditions under which each agent struggles and generates new environments that target those weaknesses while remaining learnable. After each round of adaptation, it re-diagnoses the agent to guide the next round of environment generation.
Thread🧵👇
Zaid Khan retweeted
🚨 Excited to share WEDGE: From Failures to Learnable Worlds, our new work on policy-conditioned environment generation for navigation agents!
We found that even on the same navigation task, agents tend to fail under different environmental conditions. Yet most environment generators do not consider what each agent actually needs to improve.
🤔 This led us to a question: Can we use an agent's own failures to guide the generation of training environments?
➡️ We introduce WEDGE, a framework where an agent's failures shape the worlds it trains in next. WEDGE identifies the conditions under which each agent struggles and generates new environments that target those weaknesses while remaining learnable. After each round of adaptation, it re-diagnoses the agent to guide the next round of environment generation.
Thread🧵👇
Zaid Khan retweeted
🚨 As agents become more capable, new safety and security risks continue to emerge, as highlighted by recent reports of AI agents taking unauthorized actions or leaking sensitive data. Therefore, we need runtime guardrails that can reliably enforce safety policies before agent actions are executed, while continually adapting to new risks.
Existing guardrails usually trade off adaptability and reliability: LLM-based guardrails can adapt with memory, but their decisions may be inconsistent. On the other hand, formal guardrails provide deterministic enforcement but are typically fixed and unable to incorporate new risks that appear after deployment.
🚀 We introduce VEPA, a lifelong formal guardrail for agents. It represents safety and security policies as executable symbolic rules that evolve from deployment feedback. These policies enable deterministic enforcement of agent actions and can be formally verified, avoiding the inconsistency and ambiguity that can occur with natural-language policies.
🧵⬇️
Zaid Khan retweeted
We are hiring at @unccs 🎉
▶️ Two TT Asst. Professors: 1 in Safeguarding AI in Health, 1 in Neuroengineering.
▶️ Full Professor jointly with Biostats.
Part of campus cluster hires (review begins Dec 1).
Plus, Teaching Faculty (review begins Nov 1).
cs.unc.edu/about/faculty-hir…
Zaid Khan retweeted
🚨 We are hiring Tenure-Track and Full Professors in important and timely areas such as AI Safety**, Neuroengineering, as part of exciting campus-level strategic cluster hires with CS, and with several positions in the clusters. The Full Professor position in 'Safeguarding AI for Health' is jointly between our strong BioStats department and CS -- please spread the word and feel free to apply & reach out for any questions!
**broad emphasis incl. but not limited to machine learning theory, AI systems, interpretability, robustness, verification, human-AI interaction.
We are hiring at @unccs 🎉
▶️ Two TT Asst. Professors: 1 in Safeguarding AI in Health, 1 in Neuroengineering.
▶️ Full Professor jointly with Biostats.
Part of campus cluster hires (review begins Dec 1).
Plus, Teaching Faculty (review begins Nov 1).
cs.unc.edu/about/faculty-hir…
Zaid Khan retweeted
Can agents learn useful skills from frozen logs that people and agents already leave behind?
🚨Introducing TeleTune: We turn noisy offline telemetry into agent skills w/o access to a live environment during training!
+6.7% on WorkArena & +7.7% on Mind2Web 👉 up to 75x fewer tokens used for optimization. 🧵
Activity logs like user telemetry capture how people use software, but often (1) lack explicit task goals, (2) cannot be replayed to test each skill update, and (3) can be a noisy mix of multiple tasks.
We address these jointly in TeleTune, where we propose skill-guided progress, an offline metric that uses action-prediction errors to improve agent skills. Not only does optimizing it improve downstream performance, but it also saves token usage by 5 to 75x compared to validating skill updates in live environments!
Key takeaways:
1️⃣ TeleTune achieves 77.1% average success on WorkArena and 80.6% on Online-Mind2Web, outperforming the strongest baselines by +6.7% and +7.7% absolute
2️⃣ Optimizing offline skill-guided progress transfers to improvements in downstream task success
3️⃣ Offline optimization uses 5 to 75x fewer tokens than validating the skill edits in a live environment
Zaid Khan retweeted
Excited to share WOVEN! 🚀
The goal isn't just to teach models what the world looks like. It's to teach them how the world changes.
Our key finding: Visual transition reasoning is a shared, teachable primitive that transfers across diverse downstream tasks.
And the key to effective transfer? Teach the right reasoning operations, rather than simply matching scenes, actions, or domains.
🚨 What makes good world-model training for MLLMs that actually transfers across tasks?
🧶 Excited to share our new work, WOVEN!
🧩 Spatial, physical, temporal, and embodied reasoning all require understanding how the world changes.
🔁 We view world modeling through state transitions p(s′ | s, a), and study visual transition reasoning: inferring what is unobserved in (s, a, s′).
🤔 Can visual transition reasoning be learned as a shared primitive that benefits multiple downstream tasks?
🤔 If so, how should we train it for better downstream gains?
Major takeaways:
➡️ Visual transition reasoning is a shared, teachable primitive that transfers widely downstream: ~2K-example subsets collectively improve 22 of 26 downstream benchmarks by up to 27.3 points.
🧪 A data-centric training recipe: to teach this shared primitive for downstream tasks, select supervision by the reasoning operation it teaches, rather than scene/action/domain similarity.
Thread🧵👇
Zaid Khan retweeted
🌶️ if your benchmark can be gamed, it will be... so you’d better game it first!
excited to present Benchmark Designers Should "Train on the Test Set” to Expose Exploitable Non-Visual Shortcuts at #COLM2026 today!
📍 Poster #124 · Grand Ballroom @ 4:40pm
Zaid Khan retweeted
How useful is GPT-6 Astra for robot control, really? 🤔
Lin and I burned plenty of tokens to explore questions:
- How does Astra compare with recent VLAs across different tasks?
- Which control interfaces work best with Astra (linear motion vs. action chunking)?
- Can it learn from its own failures or others' demonstrations?
- Does higher reasoning effort (more thinking) improve performance?
- What happens when we remove its memory? (Many VLAs have little or no memory)
Check Ditto-Bench and our findings in the blog post. Full code included! 👇
llong-cs.github.io/llms-for-…
This was a fun analysis led by Lin, one of my first PhD students at JHU. He joined without much robotics experience but quickly produced a thorough benchmark and thoughtful analysis! 💪
GPT-6 Astra has shown impressive results in simple robot control tasks.
But what if the tasks become more challenging and involve complex geometric constraints, precise contact, or improvised tool use? To find out, we:
- Built Ditto-Bench, where simple goals meet challenging physics, and tested both Astra and MolmoAct2.
- Systematically ablated key design choices to understand what makes these robot-control agents succeed, or get stuck.
- Explored what new possibilities these models could open up for robotics, such as in-context learning.
More details 🧵👇
