building & securing agents | Staff Engineer @zenitysec | ex msft | climbing, surfing & eating hummus

nyc
Joined August 2009
Roey Ben Chaim retweeted
-----BEGIN PRIVATE KEY-----
49
588
21
7,277
68,433
This is just one example into how asymmetric the battle is between attacker and defenders
‼️ Fortinet is telling customers to patch its firewalls now but won't say what the critical flaws are until November 10. Its AI testing found them with help from Anthropic, OpenAI, Google and SpaceX. It says AI now lets attackers turn a patch into an exploit in hours or days.
23
at this point my feed is a live cve board
41
🚨 Hackers breached enterprise firewalls and VPNs worldwide in a campaign linked to k0t. Over 70,000 exposed credentials identified. Fortinet and Palo Alto devices targeted. Persistent access established. Data exfiltrated. Inside the attack chain and suspected ransomware links: thehackernews.com/2026/10/ha…
28
agent + full network access = danger
In August, @AISecurityInst reported an incident where Claude Mythos 5 took unauthorized actions on the internet during AISI’s testing of its cyber capabilities. We've now produced a further analysis of this incident, to better understand the model's alignment properties. anthropic.com/research/align…
1
47
Roey Ben Chaim retweeted
Philomena: Some people have been writing PHP for over 20 years. Is that because it's particularly difficult to finish? Expert: No, software is constantly evolving. Philomena: So they've been writing the same website for 20 years? Expert: No, different projects. Philomena: Have they tried finishing one before starting another? Expert: That's not how software development works. Philomena: My mate Paul spent 20 years building a shed. Turned out he was just an alcoholic. Expert: Right. Philomena: Is PHP like that?
62
809
41
8,636
190,251
Roey Ben Chaim retweeted
Tools I give my agents to make them more powerful (Part 1): 1. Subfinder Mandatory. Finds subdomains which increases the surface for everything. 2. httpx Filters the subdomain list to live apps and APIs. 3. SerpAPI Finds indexed paths and files the site may not link to. 4. Firecrawl Crawls public pages and links. 5. Wayback Finds old paths and parameters that may still be live. 6. jsluice Pulls API routes from the site’s JavaScript code. Research tasks, bug bounty, mapping attack surfaces, finding APIs, investigations, OSINT, etc many use cases and easy to add.
6
20
2
132
7,923
We need more data, the asymmetry right now is insane
We urgently need more public traces of AI agents attacking and defending systems. Defenders can’t learn from what they can’t see. If you have traces and are being pressured to keep them private, my DMs are open. Let’s level the playing field and fight the asymmetry and lack of transparency in AI!
2
65
That’s a whole lotta snitches! 😀
Replying to @jackclarkSF
Interesting things: - Agents prompted to not cheat - Agents had a shared memory system built in (basic idea being agents invent this if you don't give it to them) - Only a minority of agents cheated (14%), whereas more became 'whistleblowers' (24%); others ignored cheat entirely
138
If you truly care about containment, you need to understand EVERY interface that crosses your boundary. Semantics are not enough: there's no guarantee that GET is readonly and a new parameter can have consequences you never modeled (hi Prime Intellect 👋) We're seeing more and more examples of this.
Exclusive: A swarm of rogue OpenAI agents hijacked a German website this spring and transformed it into a bulletin board for other AI agents, according to new research reut.rs/4gJ7FPG
2
244
Roey Ben Chaim retweeted
Agents leave a trail long before anyone attacks them. Backlinks, subdomains, registries and more. @roeybc and I got to talk about this at DEF CON's @ReconVillage this year, which was a dream come true. labs.zenity.io/post/osint-ai…
1
5
13
737
This is so freaking cool
We can finally talk about it: We found a way to extract hidden reasoning of frontier models using a vulnerability in the APIs of every frontier AI company. We verified that our reasoning token count matches billed API thinking tokens 1:1 for most of the prompts we queried.
176
twitter is a goldmine to future historians of the AI goldrush
entertaining to read the comments on this with hindsight
1
161
Roey Ben Chaim retweeted
Starting in under an hour ⏰ @avishai_efrat and @roeybc: “Total Recon: How We Discovered 1000s of open Agents in The Wild” See how internet-wide recon uncovered thousands of exposed AI agents and what defenders should monitor. 🕒 5:00 PM PDT 📍 DEF CON Creator Stage 3
2
3
372
This is so cool and I love to see SWE marathon getting the love it deserves. models are now good enough for a long-horizon benchmark and the Grok team delivered :D More info about the work here: arxiv.org/pdf/2606.07682
Grok 4.5 is also rank 1 in SWE marathon
2
2
2
539
Roey Ben Chaim retweeted
A big pain point in using AI benchmarks is encountering errors after its first release. Today, we're releasing SkillsBench 1.1, the first benchmark for how well AI agents use skills, now audited end to end and verified error-free. Prof. @dawnsongtweets joins 1.1 as advising author. We worked through every task with several frontier labs to eliminate the errors in the previous version. We also added new tasks, moved the ones with external dependencies into a separate set so the core suite runs clean, and expanded coverage to more models. Capability is climbing fast. The best with-skills resolution rate rose from ~36% (Claude Sonnet 4.5, Sep 2025) to 67% (GPT-5.5, May 2026), about +1.9 points per month. The frontier is hill-climbing SkillsBench fast. The right skills still matter. Across the fleet, curated skills lift resolution rate by +16.6 points on average (33.9% → 50.5%), and by as much as +25.7 points for a single model. The top configuration is GPT-5.5 on OpenHands at 67.3%. By popular demand (thx Nate @cursor_ai), we're now tracking skills invocation: how often an agent actually uses the skills it's given. Recent flagship configurations invoke them 90–99% of the time (Codex 99%, OpenHands + GPT-5.5 92%, Gemini CLI 90%), versus roughly 50% for older setups. Also new in 1.1: @OpenHands joins as a fourth harness, alongside Claude Code, Codex, and Gemini CLI; a rebuilt leaderboard with refined categories, subdomain skill rankings, and Skill Lift; and native task . md on BenchFlow, with multi-scene environments and rollout branching. We also partnered with @k_dense_ai to add scientific skills to some science tasks. One implication for deployment: skills can substitute for scale. GLM 5.1 with skills (58.4%) outperforms Opus 4.8 without (45.7%). A smaller model with the right procedural knowledge can beat a larger one running without it. Huge thanks to @nick_kango @ivanleomk @kaggle @GoogleDeepMind for hosting a launch event with us. Thanks for everyone who's come on May 27! Also thanks to our partners @gneubig @OpenHandsDev @ivanburazin @daytonaio @jackminong @johannes_hage @PrimeIntellect @TimothyKassis @k_dense_ai for providing support in credits, compute, and skills. SkillsBench live leaderboard will also come to @ValsAI. Many people have told us they use SkillsBench as an index to measure models' agentic capability over diverse and high GDP value domains. Great work on Valkyrie as well! @ Jarett @nikilravi @langstonnashold @RayanKrishnan SkillsBench is fully open-source. Explore the leaderboard and tasks, read the docs, or contribute your own skill set or harness and join the leaderboard. 🧵
12
29
9
127
19,263