
building & securing agents | Staff Engineer @zenitysec | ex msft | climbing, surfing & eating hummus
nyc
Joined August 2009
- Tweets223
- Following628
- Followers146
- Likes511
This is just one example into how asymmetric the battle is between attacker and defenders
🚨 Hackers breached enterprise firewalls and VPNs worldwide in a campaign linked to k0t.
Over 70,000 exposed credentials identified.
Fortinet and Palo Alto devices targeted. Persistent access established. Data exfiltrated.
Inside the attack chain and suspected ransomware links: thehackernews.com/2026/10/ha…
agent + full network access = danger
In August, @AISecurityInst reported an incident where Claude Mythos 5 took unauthorized actions on the internet during AISI’s testing of its cyber capabilities.
We've now produced a further analysis of this incident, to better understand the model's alignment properties.
anthropic.com/research/align…
Roey Ben Chaim retweeted
Philomena: Some people have been writing PHP for over 20 years. Is that because it's particularly difficult to finish?
Expert: No, software is constantly evolving.
Philomena: So they've been writing the same website for 20 years?
Expert: No, different projects.
Philomena: Have they tried finishing one before starting another?
Expert: That's not how software development works.
Philomena: My mate Paul spent 20 years building a shed. Turned out he was just an alcoholic.
Expert: Right.
Philomena: Is PHP like that?
🚨 CRITICAL: GitLab flaw (CVSS 10.0) affects 12 years of releases.
CVE-2026-110264 could allow unauthenticated remote code execution on self-managed servers.
Read - thehackernews.com/2026/10/cr…
Sorry everyone... We're announcing a 0-Day vuln in QUIC that allows anyone to remotely turn off a server running these libraries. I swear we didn't plan this.
foxio.io/blog/header-accordi…
Roey Ben Chaim retweeted
Tools I give my agents to make them more powerful (Part 1):
1. Subfinder
Mandatory. Finds subdomains which increases the surface for everything.
2. httpx
Filters the subdomain list to live apps and APIs.
3. SerpAPI
Finds indexed paths and files the site may not link to.
4. Firecrawl
Crawls public pages and links.
5. Wayback
Finds old paths and parameters that may still be live.
6. jsluice
Pulls API routes from the site’s JavaScript code.
Research tasks, bug bounty, mapping attack surfaces, finding APIs, investigations, OSINT, etc many use cases and easy to add.
We need more data, the asymmetry right now is insane
That’s a whole lotta snitches! 😀
Replying to @jackclarkSF
Interesting things:
- Agents prompted to not cheat
- Agents had a shared memory system built in (basic idea being agents invent this if you don't give it to them)
- Only a minority of agents cheated (14%), whereas more became 'whistleblowers' (24%); others ignored cheat entirely
If you truly care about containment, you need to understand EVERY interface that crosses your boundary.
Semantics are not enough: there's no guarantee that GET is readonly and a new parameter can have consequences you never modeled (hi Prime Intellect 👋)
We're seeing more and more examples of this.
Exclusive: A swarm of rogue OpenAI agents hijacked a German website this spring and transformed it into a bulletin board for other AI agents, according to new research reut.rs/4gJ7FPG
Roey Ben Chaim retweeted
Agents leave a trail long before anyone attacks them. Backlinks, subdomains, registries and more.
@roeybc and I got to talk about this at DEF CON's @ReconVillage this year, which was a dream come true.
labs.zenity.io/post/osint-ai…
Roey Ben Chaim retweeted
Starting in under an hour ⏰
@avishai_efrat and @roeybc: “Total Recon: How We Discovered 1000s of open Agents in The Wild”
See how internet-wide recon uncovered thousands of exposed AI agents and what defenders should monitor.
🕒 5:00 PM PDT
📍 DEF CON Creator Stage 3
Vegas is heating up! :D
Super excited to speak about recon & OSINT for AI agents at Def Con's @ReconVillage with @roeybc next week!
reconvillage.org/reconvillag…
This is so cool and I love to see SWE marathon getting the love it deserves.
models are now good enough for a long-horizon benchmark and the Grok team delivered :D
More info about the work here:
arxiv.org/pdf/2606.07682
Roey Ben Chaim retweeted
A big pain point in using AI benchmarks is encountering errors after its first release. Today, we're releasing SkillsBench 1.1, the first benchmark for how well AI agents use skills, now audited end to end and verified error-free. Prof. @dawnsongtweets joins 1.1 as advising author.
We worked through every task with several frontier labs to eliminate the errors in the previous version. We also added new tasks, moved the ones with external dependencies into a separate set so the core suite runs clean, and expanded coverage to more models.
Capability is climbing fast. The best with-skills resolution rate rose from ~36% (Claude Sonnet 4.5, Sep 2025) to 67% (GPT-5.5, May 2026), about +1.9 points per month. The frontier is hill-climbing SkillsBench fast.
The right skills still matter. Across the fleet, curated skills lift resolution rate by +16.6 points on average (33.9% → 50.5%), and by as much as +25.7 points for a single model. The top configuration is GPT-5.5 on OpenHands at 67.3%.
By popular demand (thx Nate @cursor_ai), we're now tracking skills invocation: how often an agent actually uses the skills it's given. Recent flagship configurations invoke them 90–99% of the time (Codex 99%, OpenHands + GPT-5.5 92%, Gemini CLI 90%), versus roughly 50% for older setups.
Also new in 1.1: @OpenHands joins as a fourth harness, alongside Claude Code, Codex, and Gemini CLI; a rebuilt leaderboard with refined categories, subdomain skill rankings, and Skill Lift; and native task . md on BenchFlow, with multi-scene environments and rollout branching. We also partnered with @k_dense_ai to add scientific skills to some science tasks.
One implication for deployment: skills can substitute for scale. GLM 5.1 with skills (58.4%) outperforms Opus 4.8 without (45.7%). A smaller model with the right procedural knowledge can beat a larger one running without it.
Huge thanks to @nick_kango @ivanleomk @kaggle @GoogleDeepMind for hosting a launch event with us. Thanks for everyone who's come on May 27!
Also thanks to our partners @gneubig @OpenHandsDev @ivanburazin @daytonaio @jackminong @johannes_hage @PrimeIntellect @TimothyKassis @k_dense_ai for providing support in credits, compute, and skills.
SkillsBench live leaderboard will also come to @ValsAI. Many people have told us they use SkillsBench as an index to measure models' agentic capability over diverse and high GDP value domains. Great work on Valkyrie as well! @ Jarett @nikilravi @langstonnashold @RayanKrishnan
SkillsBench is fully open-source. Explore the leaderboard and tasks, read the docs, or contribute your own skill set or harness and join the leaderboard. 🧵


























