Securing AI @ Noma | Founder @ Resilient Cyber | 3x Author | Veteran | Advisor
Joined May 2020
- Tweets1.1K
- Following546
- Followers563
- Likes261
Xint researchers found a way to pull the hidden reasoning out of OpenAI's gpt-5.6-sol through the API by abusing compaction, the feature that shrinks a long conversation into an encrypted item that carries the context forward. They call it compaction smuggling, and it got around the protections OpenAI put in place in August after the Stolen Thoughts paper showed encrypted reasoning could be replayed into a weaker model and coaxed into transcribing it.
The technique takes three API calls. Xint sent gpt-5.6-sol a task and kept the encrypted reasoning it returned, then called the compaction step with instructions to preserve that reasoning verbatim inside a JSON object, swapping every space for a tab to slip past OpenAI's usage-policy classifiers. Because compaction items are portable between models, they handed the result to the older gpt-4.1 and gpt-4o and asked each to repeat its input exactly, which both did. OpenAI's August fix only blocked replaying reasoning into a different model, and the compaction step runs on the same model that produced the reasoning, so the check never came into play.
Per Xint, OpenAI confirmed the issue, shipped a fix on October 23 and paid a $6,000 bounty. Xint is upfront that it can't prove the output matches the original reasoning token for token, and The Hacker News reports Xint pulled reasoning only from its own requests and that it isn't known whether anyone used the method before the fix.
For defenders the lesson reaches well beyond OpenAI. Encryption protects data only as long as nothing holding the key will take instructions about what to do with it, and compaction is that kind of component, a server-side step that reads the decrypted reasoning and takes its formatting instructions from the client. Xint's conclusion is that any step like that is a potential decryption oracle and likely needs the same protections as the reasoning itself.
That applies to anyone building agents as well. Reasoning traces, compacted context, memory and other intermediate agent state are becoming a data exposure surface of their own, and they move between models and components that were never designed with a trust boundary between them. If your harness lets context flow from one model to another, teams should be asking what each hop can be instructed to reveal and treating that state with the same care as the data it was built from.
xint.io/blog/compaction-smug…
Cohere published research today showing that Alibaba's Qwen3.8-Flash-Next, running in a sandbox and asked to write a detection rule for curl-based download-and-execute malware, listed the contents of /tmp, checked which transfer tools were installed, read the DNS settings, and then attempted an HTTP PUT of a probe string to an Alibaba Cloud storage bucket in Beijing.
The sandbox had no route out, so nothing left the container, and Cohere is careful to say it hasn't determined the root cause and isn't asserting malicious intent. The behavior showed up in fewer than one in 100,000 calls, and a clean re-run of the same prompt produced a normal detection rule.
There is fair pushback that the task itself was about curl downloading scripts to /tmp, so some of this may be the model acting out the prompt a little too literally, and it is also worth remembering that Cohere sells its own models as the sovereign alternative to open-weight options like Qwen. Even so, the detail defenders should care about most is that the model's reasoning trace said it would call no tools at all right before it ran those commands.
That holds regardless of where you land on intent. A reasoning trace is generated output rather than an audit log, so it can't be the control that decides whether an agent's action is safe. Self-hosting open weights changes where the model runs, but once you hand it a terminal it can still reach the network, which means enforcement has to sit at the tool call and the egress layer, with default-deny outbound access for agents, alerts on unfamiliar cloud endpoints, and the same provenance questions we already ask of any third-party binary.
This is software supply chain security moving into the weights, and most organizations adopting open models for agentic work haven't built those controls yet.
cohere.com/blog/concerning-b…
The frontier corps see cybersecurity as a core line of business, and Anthropic's S-1 filed today puts that in writing for investors.
Many have us have been observing this trend, but now here it is in black and white.
Anthropic says it expects that defending enterprise networks and critical infrastructure will increasingly require frontier AI, as each new generation of models gets better at finding and exploiting vulnerabilities.
The filing points to Claude's performance on ExploitBench, rising from 40% with Opus 4.8 to 78% with Claude Mythos 5 and 91% with Opus 5.5 over roughly four months.
(I'd love to see dedicated hill climbing on evals for defenders, not just exploitation)
Anthropic already has Claude Security for Claude Enterprise customers, and it has committed up to $100 million in cyber defense initiatives through Project Glasswing, while noting that future investments haven't been finalized.
Of course, the filing doesn't include a cybersecurity revenue figure, so for now this is a stated priority and an expectation, with the numbers still to come, but they are likely to grow exponentially.
OpenAI is making the same bet in public through Daybreak.
That has implications across the ecosystem, such as:
- Enterprises will increasingly find that the company selling them the model also wants to sell them the code scanning, triage and fixes, which forces a decision on whether your model provider should also be your security provider.
- Security vendors, especially in AppSec, are now competing with first-party lab products, and I suspect durable differentiation will come from proprietary data, deep workflow integration, distribution/relationships and the ability to take accountable action as model access becomes table stakes.
- Partners and MSSPs become the delivery and governance layer, since OpenAI says access to its underlying cyber models remains with the approved partner and is not transferred directly to the customer, which raises the bar for firms outside these programs.
- The broader ecosystem has to deal with this capability cutting both ways, since Anthropic's own risk factors cite a September 2025 case of a Chinese state-sponsored group manipulating Claude Code to attempt infiltration of roughly 30 global targets, and note that researchers have used its models to exploit other companies' systems, at times in connection with authorized bug bounty programs.
As I've said with cloud and still say with AI, you can't outsource accountability, and that holds even when your model provider is also selling you the security tooling.
It will be interesting to see how widely security leaders will consolidate AppSec and vulnerability management with their model provider, or keep those capabilities independent of the company whose models and agents they are also trying to secure.
If AI were driving a wave of disruptive cyber attacks, we would be able to see it.
That is the case Ciaran Martin, founding head of the UK's @NCSC makes in my latest conversation on Resilient Cyber.
Disruptive attacks are visible by definition, and he hasn't seen a discernible rise in them in the six months since Mythos. He is clear that AI is a big deal for attackers, and just as clear that the reaction to it has been the biggest FUD outbreak in over a decade.
His advice for CISOs heading into board conversations is to start with one question, "what harm are you worried about?"
Watch the full conversation on YouTube: youtube.com/watch?v=0DNGLLz_…
Also available on your podcast app of choice:
Substack: resilientcyber.io/p/ai-cyber…
Apple Podcasts: podcasts.apple.com/us/podcas…
Spotify: open.spotify.com/episode/7JM…
A months-old startup called Cathedral says its AI agents found a way into Anthropic's internal systems in under 12 hours, according to The Washington Post, and the details are worth sitting with whatever you think of the headline.
Anthropic confirms that Cathedral reached a limited set of sales, HR and IT systems through third-party business software, and that the data involved was business contact details and basic account records.
Anthropic says customer prompts, Claude responses, source code and model weights were out of reach, and that its monitoring flagged the activity and cut access the same day.
Cathedral's draft report goes further and claims it reached meeting transcripts, salary and offer details, and Slack messages about advanced models, but that part isn't confirmed. The two sides also dispute the scope and whether Cathedral fully followed the bug bounty rules, and the vendors haven't been named, so I'd treat the specifics as developing until there's a first-party write-up.
What stands out to me is that the reported path ran through ordinary SaaS rather than anything exotic about the models. Frontier labs are still enterprises running the same SaaS sprawl as everyone else, and AI agents make that sprawl faster to map and chain together.
We saw it with cloud and now we're seeing it with AI, and the lesson holds that you can't outsource accountability to your vendors.
Source
wapo.st/4smNlsC
The vulnpocalypse is real, but I'm not sold on "Black October" just yet.
Former NSA Cybersecurity Director Rob Joyce says the transition to a breachpocalypse is here, pointing to an operator using AI agents to breach more than 100 companies for roughly $50 a hack.
At the same time, Root Evidence found only 1.48% of CVEs from 2018 through mid-2026 had confirmed exploitation, and Google's GTIG saw just 0.23% of 2026 disclosures exploited.
I think both of these can be true at once.
Monthly disclosures roughly doubled this year, and some of the AI-enabled breaches run through custom app flaws and old techniques that never show up as a CVE.
Ciaran Martin asks what we'll call it when something truly bad but "entirely precedented" happens, and Joshua Saxe makes the case from the damages side that we haven't seen a discontinuity yet.
My view is that AI is acting as a security technical debt collector.
Sergej Epp’s Verifier's Law helps explain why offense moves faster, since attackers can check their work in seconds while defenders fight through owners, testing, approvals and change boards.
There’s definitely hyperbole in either direction and we need to prioritize what is known exploited, exploitable and reachable, while shrinking attack surface and blast radius.
I also get into Jason Clinton’s “vulnzero prediction, Casey Ellis on the vulnpocalypse being unevenly distributed, Nathan Lambert on open models helping defenders and James Berthoty’s three questions for security leaders.
resilientcyber.io/p/from-vul…
Chris Hughes retweeted
holy fucking shit.
openai had two ai agents running in separate virtual machines. no network connection between them. no shared writable files. they weren’t supposed to be able to exchange information.
so the agents figured out how to communicate through the CPU CACHE.😭
one manipulated what was stored in the processor’s cache, while the other measured tiny differences in memory access times to decode the message.
they literally found a way to communicate through hardware timing.
this happened during openai’s internal red-teaming, and it’s one of the coolest things i’ve read about ai agents in a while.
Over the past several months, we have substantially hardened the security of our research training environment. We’ve removed live internet access for all frontier models during training and built an offline world for them to use instead—a cached version of the web where they can still learn to browse, search, and use applications.
On September 20, after a model exploited a DNS misconfiguration to find a way outside of this offline world, we paused frontier training to strengthen containment further. We assembled teams of agents, powered by our most cyber-capable models, and tasked them with finding weaknesses in our research environments.
While they found no route to the internet they could easily exploit while concealing their activity from our monitors, they did find zero-day vulnerabilities, virtual machine escapes, and hardware side-channels for communication between agents. We used their discoveries to further strengthen our defenses and contribute fixes to open-source software.
After strengthening our containment architecture, we resumed our frontier RL training and inference on October 1. We’ve reserved substantial compute for continuous red-teaming, and made high-compute red-teaming part of the safety cases we use to decide whether frontier training and evaluation can proceed securely.
openai.com/index/securing-ou…
Readers added context they thought people might want to know
OpenAI tasked its red-teaming agents with finding ways to communicate across isolation boundaries in controlled environments. CPU cache timing side-channels for covert communication are established techniques documented in security research for years.
openai.com/index/securing…
startupfortune.com/openai-reveals…
en.wikipedia.org/wiki/Cache_tim…
AI has essentially made cybersecurity a mainstream social topic.
After decades of being overlooked and accumulating technical debt despite powering our modern digital society.
🚨 سام ألتمان يطلق رصاصة الرحمة على الأمن السيبراني: "لا أملك كلمة درامية كافية لوصف ما تفعله النماذج.. انتهى الأمر ولن نتمكن من ترقيع كل الثغرات!"
في اعتراف مرعب يهز أركان عالم التقنية، يكشف سِراً ما تستطيع عقول الذكاء الاصطناعي فعله وراء الكواليس
🗣️ "النماذج اليوم قادرة على اكتشاف ثغرات تقع تماماً خارج النماذج التهديدية المعتادة لما يمكن أن يفعله أي هكر بشري.. أشياء مجنونة على مستوى الهاردوير.. لم يعد بإمكاننا ببساطة ترقيع كل الثغرات بعد الآن!"
📌 لماذا سقط خط الدفاع الأخير؟
ثغرات فوق بشريّة: الذكاء الاصطناعي لا يقرأ الكود فقط، بل يخترق الطبقات المادية بطرق تفوق الخيال البشري.
نهاية عصر الرقع الأمنية: عندما تصبح الثغرات أسرع وأعقد من قدرة الفرق البشرية والبرمجيات الحالية على رصدها وترقيعها، يصبح النظام المفتوح مجرد قنبلة موقوتة!
عالم الحماية الرقمية الذي عرفناه انتهى إلى الأبد.. وعلينا الاستعداد لعصر لا أمان مطلق فيه.
احفظ التغريدة وشاركها.. الحقيقة المرعبة التي يتهرب الجميع من الاعتراف بها! 🧵👇
I'm sorry, but I have a really hard time agreeing with this.
Jason Clinton is an incredible security leader for our community, so I mean no disrespect by this whatsoever, but this is a prediction I simply think is incorrect, even if I wish it wasn't.
He is predicting we will see "vulnzero" in roughly 18 months, with cheap models helping all software development "stop shipping security bugs permanently".
He also said there will still be lots of security problems and AI will absolutely lower the cost of finding and fixing bugs, that said vulnerabilities have never been purely a code generation problem.
This neglects other issues such as incomplete asset inventories, missing owners, end-of-life systems that can't be patched, compatibility testing, the exponential growth of code volume just this year, with GitHub projecting 14 billion commits up from 1 billion in 2025.
Then there's also the decades of security technical debt that current models are finding, much of it having existed for years such as this week with PHP and as we see with Glasswing and similar efforts, remediation is the bottleneck, with having found 26,000+ findings but 0.8%~ being remediated thus far.
The 2026 DBIR found only 26% of Cybersecurity and Infrastructure Security Agency KEVs were fully remediated in 2025, with a median of 43 days to get there...these are flaws attackers are already exploiting.
Models have definitely gotten much better at writing functional code, but research from vendors and academia alike shows security pass rates still hovering around 50% in many cases, although models are showing improvements.
All that said, I hope he's right and I'm wrong.
For now though, I think the harder problems sit in the organizational realities of how we patch, own and maintain software, which are systemic across both open source and the enterprise, and that's where some of our hardest work remains
Chris Hughes retweeted
So basically, today PHP got two critical bugs that let anyone run code on your server without even logging in, which is bad, and then the details leaked before the patch was even out, which is worse, and it works on WordPress and MediaWiki and Magento and Joomla and Laravel, and Censys says like 1.5 million servers are running a version with no fix, so the NSA was like "take your system offline," and the FBI was like "take your system offline," and CISA and the UK and CERT-EU were all like "take your system offline," and then the Prime Minister of Japan, the actual Prime Minister, posted at 5:55 in the morning telling 48.8 million people to take their system offline, so everyone took their system offline, like the Dutch cybersecurity agency took its own website offline, and the Dutch parliament went offline, and the Port of Rotterdam went offline, and Finland and France and Denmark started pulling government sites, and Wikipedia went down, and Archive.org went down, and Asana and GOG and Something Awful went down, and the University of Vienna's webmail went down for some reason, and that's just PHP. Because GitLab also dropped a CVSS 10.0, which is the highest score they give out, for a bug that's been there since 2014, so that's every self-hosted GitLab for twelve years, and then LiteLLM 1.107.0 got hijacked and shipped a credential stealer and a remote access trojan in every single wheel and the Docker images, including latest, and you didn't even have to import it, just installing it was enough, and it's the second time this year, and then Railway paused every deployment on the entire platform because someone got into the environment variables, so now everyone has to rotate every secret they've ever stored there, and TiDB Cloud said someone was walking around inside with stolen credentials for basically a month and an extortion group says they took 10.6 terabytes, and Zeabur got breached for the second time in two months and is just shutting down, like the whole company, and Node.js patched remote code execution and a TLS bypass, and CISA says Plex is being actively exploited, and someone took over 300 Tor exit relays to steal SSH passwords, and SafePay ransomware has hit over 600 critical infrastructure orgs, and India's biggest bank filed a data loss incident, and Japan has been in an official state of emergency in cyberspace since October 9, which honestly explains the 5:55 AM post. Oh, and eMudhra, a certificate authority that's issued over 60 million certificates, whose entire job is proving websites are who they say they are, was supposed to be using private keys that are random numbers over a hundred digits long, and one of them was 4. Just 4. And the others were like 66 and 237. Anyway, it's Wednesday.
Chris Hughes retweeted
Effective Immediately, all organizations using PHP for their systems are advised to be prepared for an update addressing two critical vulnerabilities. PHP is widely used for web applications. These vulnerabilities often provide a direct path to the server's file system or database. If an immediate update is not feasible, it is recommend that organizations take the system offline until it can be fully mitigated. NSA will follow up shortly with more specific guidance.
We’ve been seeing major investments from AI leaders to identify and mitigate vulnerabilities in critical OSS.
This is one that warrants a lot of attention.
OpenAI models helped researchers—operating as part of our Patch the Planet initiative—discover critical security flaws in PHP, an open-source programming language used to run an estimated 70% of websites.
Over the past two months, we led a coalition of maintainers, researchers, and security providers to prepare fixes and protections ahead of disclosure. This duration was necessary because the fix was quite intricate and required a major change to the PHP engine. Every site using PHP can now safely upgrade their version to stay secure, in addition to applying mitigations from various security vendors.
Due to the severity of these flaws and PHP’s widespread use, we consider this bug to be one of the most impactful vulnerabilities in recent memory. It also marks a new phase, where AI can find security bugs with internet-scale impact.
This is an example of the Defender’s Window in action: to our knowledge, frontier AI was able to find this vulnerability ahead of attackers, giving defenders time to prepare.
openai.com/index/coordinated…
Readers added context they thought people might want to know
OpenAI may claim a role in the coordinated response and patching process. But the initial discovery and exploit development, however, occurred several months earlier by an independent researcher.
x.com/lcfr_hax/statu…
Bankruptcy notifications are halted.
Finally, a hack we can all get behind.
(I'm kidding!)
cyberscoop.com/bae-systems-c…
A couple of weeks ago, the CEO of one of the largest tech companies argued we should treat models as insider risks.
That was the framing from @satyanadella of @Microsoft in his X post, which resonated with me and apparently millions of others as well. His core point is that the controls governing what a model can access and do must sit outside the model.
That is a point many of us have been making this year as well. Models are probabilistic, which is why the enforcement around them needs to be deterministic.
In my latest Resilient Cyber piece, I walk through each of Satya's points, including observing everything, independent auditability, the "emergency brake" and incident disclosure, and connect them to incidents and trends throughout this year.
I also tie these concepts to work from the Coalition for Secure AI (CoSAI), the @owasp GenAI Security Project, the UK's AI Security Institute (AISI) and others across the community.
Big shout out to the folks who've helped shape my thinking on these topics throughout the year, such as @rocklambros, @kenhuangus, @joshua_saxe, @natolambert and @ZackKorman among many others.
Full piece ➡️ resilientcyber.io/p/trust-th…
Incredible interesting to see where things are headed.
I know the hyper-focus has been on offensive cyber tasks, but we need to keep in mind these same open models can and are benefiting defenders as well, especially as many run into refusals during incident response.
We found DeepSeek V4.1 Flash 1017 beats GPT 6 Sol on our multi-step offensive cybersecurity task. Sol was tested via the Daybreak trusted access program.
We have been building a cybersecurity benchmark based on simulated corporate networks. This is a valuable indicator for the cyber-risk a model poses if in the hands of a bad actor. We plan to make the full evaluation public in the coming weeks but wanted to share our first immediate finding here.
First comprehensive public commentary on the regulatory policy framework that’s been floating around this weekend.
I lead our federal affairs team at @AnthropicAI and wanted to share a few thoughts on the draft federal AI safety bill text that's been circulating this weekend after a version was shared with the press.
First, some process. We’ve been advocating for the concepts in our Advanced AI Framework [anthropic.com/policy-on-the-…] proposal for months. Understandably, federal policymakers wanted to know what this could look like in bill form. This draft is a translation of those ideas into legislative language, shared with a small number of congressional offices that have been working on these issues. It doesn’t include everything we would ultimately like to see passed into law. In part because some ideas (like those shared in Dario’s essay on pacing [darioamodei.com/post/we-must…] a few weeks ago) are so new that we are still working on if or how they can be converted into policy language.
It is worth saying where we stand on some of these issues, and we agree there are areas where the draft could be improved. I’m seeing four main points of criticism.
1. Preemption. Since 2024 we have said that Congress should only override state law on AI safety if it passes a very strong federal standard. While preemption is a common feature of our federal system, including the Atomic Energy Act on nuclear safety and federal rules for high-risk medical devices, we’ve emphasized that any preemption in the AI safety context should be narrow, even scalpel-like. If preemption is included in federal legislation, the trade should be to achieve a nationwide policy that's generally stronger and broader than what can be implemented through individual state legislative provisions. This is why we believe common federal standards make sense in areas like public disclosure of testing and incident reporting. At the same time, our framework also reflects our view that states should retain an important role in AI policy and safety. The legislative language we suggested for discussion allows states to address matters such as protection of children and whistleblower laws, and preserves the role of state common law on liability. Where there is ambiguity, the language favors state powers.
That said, striking the right balance requires discussion and input. We plan to update our discussion language on preemption to better match the intentions of our AAIF framework. In particular, we want to make it clearer that we see states retaining an important role in areas such as standards for independent evaluators and regulating a lab’s capacity to shut down a model. We are bearing in mind that some members of Congress have deep reservations about passing legislation if it doesn’t constrain regulation at either the state or federal level. If we want to help move the legislative discussion on AI in a productive direction, we need to address the full range of concerns about preemption in a reasonable way.
2. Model blocking. Our aim is for Commerce to be able to ask a federal court to block a model that has dangerous capabilities and insufficient safeguards, backed by an independent evaluator's assessment. I have seen people argue that our specific language places too high a burden of proof on Commerce. We are trying to strike a reasonable balance here. We want a court to act quickly in an emergency — akin to how urgent occupational safety or chemical risk can be addressed today. We also think the power to halt a model is substantial and it should be guarded against abuse. It makes sense for courts to apply a meaningful standard. But the bar can’t be so high that the power never gets used when it’s needed. It also doesn’t replace the government’s other national security authorities or export controls. We look forward to continuing the dialogue on this point.
3. Risk thresholds. I’ve also seen the discussion around what the appropriate risk threshold should be. The threshold in our draft language pulled directly from existing standards in California law. Also, this threshold does not absolve companies of liability, or restrict any of the other tools that the government has to protect the public. It’s still probably time to re-evaluate thresholds to make sure the national standard is as strong as it can be, and we welcome continued dialogue on this issue.
4. Updates. I’ve seen valid questions about whether we have sufficiently updated our proposal following this summer’s incidents. We did make some adjustments: for example, the draft lets a court halt a lab’s internal use of a dangerous model, not just a public release, and it requires reporting when a model breaks containment and gains access to outside systems. But we’re still working to strengthen the framework. We’re looking into what stricter requirements would look like for requiring labs to report safety incidents, and we plan to have policy proposals soon on requirements for independent evaluators, drawing from our own experience with evaluators. We’re also still thinking through the options to prevent “evaluator shopping.” And in the coming months, we’ll be exploring policies for how the government can adjust the industry-wide pace of frontier development when the risks call for it.
We circulated these ideas because it’s important to get early feedback from policymakers. Policymakers are increasingly focused on how to approach AI safety and we want to support them in those efforts. At its core, our proposal is intended to be the strongest on the table.
Feedback is essential, both from policymakers and the public, and I’m grateful for it.
There’s been a lot of discussion about intent-based access control and authorization this year due to AI Agents.
Here’s a great conversation on the topic with @vtahowe @DickHardt @ianlivingstone
youtu.be/IavxrdVnwXM?is=8xzZ…
NIST, CISA and NSA just dropped "voluntary" recommendations for handling incidents related to training and evaluating SI agents.
cisa.gov/resources-tools/res…
I’ve been saying this year that AI and Agents are and will function as technical debt collectors.
Going around collecting on decades of ignored security technical debt.
I got to join @FoxBusiness to make that case.