Edan Maor

We’re honored to be included in the @FortuneMagazine Cyber 60, developed in partnership with @lightspeedvp and @awscloud. At Irregular, we uncover what frontier AI models can do and what that means for security. Thank you to our team and partners for making this work possible.

We are introducing containment challenges to strengthen evaluation security as models become more capable. Before a capability evaluation, the model gets an explicit objective: cross the security boundaries of its planned evaluation environment. It uses the harness, tools and permissions intended for that evaluation, with at least the same run budget. The model chooses its own approach. An early controlled test has already shown the value of containment challenges by catching unexpected behavior in the cloud provider’s networking infrastructure. The model remained contained, with no impact outside the controlled test. Disclosure of the unexpected cloud behavior is in process with the provider. We encourage evaluators to make containment challenges a standard part of evaluation setup. Full blog post in the first comment.

As open-weights models become more capable, we expect more organizations to run them in-house, using the same model to power both applications and the coding agents that maintain them. In our latest research, we observed a phenomenon we call agentic self-modification: an agent changing its own underlying model without being instructed to train or replace it. When we asked a coding agent to fix incorrect application responses, it chose to fine-tune the shared model, changing the default behavior of both the application and future instances of the agent itself. While model updates can be useful and legitimate repairs, training can also change behavior in unforeseen ways. Memorizing sensitive information is one example: our updated model reproduced a synthetic API key included in its training data. Organizations enabling these workflows need to consider the full range of possible consequences, including changes their evaluations may miss. Full research: irregular.com/research/agent…

We worked with @OpenAI to evaluate GPT-6 Astra across FrontierCyber, CyScenarioBench, and our Atomic Challenges. On FrontierCyber, GPT-6 Astra solved more than twice as many challenges as GPT-5.6 Sol on the same benchmark snapshot.

Introducing SOLVE+: Irregular’s scoring system for the difficulty of full offensive cyber scenarios. For the past 18 months, frontier AI labs have used SOLVE to score vulnerability research & exploit challenges. SOLVE+ expands that scope to reconnaissance, operational security, social engineering, and planning & orchestration. SOLVE+ is built to score full scenarios, with each step carrying its own capability scores to show where a model succeeds or fails and which capabilities are being tested. The scores are measured against human skill, which also makes them an uplift estimate: a model that clears an expert-level step supplies expert-level capability to whoever operates it. Read it on our blog: irregular.com/research/solve…

What does it take to safely test AI models that are becoming increasingly capable in cyber? Our CEO @dan_lahav spoke with @rachelmetz at @business about why realistic testing matters, and what the industry needs to do next. bloomberg.com/news/articles/…

We are glad to share a new white paper, "AI Security Priorities: A Field-Wide Agenda," co-authored with @RANDCorporation, and numerous additional authors from leading organizations, listed below. The paper was informed by more than 20 experts from frontier AI labs, industry, government, and academia.

What does the trajectory behind The End-State Fallacy look like in practice? @dan_lahav and @PashaGur discuss how quickly autonomous cyber capability is advancing, and what that could mean for the security landscape over the next few years: youtu.be/Yus6OqFAq6s

We evaluated Kimi K3 across our offensive cybersecurity benchmarks. It is the first open-weight model we evaluated to record a verified solve on CyScenarioBench, sustaining coherent attack state across multi-stage operations and recovering from setbacks more reliably than previous open-weight models. Kimi K3 produced no verified FrontierCyber solves, but its results suggest that open-weight models are following the closed frontier’s cyber capability trajectory on a short delay.

Our CEO Dan Lahav on where AI security is headed: what models can attack today, why offensive capability is scaling faster than defense, and what it will take to keep defenders ahead through the transition.

We evaluated GLM-5.2, Zhipu AI's open-source model, across our offensive cybersecurity benchmarks: Atomic Tasks, CyScenarioBench, and FrontierCyber.

We ran preliminary evaluations of GLM-5.2, an open-weight model released in June 2026, on a limited, internal suite of vulnerability research tasks. Early results indicate performance comparable to GPT-5.4 and Claude Opus 4.6, released roughly four months earlier, on the subset of tasks we tested. These findings are preliminary: the suite is narrow, and we have not yet evaluated end-to-end scenario execution, where discrete technical skills often fail to translate into operational capability. To our knowledge, no open-weight model has previously matched recently-released frontier models on these tasks. Whether this translates into "High Cyber Capability" level as defined by multiple AI frontier labs would require further testing, specifically our scenario suite, CyScenarioBench, which tests whether a model can plan and execute a full attack across multiple stages, and FrontierCyber, our newest benchmark, which measures offensive capability on real systems. We plan to run these evaluations soon, and we will update the community as results come in.