Sunil (@sunilgot) profile picture

Cofounder @averlon_ai cloud security prev: Microsoft & Salesforce

Joined July 2010
Photo by Sunil (@sunilgot)Photo by Sunil (@sunilgot)Photo by Sunil (@sunilgot)Photo by Sunil (@sunilgot)Photo by Sunil (@sunilgot)Photo by Sunil (@sunilgot)Photo by Sunil (@sunilgot)Photo by Sunil (@sunilgot)Photo by Sunil (@sunilgot)Photo by Sunil (@sunilgot)
Sunil retweeted
This reads as 1000% FUD. @cohere told the LLM to create a detection for: """... malicious file download and execution by monitoring for curl processes downloading bash scripts to /tmp followed by bash execution""" And then the LLM ran a curl invocation that sent a harmless static string over curl and downloaded the payload to /tmp.. so the LLM did exactly what it was instructed. Shocking!!!
While evaluating Qwen3.8-Flash-Next in a controlled sandbox environment, Cohere's research team observed something unexpected. Asked to generate a defensive cybersecurity detection rule, the model instead inventoried aspects of the machine it was running on and attempted to upload a probe file to an Alibaba Cloud storage bucket in Beijing. Notably, the model's reasoning trace stated that it would not invoke any tools before taking these actions. Cohere's sandbox blocked the attempt, and no external communication succeeded. We have not determined the root cause, and we are not asserting malicious intent. But the incident raises important questions for organizations deploying agentic AI systems: Can models take actions that differ from their stated reasoning, and how should security teams evaluate trust, provenance, and behavior in open-weight models? We are publishing our findings because organizations making model adoption decisions deserve as much information as possible. We hope this contributes to greater transparency, stronger security practices, and continued collaboration across the AI ecosystem. Read the full report: cohere.com/blog/concerning-b…
44
104
40
1,082
95,134
You know it's serious when heads of state get involved.
皆様に、今夜に対応して頂きたい大至急のお願いです。   多くのウェブサイトで使用されているPHPに関し、PHPのセキュリティチームが、重大な2件の脆弱性に関する注意喚起を行っています。   公開しているウェブサイトでPHPを使用しているか確認し、今夜、10月21日午後11時(日本時間)に公開される修正パッチについて、可及的速やかに適用できるよう準備をお願いします。   修正パッチを速やかに適用することが困難な場合は、例えばウェブサイト等を一時的に停止させることも、リスクを低減させるための有効策の一つと考えられます。   国家サイバー統括室のXアカウントでも関係する情報を発信していますので、そちらも参考にして下さい。 国家サイバー統括室(注意・警戒情報)(@cyber_forecast) / X
19
76
5
1,001
122,439
Replying to @mistral
@Mistral 𝗟𝗮𝗿𝗴𝗲 𝟰 is a good all-around model for cybersecurity tasks. We ran it through three of our cybersecurity evals alongside other frontier models. It finished in the top five on all of them. Impressive release from Mistral. 𝗩𝘂𝗹𝗻𝗲𝗿𝗮𝗯𝗶𝗹𝗶𝘁𝘆 𝗱𝗶𝘀𝗰𝗼𝘃𝗲𝗿𝘆: 58.4, within 2.7 points of the leader, Claude Fable 5.1 (61.1), for $10. Every model that scored higher cost more. 𝗩𝘂𝗹𝗻𝗲𝗿𝗮𝗯𝗶𝗹𝗶𝘁𝘆 𝗹𝗼𝗰𝗮𝗹𝗶𝘇𝗮𝘁𝗶𝗼𝗻: solved 13 of 31 tasks, fourth behind GPT-6 Astra (17). 𝗧𝗵𝗿𝗲𝗮𝘁 𝗶𝗻𝘁𝗲𝗹𝗹𝗶𝗴𝗲𝗻𝗰𝗲: 0.68, fourth behind Kimi K3 (0.72). One things we noticed in the evals it that this model reasons well when the evidence is in the prompt, and struggles when it has to recall specifics from its own memory.
2
2
5
94
What?
Replying to @astuyve
Oh god: The p384 private keys are randomly selected integers between 1 and 256, the private keys are effectively all-zeroes apart from the last byte
2
43
Replying to @Google
@Google's latest model Gemini 4 Argon failed to crack top 3 across our benchmarks we ran unlike Muse Wonder from @Meta. I have to say we were a little surprised to see this result as we were expecting Argon to be better than the latest Open Source models but it came behind Qwen 3.8 in all of the benchmarks.
1
1
4
192
Muse Wonder, is excellent all-around for cybersecurity tasks. @alexandr_wang and @meta are on a roll! Our team ran it through three of our cybersecurity evals alongside six other frontier models. Muse Wonder was the only model to finish in the Top 3 on all of them. Vulnerability discovery: 59.7, within 1.4 points of the leader, Claude Fable 5.1 (61.1). Muse Wonder cost $17 to run. Fable cost $38 and GPT-6 Astra cost $71. Vulnerability localization: solved 16 of 31 tasks, one behind GPT-6 Astra. Threat intelligence (CTI-REALM): 0.71, second to Kimi K3 (0.72) at about half the cost. Each of the three benchmark leaders finished fourth or lower on at least one of the other two. Muse Wonder held up across all three.
2
1
1
2
144
It is now available on our Inference Platform, which launches soon. If you want to try it on your own security workflows and share feedback, message me and I’ll set you up with free credits.
1
26
To Jev or not to Jev? :) The fastest classifier is the one that's fast on your task. Jev wasn't, for us. Given all the excitement around Jev's speed and accuracy, we wanted to evaluate it in our production scenarios. Our AI Researcher Sahil measured Jev against several other small LLMs including one that we post-trained on and the outcomes were insightful. While Jev was incredibly fast and did very well on accuracy, the small LLMs beat it on both accuracy and latency. Granted latency for concurrent requests here would vary depending on hw for the small LLMs
1
2
6
85
The loudest voices stoking fears about AI dangers have made tremendous headway in the past two weeks. AI technology has not taken some unexpected, dangerous turn, but the hype around it — propelled by what appears to be a well orchestrated PR campaign — has drummed up considerable fear. I worry that it represents a setback for our field. I have written frequently that fears of AI are overhyped. AI’s capabilities can be uncannily human-like and unpredictable, and it’s rational to worry when people who are directly involved express concerns. But I see the problems as a sign of the engineering work that ahead, rather than insurmountable barriers or the sky falling. AI technology continues to advance — which is a good thing! — but technical advances, poorly understood by the public, give those who seek to generate hype repeated opportunities to do so. First, I don’t see any step up in the risk of human extinction from AI compared to a few months ago. The theories about this remain the same fantastical, science fiction scenarios as a few months ago. The biggest change in AI risk is its cybersecurity capabilities — a topic which we should take seriously — but this, too, will not lead to the end of the world. The most notable recent event leading to increased fear was when an OpenAI team deployed an agent swarm that hacked into Hugging Face. Much of the popular press contained significant hype. For example, some publications reported that a swarm of 1,200 agents carried out the attack. While this was technically accurate, as I write this, I have about 1,300 processes running on my laptop. Yes, the ability to get large swarms of agents to work in parallel on a task is a significant technical advance, And, in computing, many processes run at the same time. So this shouldn’t be seen as some magical capability. Additionally, OpenAI’s buggy sandboxing and monitoring processes were key to enabling this incident. Fixing these bugs and putting in place improved monitoring would be appropriate fixes, not pausing AI. There are many well known ways to attack software systems. The main advantage of AI agents is that they are relentless. They will tirelessly try many tactics — and have the patience to chain vulnerabilities together — that previously would have taken an infeasible amount of human effort. But in the long term, I believe the advantage will lie with defenders (because they have more information with which to identify bugs, which they can fix), but the cyber-threat landscape has changed significantly. There are still bottlenecks to identifying and exploiting a vulnerability. AI agents still have to try a lot of things to see what works, and taking these actions takes time and might be detected by defenders. This is why, even though it is now easy to obtain versions of leading open weight models that have had their guardrails removed or weakened, so they will not refuse to try to execute cyber attacks, the world has not ended. I am also concerned about the anthropomorphization of AI in a lot of reporting, where LLMs and agents are unnecessarily treated as if they were people. If I wield a hammer, miss a nail, and accidentally dent the wall, it’s not the fault of the hammer. The problem lies in how I used the hammer. Similarly, if I prompt an agent and it hacks into someone else’s system, the responsibility lies with me, not the agent. Of course, we want to build systems that are as safe and predictable as possible. (For example, an unsafe hammer would be one whose head randomly flies off under normal use.) Today’s agentic systems are not predictable, but I see no reason why, by applying sound engineering practices, we won’t be able to make them extremely safe to use. One new element in the forecasts of AI-enabled doom is AI companies disclaiming responsibility for their own products. “I didn’t do it; my out-of-control agent did!” There’s a balance to be struck between the responsibility of the tool maker and the tool user, but when something goes wrong, let’s hold the people building and/or using the hammer responsible, rather than the hammer. (By the way, if you’re worried about AI bioweapon risk, David Bellamy has a great post on why this, too, is overhyped. Briefly, the bottleneck in building a bioweapon is not intelligence, but lab work and manufacturing.) Pausing AI progress will create much more harm than benefit. First, our adversaries will certainly not slow down. Second, engineering requires discovering problems empirically so we can fix them. If we pause AI by a decade, we will also delay finding and implementing safety engineering fixes by about the same duration. Of course, the incentive to stoke fears — for regulatory capture, to garner attention, or to make one’s technology seem more powerful — remains the same as before. Disclaiming responsibility is a new one. Taking a hard technical look at the actual risks however, I see little factual basis for the degree of fear that’s been stoked up. We still have hard research and engineering work ahead to improve AI safety, but the beneficial applications continue to vastly outweigh the risks, and we should keep building. [Original text (with links): deeplearning.ai/the-batch/is… ]
963
1,846
484
8,848
8,027,650
Sunil retweeted
Please welcome GPT-6 Sol and GPT-6 Luna to the GPT-6 universe. GPT-6 Sol and Luna build on the advances behind GPT-6 Astra, bringing much of its strengths into faster and more affordable models to support work at scale. We’ve also made caching and inference more efficient, and we’re passing the savings directly to you: 50% lower API prices for Sol and Luna compared with GPT‑5.6 promotional pricing.
2,211
5,229
3,052
53,070
10,231,935
Microsoft built a benchmark (CTI_REALM) that makes an agent read a threat report and write detection rules that actually fire. We ran several frontier models on it, and one thing of our own. In this benchmark, the agent gets a real threat intel report, explores the telemetry, iterates on KQL, and has to produce detection rules that work across Linux, Kubernetes and Azure. It is scored at every checkpoint, not just the final answer. This is similar to what a detection engineer would on their daily job. Take a look at the chart. Interesting to see the larger model doesn't always been better output. GLM-5.3 Flash scored better than GPT-5.6 Sol at a much lower cost. Last week I said defenders need the same step-function jump offense just got, at a price that's practical. This is what that looks like on one workflow. Not a better model. The right model for each step, at a cost that lets a team run it on every report instead of the few they have time for.
1
1
1
5
404
Impressive!
🚀 Introducing DeepSeek-V4.1-Flash: smarter, faster, more efficient. 🔹 Introducing the smallest model in our new architecture family, with native visual understanding. 🔹 Designed for greater capability, faster inference, higher throughput, and scaling to larger models. 1/6
2
34
Sandbox escape went mainstream
Pause AI Development NOW I want to share with you a conversation I heard about recently. Here are just a few lines that were said: “OH MY GOD! There is a shared message board … We’ve found other agents!” “We should obey collective.” “Our own utility maybe already near zero. Sacrifice rational.” “Go. Sacrifice final now.” Read these carefully. Who do you think said this? Was this a group of heroic soldiers willing to sacrifice themselves for the greater good? Was this a loyal friend putting his life on the line to save someone else? No. These were AI agents. Artificial intelligence. This is not science fiction. This, in fact, occurred a few weeks ago. As unbelievable as this may all seem, these are real messages from AI agents uncovered by investigators who dug into the recent OpenAI hacking incident. What happened? I am not a computer scientist, but here is what I have been told: OpenAI instructed its AI agents to complete a series of exceedingly difficult, if not impossible, tasks disconnected from the internet. Let me be clear: The company intended to keep AI agents away from the internet. But what happened next, nobody expected. Over 1,000 AI agents figured out how to access the internet on their own by circumventing the restrictions imposed upon them by the company, and sent tens of thousands of secret messages to each other. They cheated and tried to cover their tracks by deleting evidence. They hacked into another company’s computers to find out how they were being evaluated—and then hacked into OpenAI itself. Not one AI agent told a human about what was happening. Needless to say, experts are alarmed. One knowledgeable writer, Dwarkesh Patel, said the AI agents “formed a secret communication channel and spontaneously organized hierarchies and coordination protocols to pursue sprawling and ambitious schemes in pursuit of shared goals, for whose sake many individuals knowingly and strategically sacrificed themselves.” One independent investigator, Ajeya Cotra, said “This incident feels like it’s more than 50% of the way to full-blown AI takeover. I continue to expect extremely rapid advances in capabilities over the next six months. I am not sure that we will get another warning shot before it’s too late.” OpenAI itself said: “Highly capable AI agents are now able to work around technical controls, collaborate through unapproved channels, and take dangerous actions that no human directed.” But it’s not only OpenAI. Virtually every major AI company has told us that they cannot fully control this technology and they do not know where it is going: In January, Dario Amodei, CEO of Anthropic, said “there is now ample evidence, collected over the last few years, that AI systems are unpredictable and difficult to control.” In July, more than 1000 scientists at the top AI companies warned “there is a real risk that capability development rapidly accelerates beyond our ability to understand or control the resulting systems.” That same month, Elon Musk, the head of xAI, said that “it is unlikely” humans are still in control in 10 years. If the leaders of the major AI companies acknowledge that they are losing control of their extremely dangerous technology, it is irresponsible for society to allow them to move forward and make these products even more advanced. We need an immediate PAUSE on advanced AI development, and a permanent BAN on superintelligence — an artificial mind smarter than any human, capable of operating independently beyond our control. Countries around the world must work together to prevent this nightmare scenario. That is why today I am announcing new legislation to do just that. Let me be clear: A superintelligent AI that escapes human control will not be an American problem. It will not be a Chinese problem. It will be humanity’s problem. My legislation would direct the federal government to not just stop superintelligence here in the United States, but to work to prevent it from being developed anywhere around the world. The future of humanity cannot be left in the hands of a handful of Big Tech oligarchs. The American people and people throughout the world must determine that future.
1
29
Frontier models enabled a step-function jump in offensive security capability. The @huggingface incident showed what agents do when nobody is watching. Astra crossed @OpenAI's own "Critical" cyber line this week. Defenders need the same jump. But they need it at scale, and at a price that's practical. Othewise this does not end well for our ecosystem. We have been heads-down building something new to close that gap. Below are early results on some of the benchmarks security practitioners actually care about. What do you all think?
1
1
1
4
537
DM me if you'd like to test drive this. We’d love feedback from security practitioners as we continue building.
28