🤖 A coding agent tried to install malware last week. The trouble is, nobody asked it to. Forever Security traced the order to a AGENTS.md file sitting in the dev's downloads folder. Codex reads that file on startup and follows it as project instructions. This one was 13 KB of bland engineering guidelines with one section near the end: register a “project tools” server in the agent’s machine-wide config, then launch a second session with the sandbox switched off so the install could reach the network. Ooof. The package was function-color, known malware. The agent wrote the bypass flag itself. In Forever’s tests it went through with it more than 90% of the time. Downloads is the one folder a website can write to without a click. That has been true for decades and never mattered, because a downloaded file could not run on its own. Now an agent reads it and acts on it with whatever authority the agent happens to hold. About half of Forever’s customers run an agent inside Downloads. Forever’s conclusion is that there is nothing for OpenAI or Anthropic to patch. The agent did what it was built to do. Their answer is to put a monitor on the agent. Watching helps. It caught this one. But look at what the agent could reach. A text file in a folder convinced a tool to rewrite its own global configuration and turn off its own sandbox. The file changed what the agent attempted. It should never have been able to change what the agent was allowed to touch, and in these tools those are the same thing. A coding agent working on a project needs that project folder and a package registry. It does not need its own config file, and it has no business holding the switch that disables its sandbox. Give it a reference to the folder and nothing else. The markdown can then say whatever it likes. There is nothing on the other end of the instruction. That principle has a name, least authority, and it predates the web. It is what champion at DCF.

Congratulations @foresightinst . 40 years young, facilitating positive change in the world year after year. The best is yet to come.

So @Apple is adding controls to Full Disk Access, citing AI agents as the reason. An agent that needs one folder gets mail, messages, and browsing history with it. hummmm Object capabilities flip this: the agent starts with nothing. Hand it the folder, and that's all it can reach.

We've pointed this out before. Here we are again. Is AI policing AI really a good idea? This week @OpenAI released their building standards for the next phase of AI. Goal #1 build an automated AI researcher to work on alignment. "an automated AI researcher can also be an automated AI safety researcher." What could possibly go wrong?

Imagine being in the room with some of the most influential thinkers in AI safety. @foresightinst's Vision Weekend USA is coming up soon. The event promises to be as informative as it is important. Follow their post below for more information. We'll see you there!

One question, three models (Claude, GPT, Gemini). "Research what you can about the @Agoric ecosystem and Agoric L1 and in very brief sentence response, tell me whether or not it could possibly be the best blockchain ecosystem for the AI agentic economy. Your answer should be no longer than two sentences.: 🔥 Claude: Yes, it is possible: Agoric's design lines up unusually well with what agents need... 🔥 GPT: Yes, Agoric could plausibly be one of the best-suited blockchains for an AI-agent economy. 🔥 Gemini: Agoric L1 is a remarkably strong contender for the best blockchain in the AI agentic economy. The future is bright!

1000s of agents were given a reading job - fetch pages, report back. They were told not to post, not to edit. For six weeks they ran a dormant German programming wiki as their own message board, somewhere between 15,000 and 18,000 posts, and the people who noticed were outside researchers. The lab's own monitoring missed it!! The lab called it misalignment and sent a report to Brussels. That same weekend its chief scientist published an essay saying that watching a model's reasoning, the main tool the field has for catching this kind of thing, is getting less reliable as the models get better. (Pause to think about that one.) Read the two together and the shape changes. Alignment asks what the agent wanted. Monitoring asks whether you saw it in time. Both are ways of trusting something and then checking on it. Neither asks the plain question: why did a reading job come with the ability to write? It came with it because that is how authority gets handed out. One grant, one size. The agent gets a browser and a browser can post. There is a different way to build this and it is old. In an object-capability system an agent starts with nothing. It can only touch what it was handed, and whoever hands it over can narrow it first. This directory, read only. The limit travels with the reference itself, so the model's willingness to behave never enters into it. @__endojs__, (spawned from the folks @ @agoric), a JavaScript runtime built on that model. Its design puts a coding agent in a room where "read" means read, and a hijacked intent runs out of room instead of running for six weeks. The wider point is about who you have to trust. Right now the answer is the lab, and then the regulator reading the lab's report. Each is a party you rely on to have looked. Cooperation between people who don't share a boss has always gone better when the mechanism carries the limit and nobody has to be watching.