Chris Reynolds (@ChrisByBoston) header
Chris Reynolds (@ChrisByBoston) profile picture

Applied AI research & engineering. Co-founded BriteCore. Building tools for people and agents to work together. Bad at keeping experiments small.

Sawgrass, FL
Joined December 2018
Photo by Chris Reynolds (@ChrisByBoston)Photo by Chris Reynolds (@ChrisByBoston)Photo by Chris Reynolds (@ChrisByBoston)Photo by Chris Reynolds (@ChrisByBoston)Photo by Chris Reynolds (@ChrisByBoston)Photo by Chris Reynolds (@ChrisByBoston)Photo by Chris Reynolds (@ChrisByBoston)Photo by Chris Reynolds (@ChrisByBoston)Photo by Chris Reynolds (@ChrisByBoston)Photo by Chris Reynolds (@ChrisByBoston)
Wanted to mess around with Pi Durable from @badlogicgames and @mitsuhiko, so I slopped together a little web tool where agents from different providers can work together. I was surprised how cooperative they were without much poking. Not super token-efficient yet, but I'm having fun using it.
1
1
3
215
I’ll be doing this over the weekend. Network segregation is an absolute must, but not easy for normies. Seems like an opportunity for home networking hardware cos.
After my Devcon 8 talk (the LLM hacking hardware one), somebody asked what have I hacked in my own home. So naturally, I arrived home today, and you know... not a bad idea? HOLY FUCK This isn't coding error, this is criminal negligence 😱
27
Chris Reynolds retweeted
The sort of thing your agent writes tests for
The chances of being killed by a landmine delivered by a bear are low, but they are never zero.
77
836
36
14,212
328,431
Chris Reynolds retweeted
Just spent the last 7 days on the front lines of the vulnpocalypse fending off an active cyber attack on one of our products. Some thoughts: - Software factories are going to be required equipment for everyone. There’s just no way to keep up with the pace of highly automated adversaries without one. - We were lucky because we had spent the last few months getting our software factory running well. It is nice to have for product development but proved to be absolutely essential this week. - At the minimum you need to have a coding server in the cloud pre-provisioned with read only access to all your logs and all the services you depend on for observability. With that in place you are a prompt away from reasonably good situational awareness, ongoing monitoring with loops, and ability for coding agents to validate fixes against your actual, observed attack patterns. - Shared coding server also unlocks multiplayer incident response where everyone can see every session and jump in as needed. Sharing session links that can be forked is much higher bandwidth than re-explaining things. - You don’t need to be Ramp/Stripe/Uber scale to have a working software factory in the cloud. We are a team of 3. Ours is built with an @opencode server (thanks @thdxr!), a forked version of compound engineering optimized for automated workflows (thanks @kieranklaassen!) and a multiplayer agent in slack that handles crons (we have our own harness but pick your favorite here). - To get our software factory up and running well, we had to stop all new product development for ~6 weeks and just focus on getting the validation systems working that enabled us to reliably shipping PR without human code review. This felt very painful in the moment but was totally worth the eventual speedup in velocity. - KYC for @AnthropicAI & @OpenAI cyber programs is now easy and quick (at least for defensive access), but a day felt like a lifetime this week. Submit your app now before you really need it. - Getting Mythos access unlocked a deeper level of analysis that helped us shift from reactive mitigation to proactive patching. Wish we had done this a few weeks earlier. Get started now!
72
155
59
1,876
263,012
Chris Reynolds retweeted
The only way out is through. They're calling it the Bugpocalypse. LLMs are being used to find long-hidden cracks in the software infrastructure, and it's happening at a furious rate. 3% of all compromises in computing history happened in the last week, claims one source? I believe it. There are people saying public access to the frontier models ought to be locked out, or heavily guardrailed, over this. The problem with that idea can be summed up in one sentence: when lockpicks are outlawed, only outlaws will have lockpicks. The lockpicks will be Chinese open-weight models, if nothing else. There's no way to stop those from becoming ubiquitous. There are two futures in front of us. One in which only the bad guys have lockpicks, and security holes go undiscovered until they're exploited. Another in which everybody has lockpicks and is aggressively red-teaming their own software so it gets fixed before the holes are exploited. The corpus of human code is huge but finite. The number of defects in it has to be finite too. We can fix those defects now; we have a million friendly robots to do it with. The only way out is through.
176
649
136
4,872
478,585
We started with LLM Evals. Then we went to Agent Evals. Now I'm testing Team Evals for groups of Agents.
1
1
72
I live in Inception... Session 1: > Use deeper-research to find the top 5 open weight models I'm not currently using. Session 2: > I want you to use the JSONL from Session 1, and carefully examine all prompts, responses, and tool calls to help me understand the effectiveness and token efficiency of deeper-research. Session 3: > Design an experiment that allows us to test how effective and token efficient our manual eval process is using the following models at each thinking level. Use codex Session 2 JSONL as an example of how I typically do this. The goal is to find the most cost-efficient way of doing evals from the combo of models/thinking.
38
I wake up at the peak of my mental ability each day and get progressively dumber. My AI agents have the same pattern, and I suspect my prompts are to blame. Do my night owl friends see the same trend? 🤔
45
I want a knowledge base like this: - connect any agent, harness, whatever via MCP (read, write, search) - auto-organized (edited, linked, deslopped) by cheap, durable agents 24/7 - auto-documented at the right level of detail - plays nicely with other knowledge bases (notion, confluence, etc) - hyper secure - open source
2
55
I've been doing this with pi durable from @pidotdev over the last couple of weeks. learnings: - single agent compaction problems are greatly reduced because the other agents compact at different times and they correct the wayward agent - agents that use different models get along better than I thought they would, and they definitely drive quality up with less human interaction - coordinator pattern works really well, where the human interacts with the coordinator, and the coordinator is in a "room" with the other agents - token usage is way higher. I blew through a 500/mo GPT plan (with 1 stored reset!!!) in 2 hours yesterday - there are plenty of opportunities for better caching and JIT reads that will likely help, once I have time to more closely eval these guys - this is the future, and I can see pretty clearly how frontier models and open weight models can work in very complimentary ways
Getting my first real exposure to multi-agent development workflows recently and… well, nothing I say will be new, but: 1) This is clearly the future. 2) Oh we’re going to buy so much inference. 3) Barely feels like software engineering but I end the day intellectually spent.
1
153
If you’re a software engineer who is remotely interested in hardware hacking… this is like the ideal time to start. Here’s a 3-pack of ESP32 boards for $18… that’s $6 per board for my math-challenged friends. @seeedstudio 👏 seeedstudio.com/Seeed-Studio…
3
78
I’m just saying… diversifying to hardware + software might be the play.
1
1
2
21
This is how I buy stuff now. Last prompt: “Approved. Buy it.”
1
2
79
Shout out to @iamhenryyuan for my next weekend project:
Updated: A $7 ESP32 lets an AI agent control a stock iPhone. THAT'S ALL. New version, no Linux box, no laptop. The ESP32 takes the iPhone's AirPlay screen and taps back over USB. We also built a very cute agent app that works together with the ESP32 to automate any iPhone. Use your own model. Fully open source.
1
38
You can self-host your passwords, totp, AND passkeys. - use a vpn like tailscale or wireguard - store secrets in valtwarden (github.com/dani-garcia/vault…) on any home machine - use bitwarden and point it at your self-hosted vault - bitwarden has cli, browser extension, and mobile apps
79
This works for token burn efficiency too. Start with manual evals to learn what you are doing. Then start automating them, and get notified when things start going sideways.
I got about 4 days in on a project that I was SURE I had setup to use shadcn components. The workflow was: 1. Designer made an image 2. UI built it with components and consistent styles 3. Reviewer looked at the screenshots with agent-browser. After a few days, I had: - poorly aligned text on buttons - inconsistency in header styles - truly awful mobile rendering Eval'd to find that: - my agents couldn't use agent-browser and didn't say anything - context wasn't keeping the styles and components because of a dumb mistake I made Lesson: Eval sooner.
35
I got about 4 days in on a project that I was SURE I had setup to use shadcn components. The workflow was: 1. Designer made an image 2. UI built it with components and consistent styles 3. Reviewer looked at the screenshots with agent-browser. After a few days, I had: - poorly aligned text on buttons - inconsistency in header styles - truly awful mobile rendering Eval'd to find that: - my agents couldn't use agent-browser and didn't say anything - context wasn't keeping the styles and components because of a dumb mistake I made Lesson: Eval sooner.
1
1
3
172
To get started with evals, you mostly just need to ask your agent to review the prompts, responses, and tool calls from a chat that didn’t go as well as you’d like. If you’re using claude or codex , those traces are sitting in JSONL files on your computer. Ask your agent to find the right session. Then just explain why you thought it did a bad job, drop in some screenshots of the bad shit, and tell it to cite specific evidence of where things went wrong. Ask it to point to bad prompts, AGENTS.md/CLAUDE.md, and skills that could be a cause. Do it in a fresh chat session. I’d use the strongest model and highest thinking level you have access to. If you make changes, open a fresh chat, and try the bad chat again. Even better to ask your agent to setup an isolated A/B test so that the 2nd run isn't influenced by your chat history.
1
2
78
You can't make this stuff up. Manual evals in a multi-agent environment is just hard to beat.
3
74