Building AI companies between tee times. Agency/lab: @theGPTlab Portfolio: @Onboard_Fi @TractionDesk @goGameSwing
Joined May 2014
- Tweets9.7K
- Following2.2K
- Followers1.6K
- Likes44.7K
Bobby Gilbert retweeted
Build and own your harness, folks.
Very few people understand the magic behind customizing and optimizing an agent harness for your work.
You can start seeing incredible results quickly, even with a minimal harness. Just better code, better outputs, better costs, better writing, and so on.
As an example, I see way too many people complain that they are tired of their out-of-the-box agent harness producing verbose outputs. If you built a harness, that's easier to fix with a system prompt optimized for you.
This is why Pi is getting insane adoption these days. You want to control your harness. I agree that it's hard to optimize where things stand, but self-improving algorithms will make it easier to learn from trajectories. And if you use local models, you can tune all of that end-to-end (through a co-evolution of the harness and the models) as you work with them.
Look into open-source projects like Eve, Exo, and Prime Agent, and you will see what I mean.
OK. It's time to rethink this.
I've spend the last several weeks working on a harness that tightly constrains the agents to work the way that I want them to work. I set up all kinds of gates, and tests, and tools, and protocols, and ...
And while I was heads-down getting that to work, the agents got a LOT better. So much so that when I came up for air, the need for my harness was obviated. Indeed, the need for _any_ but the most liberal of harnesses may be obviated.
Just how good these things have gotten blows me away. I have had long debates with grok and codex about the structure of systems -- as if they were senior engineers. They often disagree with me and have their own perspectives. I have, more than once, found myself agreeing with their views.
I have not given up on constraints and tooling. Unit testing is still important. So is CRAP and Mutation testing. These tools still find bugs and offer useful constraints, though they can leave scars.
However, the agents have gotten so good that I can now give one a very significant task with a few guidelines and it will faithfully implement it. I can walk away for 40 minutes and when I return it will be done. CRAP will be satisfied, Coverage will be high, and Mutation testing complete. The architecture will be clean, and the code will be very good.
The end result may not behave perfectly, but it's so close that a couple of tweaks usually puts it into place.
What does this mean going forward? I'm not sure. But I'm beginning to think that harnesses should not treat agents as components within a software design.
Bobby Gilbert retweeted
OK. It's time to rethink this.
I've spend the last several weeks working on a harness that tightly constrains the agents to work the way that I want them to work. I set up all kinds of gates, and tests, and tools, and protocols, and ...
And while I was heads-down getting that to work, the agents got a LOT better. So much so that when I came up for air, the need for my harness was obviated. Indeed, the need for _any_ but the most liberal of harnesses may be obviated.
Just how good these things have gotten blows me away. I have had long debates with grok and codex about the structure of systems -- as if they were senior engineers. They often disagree with me and have their own perspectives. I have, more than once, found myself agreeing with their views.
I have not given up on constraints and tooling. Unit testing is still important. So is CRAP and Mutation testing. These tools still find bugs and offer useful constraints, though they can leave scars.
However, the agents have gotten so good that I can now give one a very significant task with a few guidelines and it will faithfully implement it. I can walk away for 40 minutes and when I return it will be done. CRAP will be satisfied, Coverage will be high, and Mutation testing complete. The architecture will be clean, and the code will be very good.
The end result may not behave perfectly, but it's so close that a couple of tweaks usually puts it into place.
What does this mean going forward? I'm not sure. But I'm beginning to think that harnesses should not treat agents as components within a software design.
Bobby Gilbert retweeted
The next era of enterprise software design is Civilization-style interfaces on top of company brains and agents.
Bobby Gilbert retweeted
There are seven main steps for your company to go from tool adoption to agentic org.
Most companies sit on step one, shopping for step seven technology. They need step four discipline.
Step one: tool adoption. AI tools exist across the org, humans run each one. Developer closes the lid, automations die. Somebody goes on vacation, the workflow goes with them.
Step two: context layer. Knowledge graphs per department, each behind its own API gateway. Sales queries engineering through an API. Separated domains keep agents from hallucinating across departmental boundaries.
Step three: harness engineering. Build evaluation criteria, routing logic, and quality gates before you pick a model. The LLM comes in last. Your harness is your IP and your capex. The model underneath is a commodity you swap when something better ships. If changing the LLM breaks your system, you built a dependency, not a platform.
Step four: go headless. Host every automation outside a laptop. It fires on schedule, on conditions, without a human logging in at 9 AM. Companies stall here because nobody forces the migration from "script on my desktop" to "service that runs on its own."
Step five: governance. RBAC on every knowledge structure from day one. TTL on every agent. Dead agents are tech debt with active API keys. Past 100 tools, you need lifecycle management. Retire anything that hasn't run in 90 days. Deprecate anything whose eval scores drop below threshold.
Step six: agent-to-agent. Stitch headless instances together within a value chain. Agent A finishes, hands off to Agent B. An LLM sits between them as a quality gate, judging every handoff before it passes through.
Step seven: step functions. Orchestrated, semi or fully automated workflows. Agents handle volume. Humans handle exceptions. The org runs on agent output.
We walked a VP of Engineering through all seven.
Told him: "You're on step one. But over a hundred tools means step four is closer than you think."
Bobby Gilbert retweeted
Systems of record have never been more important than in a world where you have AI agents that will do 100X more work on these platforms than people ever did.
Agents will be querying the data in these systems, processing tasks, executing workflows, collaborating with human and agent users, and more.
Thus, the governance, reliability, security, access controls, and business logic remain critical more than ever — the OpenAI Hugging Face incident is just a small peek into what the future will look like with agents running around trying to execute on their goals.
But this only works for the systems of record that respond with the right product experiences, APIs, and business models to support agents on and off their platform executing in these systems.
Huge moment of opportunity and disruption right now in all of software.
Bobby Gilbert retweeted
If I ran an engineering team and I found out my $200k per year devs were spending time vibe-coding SaaS clones to save the company $299 per month, instead of working on improving our own product...
I'd be pretty mad about it.
Bobby Gilbert retweeted
Travis is an apex predator put on ice in 2017 and only recently reactivated from cryogenic slumber.
Standards for male behavior have moved so far that he's actively perplexed by it.
Our current ecology was last home to performative frat bros nearly a decade ago.
So when Travis tells TBPN he jet skis to work, and Jordi + John ask whether he does his best thinking on the jet ski, he pauses.
You can visibly see him get confused, with an expression like "What the fuck are you guys talking about?"
Then he quite literally voices his frustration: "No guys...it's a fucking notch on the belt. Who else do you know who jet skis to work? Nobody. I'm the guy."
Even the literal technology brothers are incapable of fathoming a man who does things just for the bit.
Travis in 2026 is like the T-rex from Jurassic Park 2 who finds himself in modern day San Diego being like, yo, what the FUCK is this place?
And then just stomps everything in sight.
teapawt.net/tbpn/status/2081039302…
.@travisk's commute takes 5 minutes on a jet ski going 70 mph on Lake Austin.
"Who else do you know is jet skiing to work? Nobody. I'm the guy."
Bobby Gilbert retweeted
Zuck just wrote an e/acc manifesto and you're blackpilling anon?
Give it a read: wsj.com/opinion/the-ai-futur…
Bobby Gilbert retweeted
Give it a read: wsj.com/opinion/the-ai-futur…
Bobby Gilbert retweeted
lots of big announcements breaking so stay tuned. cerebras on thursday (which means sol max at 10x speed) i’m the first to break the ssi news and 4 billion dollar investment, lots more to come there and on gpt6. make sure to subscribe and enjoy the summer.
the next 6 months of progress will eclipse the previous three years. it’s going to feel incredibly slow looking back. which is startling on its own. i’ll see you there frens.
oh, some big updates on fable as well. and from google / open source models but i’ll be reporting more on news that sig shifts the frontier so i’ll rarely speak about those.
time to lock in i think.
Bobby Gilbert retweeted
as exclusively broken by strawberryman two weeks ago.
huge announcement from ssi as their research reaches inflection point.
huge moment in the race toward super intelligence.
rooting for you ilya!
We are announcing a long-term strategic partnership with NVIDIA. NVIDIA is making a substantial investment in SSI that will let us 10x our compute in the next 12 months. We reached the point where our research is worth scaling and with this partnership we will be able to. We are honored by NVIDIA’s conviction.
Bobby Gilbert retweeted
software engineering is the profession of automation
Something I have been thinking about: in the past, the best engineers I knew spent a lot of time automating their work in various ways. Better vim/emacs automations, writing lint rules to catch repeat code issues, building up a suite of e2e tests so they don't need to smoke test the app manually. These kinds of things were the highest leverage activities an engineer could do, because it multiplied their own output, which in turn meant they could build more things.
I think many of these automations have become even more important now. This is true for a number of reasons.
First, infra and DevX automation speeds you up. And if you are running an army of agents, each of those agents will be sped up also. More automation == more output per unit of time.
Second, moving things to code improves efficiency. Your agent could fix an issue every time it sees that issue happen, but that uses tokens and might miss cases. If Claude instead writes a lint rule, CI step, or routine, that class of issue can be fully automated forever. This is really what people are talking about when they talk about loops -- it's about automating entire types of busywork rather than solving them one off. This isn't a new idea at all. Engineers have been doing this for a long time!
Third and most importantly, automation makes it possible for others to contribute to the codebase more easily. Increasingly what I am seeing is engineers are contributing to codebases on day one because Claude can navigate the codebase for them, and that non-engineers are able to contribute to a codebase as effectively as engineers can. What gets in the way of both of these is domain knowledge that lives in peoples' heads rather than in automation -- the stuff you used to have to learn when ramping up. What has changed thanks to agents is the domain knowledge that can be encoded as infrastructure is no longer limited to what is expressible in lint rules and types and tests; it can now capture nearly all domain knowledge, encoded as code comments and skills and CLAUDE.md rules and memories. If I put up a PR for an iOS codebase I don't know and a code reviewer rejects it because it doesn't use the right framework, or if a designer builds a new feature and it gets rejected because it doesn't follow the right architectural patterns, these are failures of automation.
Every team should be writing the CLAUDE.md's, REVIEW.md's, skills, and docs that enable agents to productively work in their codebase with zero additional context from the prompter. This sounds crazy, and at the same time is a natural extension of the stuff engineers have always done: automate, and encode domain knowledge as infrastructure. As the model gets smarter and as the harness matures, this task becomes easier. In the meantime, it is on every team to look for ways to convert their domain knowledge to infra so that Claude can write code better, so that code review catches issues automatically, and so the next person working on your codebase can contribute more easily.
Bobby Gilbert retweeted
And here we go - first *public* experimental evidence of RSI.
As a side note, this post came on my timeline even though I don’t follow the author, so yes, the algorithm is still able to provide me with the content that I am interested in outside of my narrow circles of mutuals.
Bobby Gilbert retweeted
If true, this would be much bigger than just another model release.
Memory efficiency is one of the core bottlenecks for long-context models, agents, and inference economics. A real architecture-level breakthrough here could make longer-horizon AI systems dramatically cheaper and more practical.
Andrew is one of the most reliable sources. Therefore, I'm taking this very seriously. We could truly be at a turning point.
Bobby Gilbert retweeted
I’m excited to share that I’ll be joining OpenAI and look forward to working with the exceptional team there.
It was a difficult decision to move on. I’m incredibly proud of the amazing team at Google and everything we’ve built together. It has been an honor and a pleasure to work with all of you.
Bobby Gilbert retweeted
Introducing the Open Knowledge Format (OKF), an open specification that formalizes the LLM-wiki pattern into a portable, interoperable format.
AI is only as smart as the context we give it. As we build more advanced, agentic AI systems, they need accurate metadata and context to be useful. But in most organizations, that context is locked inside fragmented data catalogs, isolated wikis, scattered code comments, or the minds of senior engineers. Every time a new AI agent is built, teams are forced to solve the exact same context-assembly problem from scratch.
To solve this, we've announced OKF, a vendor-neutral, open specification that formalizes the "LLM-wiki pattern" into a portable, interoperable format. It provides a standardized way to represent the enterprise knowledge that modern AI systems rely on.
— Just markdown: readable in any editor, renderable on GitHub, indexable by any search tool
— Just files: shippable as a tarball, hostable in any git repo, mountable on any filesystem
— Just YAML frontmatter: for the small set of structured fields that need to be queryable: type, title, description, resource, tags, and timestamp
We’ve also shipped reference implementations to help you hit the ground running, including an enrichment agent for BigQuery, a static HTML visualizer, and live sample bundles on @github → goo.gle/4uGvAEe
➕ Knowledge Catalog can now natively ingest OKF!
Stop reinventing data models and building bespoke integrations for every new AI tool. Here's more about how OKF works → goo.gle/4uGvBbg
