co-founder @sandstonehq eng + design | go: https://xitter.cf/t.co/b343l2YUvz @landidofficial
nyc
Joined March 2015
- Tweets510
- Following2.6K
- Followers387
- Likes4.6K
Liam Germain retweeted
SlopCodeBench is the most relevant coding agent benchmark left--the top score is 16% vs. 50-75% on other coding benchmarks like Terminal-Bench, FrontierCode, and DeepSWE
summary of how it works: an agent receives a spec for a small program, like a backup scheduler or a mock HTTP server. the agent implements it, then the agent receives a new spec and has to extend the program. this repeats 3-8 times, and with each new spec there are hidden tests that check the new feature and all the old features.
there are a few reasons why I really like it:
1. it comes far closer to simulating real, long-horizon work with new and updating requirements, which is what it’s actually like to work on a codebase vs. solving a single isolated task
2. code coherence, maintainability, and extendability naturally matter more
3. it’s significantly less saturated than the other coding agent benchmarks
full results in the image (credit to @dexhorthy for the benchmark run + charts), the “core” number measures whether the main tests for the new feature pass; the “strict” number is the headline and measures whether every test passes, including regression tests for all earlier features
astra and sol are at the top (interestingly 6.1 is slightly worse than 6 although likely just noise), but given how low the scores are overall + how small the spread is, i don’t think it’s super meaningful yet in terms of determining which model you should use--opus 5.5 is still my daily driver, and i’m kind of surprised it’s not higher
the big caveat imo is that these runs all used the base harness with no customization. i personally think having a couple of core skills + agents.md instructions (planning, verification, etc) is necessary and improves performance, and that improvement is much more likely to show on SlopCodeBench than the other benchmarks
my prediction is that custom context + skills would improve performance by 5 percentage points or more and (hopefully) result in a ranking that’s a better representation of how good different models are to actually use for engineering work
do you think the improvement from proper skills/verification would be more or less than 5%?
so cool, all this team does is ship ⛽️
as all software teams frantically try and save on CI time and costs in the ai era we have been super fortunate to be working with @useblacksmith since the beginning of the year! super excited to try their next set of features!
@sequoia is leading our Series C at a $1B valuation.
We started Blacksmith in 2024 to make CI faster and less painful. In January we ran 5 million CI jobs a week. Today it's over 90 million and growing, and we’re still early.
ALT Line chart of weekly CI jobs run on Blacksmith, growing from 5.4M in January to 90.5M in October 2026.
Liam Germain retweeted
LLMs are great at software engineering but still quite bad at AI engineering, here's what i'd put in a skill to fix that based on my experience building agents for private equity investors and lawyers:
1. no target leakage. LLMs are laughably bad at this by default; if you ask it to hill climb an agent’s system prompt it’ll put in the exact examples you’re testing against. i’ve found adversarial review works well here
2. show, don’t tell. detailed instructions telling an agent how to do something are less clear / hard to follow and you’ll bitter lesson yourself as the agent finds ways to complete its task that you didn’t think of. when you see undesirable behavior, try to add an (actually generalizable) example that clearly displays what it shouldn’t do
3. keep prompts modular, MECE, and free of contradictions. @wulfie_bain_ has a really good article on this (pinned to his profile) but basically agents just continue adding to prompts and you end up with prompt spaghetti with repetition and contradictions throughout
4. know when to raise. we’ve largely moved past reading the code but i don’t think we’re there yet for prompts; sometimes a quick human read-through of the prompt can save 5-10+ hours worth of autonomous hill-climbing (given the much slower cycle time for agent engineering vs. software engineering). i experienced this firsthand when i built Retrodict, a frontier agent harness for ARC-AGI-3
5. evolve the eval. hill-climbing against an eval that’s subtly incorrect or just measuring the wrong thing can waste a lot of time and make it seem impossible to improve the agent. this isn’t always necessary but evals (especially domain-specific ones) should usually be human-curated and that means they can have mistakes and gaps. you even see this at the highest level with frontier LLM benchmarks changing and improving over time
anything i missed?
Liam Germain retweeted
Rebuilding @sandstonehq's entire frontend with a small army of Claude Code agents and a team of one designer + one engineer. This is how it all works:
· One coordinator keeps the records and runs the loop.
· Builders own a component each and its PR stack.
· Reviewers read every stack from a design point of view and judge it against our Figma specs.
· Thinkers answer Builders’ questions on tokens, icons, specs and content.
· Fixers triage mid-migration bugs and spec the fixes.
All in Claude Desktop for me, though you could manage this in all sorts of ways.
shoutout @c_amataa for getting a prototype working on this without anyone even knowing it was a possibility yet, v2 is going to be even better for us...
Liam Germain retweeted
for the last ~2 months i’ve been using an orchestrator within @get_bb_app as a single entrypoint to manage my agents
with the launch of cursor projects and @nicbstme’s viral post about AI feeds, figured it was time to have opus vibe up a hype video for my workflow
at this point there are a lot of ways to have one agent orchestrate a bunch of other agents to do work, but there are a few things an ideal experience needs to solve for:
observability: you should easily be able to talk directly to any of the spawned agents; especially for me as an fde there’s still a decent amount of in-the-loop work where going through the manager would be an unnecessary layer of translation
attention: there’s only so much i can focus on at once. the split in bb feed makes it easy: agents still working are in a separate section, so the ones that actually need responses are clearly emphasized. if you have too many that need you, you can “pause” some to set them aside (bottom section) for later
interaction: support various levels of interaction with the agents - let the manager handle it, talk to the manager directly and have it talk to the agents, reply to the agent directly without leaving the feed (either based on the summary line or expanding the card to see the full most recent message), or fully click into the agent session
there might be other things out there that have all three of these but i’m a tinkerer at heart and constantly customizing so bb’s plugin system makes this seamless, shoutout @sawyerhood for building a great product
Liam Germain retweeted
legal engineers are one of our biggest assets at @sandstonehq.
they combine legal expertise with engineering capability and can build complex legal features (like redlining and contract drafting) end-to-end, without the usual friction/slowdown that comes with engineer <> lawyer coordination
Looks like everyone is hiring Legal Engineers this week:
Legal Engineer @Microsoft ($278.9K)
Legal Engineer @heyivoai ($320K + equity)
Legal Engineer @Holland_Knight ($245K)
Legal Engineer @EpiqGlobal ($245K)
Legal Engineer @goodwinlaw ($236.5K)
Legal Engineer @harvey ($320K)
Legal Engineer @LexisNexis ($209.7K)
Legal Engineer @WeAreLegora ($325K)
Liam Germain retweeted
It's been so fun to optimize with the new suite of models (Luna, Jev and now Haiku) on low-intelligence tasks against our evals at much higher speed, whilst preserving the quality that we had baked in
Liam Germain retweeted
After six years in stealth, Atomic Machines is out. Our mission: on-demand, universal command of matter. First beachhead: the Matter Compiler, an AI-native manufacturing system that builds working micro-machines from code alone. No per-product tooling. No process development. Different code, different machine.
Liam Germain retweeted
it’s been ~2 months since this banger tweet, I think it’s still super true and it influences how I think about feature development at @sandstonehq
TLDR of the tweet: long-horizon agents are a poor fit for truly novel product work because complete requirements do not exist upfront and present models cannot reliably judge what humans find intuitive, enjoyable, or valuable. they are better at reproducing known forms—like a Call of Duty clone whose requirements have already been absorbed into model weights—than inventing profitable new products alone; novel work needs tight human-agent loops that prototype, test with people, question the approach, and iteratively discover requirements.
especially as an FDE a big part of my job is to talk to the customer and figure out what’s actually needed to solve their pain point; balancing their specific ask with something that’s cohesive with the rest of our product
but for a decent portion of features, once that’s figured out, it is possible to create a complete definition of the requirements upfront
with those requirements, an agent can autonomously plan, build, self-review, and end-to-end verify without my input, only raising things to me when implementation reveals that we should deviate from the requirements for some reason
because of the way AI agents write though, creating that definition in a way that’s easy and reasonable for me to review + iterate on is nontrivial
i’ve been playing with the idea of a “behavior spec”; a Linear document with a set of scenarios (grouped under various headings) that have a descriptive title and three bullets: one for the setup (”before”), one for the action taken, and one describing the state “after” (including a comparison to the current behavior without the feature)
they have to be quite detailed/granular in order to be complete; for a feature i’m currently working on, there are 62 scenarios split across two behavior spec documents. it seems long and daunting, but because the scenarios are written in such a simple way (instructed with a skill) they’re actually quite quick to review, quicker (at least for me) than agent-written documents that are shorter but require more mental leaps / context held in working memory
i leave comments on Linear and iterate with my agent until i agree with the entire document, at which point autonomous implementation begins
note that this on its own isn’t enough for features that have:
- a large amount of net new UI
- complex or ambiguous architecture decisions (things that aren’t clear from following existing codebase conventions + docs)
for the former, aligning first on a figma is super helpful; for the latter a separate document to align on key architecture decisions is usually necessary
opus 5 is a sign that the obsession with “long-horizon agents” in model training is finally backfiring
i don’t like long-horizon agents, and i’ll explain why they fundamentally don’t work
some people will immediately jump out and say “skill issue”. well, show me one profitable business you built with a long-horizon agent working all by itself - i’d love to learn
so far, the only thing they were able to build that’s even interesting enough for people to talk about are those 3d games that are a partial clone of something that already existed
the reason an agent was able to build a working prototype of complex games like call of duty was that a team of humans already figured out all the requirements years ago for how such games should work, what kind of controls are intuitive, what mechanics are fun etc
all those requirements were already absorbed into the model weights, so when you say “build me call of duty” the model already knows the details. its long horizon execution capability can get all the requirements implemented, which i must say is indeed impressive
but now you can see - the value of long horizon execution has a prerequisite of a massive amount of high quality requirements clearly defined upfront. it took a big team of very talented humans months of effort and many iterations to define that for call of duty
now imagine games like call of duty don’t exist yet, how would we use agents to build it for the first time? we can’t say “build me call of duty” any more. and there’s no way we can define months-worth of game design details upfront
we’ll have to build a tiny prototype of the most basic mechanics, play with it, see if it’s fun, then iterate and expand the complexity. even with the smartest humans, that’s how we work towards something great
we don’t need agents to go dark for a long time, spend tens of thousands of dollars worth of tokens, and come back with a product the agent randomly decided to build - try build something truly novel with this and you’ll see it can’t come up with anything that’s actually profitable (i’ll show you why in a bit)
we need a tight feedback loop where we can collaborate with the agent, plan with it, understand what it’s done, question its approach, apply our judgement, give it real world feedback and iteratively arrive at a good outcome
and that’s exactly what opus 5 absolutely suck at. why? i explained it in more depth with my previous post on how RLVR works - RLVR trains the model to generate code that can pass predefined tests in an isolated environment, which is fundamentally incompatible with the idea of having human in the loop. the more we train the models with RLVR to be “long-horizon”, the less they care about talking to humans
ok now - why do they have to talk to humans? why can’t the models iterate and apply judgement by itself?
maybe one day they could, but not today, due to many limitations. two examples -
1. LLMs today can’t “watch a video” yet. they can look through a lot of screenshots, which is extremely inefficient at observing a high fps animated signal. so anything that requires continuous visual attention is something LLMs can’t do very well
2. LLMs don’t truly understand what’s “intuitive” or “pleasant” for humans. they know what’s already proven to be intuitive and pleasant in the past, but if you present a truly novel concept, it can’t predict whether humans will like it accurately
because of those limitations, human judgment is still needed for almost anything valuable. without humans in the loop, agents will only be able to repeat something that already existed, or go in random directions without true understanding of whether it’s building something useful
in summary, long horizon agents assume requirements all exist upfront. they are fundamentally against human in the loop. and they don’t have true judgement for what humans like
that, my friend, is why they don’t work
Liam Germain retweeted
We’re releasing a broad range of new mathematical results produced by an internal frontier model.
We’ve been consulting with the independent Advisory Group on Mathematics and Artificial Intelligence at the Institute for Advanced Study, and we have drawn on their advice and public recommendations to inform how we release these results.
github.com/openai/math
Liam Germain retweeted
Continue to love Open AI’s commitment to the “intelligence too cheap to meter” strain of the AI ecosystem
First @sandstonehq production use case will be shipped tomorrow
shoutout @c_amataa for building the best slack and salesforce experience on the planet
@Sandstonehq brings your legal team's workflows into Slack. intake, triage, drafting, redlining, all in real time. @mention it and it asks the right questions, pulling from your business context.
Tag: @Sandstonehq👇 sforce.co/3TG1DHV
Liam Germain retweeted
We deleted 800k lines of unit tests from the @sazabi monorepo earlier this week.
Three hypotheses:
1- Unit tests "lock-in" slop code by making it hard to change or remove
2- Unit tests slow down dev cycles by making your agent do more work locally and by increasing CI times
3- Unit tests waste money by making your agent generate more tokens and by adding useless CI runs
Will report back soon with the results!
one of the most exciting parts of growing a company is finding individuals that push not only themselves but everyone around them and the impact that @NotRyanB has had in his first couple of weeks is amazing. only the beginning.
inspired by @spakhm for the format + emphasizing the lower friction of tweet vs. blog and @kunchenguid for the inspiration for OPINIONS.md, sharing my process for generating opinions and a selected few of my favorites:
process
- kun chen’s opinions.md agent takes in his tweets / blog posts as input; i don’t tweet or blog nearly enough and half of the time they’re about side projects anyways, so that wouldn’t be a valid source of opinions
- instead i leveraged Readwise, which is what i use to centralize my reading + storing of content from substack, twitter, web blogs, etc.
- i highlight interesting or important snippets on some articles when it feels worth saving for later; the agent runs on Railway on a weekly cron, receives the highlights, my existing opinions, and can read more into each article if needed
- since these aren’t things i wrote myself i may not agree with them, so the agent sends me telegram messages with opinion proposals that i can accept, reject, or reply with feedback. it can revise existing opinions or attach new evidence that supports existing opinions
- this ended up being a super interesting project on agent and eval engineering + evolution; what defines a good opinion? what kinds of mistakes (false positive vs. negatives) do i care about more? how can i make the agent consistent on something inherently qualitative?
opinions
- people remain accountable for AI-generated artifacts they ship and should understand them well enough to defend them under questioning; review culture should give harsh, specific pushback when output crosses into slop.
- AI makes volition more important, not less. the people who benefit most will actively wrestle with it to strengthen their own capabilities and accomplish more, rather than using it merely to avoid effort. education should therefore prioritize the desire to work hard, learn, and put abundant machine knowledge to creative use.
- a vertical AI startup can defend itself against foundation-model labs by obsessing over a narrow set of real customer workflows. continuously turning those workflows into benchmarks and product improvements creates a learning loop the labs are unlikely to match because the vertical is the startup’s whole business, not the lab’s main quest.
- as AI makes products easier to imitate and more superficially similar, specialized product taste becomes more defensible. taste comes from deep understanding of a particular user’s work and compounds across many small choices in the UX, model, defaults, and interaction; individual pieces can be copied more easily than the coherent whole.
these last two are (part of) why i’m so bullish on @sandstonehq
and these four are a small subset of the 84 total opinions as of today; the list keeps growing every week, the full thing is on my website/github
Wild to watch this play out in real time
Design system audits that used to take months now take hours. Took less than a day to set ours up at @sandstonehq—a series of Claude artifacts grounded in codebase, scheduled to rerun and update once a week.