Leo Gorodinski (@eulerfx) header
Leo Gorodinski (@eulerfx) profile picture

Founder @CohesiveSystems, Prev: Co-founder/CTO @AlvysTeam, First engineer → VPE @JetTechnology acq. by @Walmart

Del Mar, CA
Joined May 2008
Photo by Leo Gorodinski (@eulerfx)Photo by Leo Gorodinski (@eulerfx)Photo by Leo Gorodinski (@eulerfx)Photo by Leo Gorodinski (@eulerfx)Photo by Leo Gorodinski (@eulerfx)Photo by Leo Gorodinski (@eulerfx)Photo by Leo Gorodinski (@eulerfx)Photo by Leo Gorodinski (@eulerfx)Photo by Leo Gorodinski (@eulerfx)Photo by Leo Gorodinski (@eulerfx)
Property-based tests generated from the spec and from the code catch different bugs. Use both. In a new paper, across 25 agent-built apps, Spec-Driven PBT found 56 bugs the Code-Driven tests missed, and Code-Driven PBT found 43 that Spec-Driven missed, so 59% of bugs were found by only one approach. Setup - An agent writes invariants in plain English from the requirements doc - Then it reads the code to turn them into property tests - The baseline derives properties from the code alone Why PBT? - Each property-based test kills about 50× as many mutants as the average unit test (prior work cited in paper) - Properties double as documentation, far shorter than the implementation Bugs only one approach found, by type - Spec-Driven: mostly state, lifecycle and logic errors (61% of 56) - Code-Driven: mostly input handling (49% of 43) Limitations - Bugs were judged by the same model that wrote the tests (Claude Opus 5); 25 sampled by hand, all held up - The prompts differed between approaches - The method needs a good spec Insight "Invariants obtained from the specification are not biased by the behaviour of a particular implementation, and can therefore falsify behaviour about which the implementation is internally consistent." My take The code can’t be the only oracle, the spec must be one too. Next: move key properties into types and checks, so they hold by construction. arxiv.org/abs/2611.15920
4
5
3
48
3,049
Leo Gorodinski retweeted
Orleans is a classic paper worth revisiting, from before most modern durable execution frameworks. I think virtual actors/durable objects are in some sense the object-oriented cousins of durable workflows. Both make applications reliable, but they focus on different abstractions: durable state and identity vs. durable execution progress.
3
5
62
2,798
Leo Gorodinski retweeted
Now I'm tackling something more ambitious: formalizing XState's runtime behavior with LemmaScript. I started with XState's lightweight FSM, proving 31 laws governing state transitions, context updates and sequences of events. Comparing the formally verified model against XState's implementation helped uncover two bugs, both since fixed upstream. That led me deeper into XState's runtime. In my first week, I filed 7 bug reports involving subtle problems with transitions, event ordering and state behavior. It's exciting to see formal verification help uncover bugs in a mature, widely used open-source library.
2
3
1
9
1,476
We keep reinventing how software executes and recovers, but still lack a model of how systems compose and what must stay true across boundaries. Why I'd separate domain semantics, composition, and infrastructure.
6
6
1
61
3,299
Frontier agents can build software, but they struggle to verify it. That’s the conclusion of TestSuiteBench from ETH Zurich, a new benchmark evaluating how well coding agents, given only a prose spec, generate tests that distinguish correct behavior from incorrect behavior. Results - Writing tests from a spec (Opus 5): 96.1% are valid, but they catch only 55.5% of injected faults (mutants). - Writing the program from the same spec (Opus 5): its code passes 91.8% of the reference tests. - Giving an agent (GPT 5.6 Sol) the reference executable doesn’t help. The bottleneck is test design. Setup - Start with 165 real applications (C++, Rust, Go) and build on ProgramBench’s reference tests, synthesized by agents from existing repository tests and executable behavior. - Generate prose specs from those tests, then iteratively refine them to remove ambiguity and contradictions. - Generate requirement-targeted mutants: agents deliberately introduce faults into the reference implementations, then refine them to minimize unrelated test failures. - Ask agents to generate new tests from the specs, without seeing the reference implementation. Measure whether they pass on the reference (validity) and catch the mutants (mutation score). Limitations - Mutants are synthetic, designed to violate behaviors covered by reference tests. They aren’t necessarily representative of real-world bugs. - The specs are unusually favorable: derived from known tested behavior and repeatedly refined. Using existing repository documentation instead causes mutation scores to drop dramatically. - The benchmark doesn’t directly evaluate agents testing their own generated implementations, and it covers only CLI applications. My take A passing test says little about whether the test itself is good. Implementation has a feedback loop: run the tests and fix what fails. Test generation lacks an equivalent signal of adequacy by default. Unprompted, Opus attempted mutation testing in 33 of 165 applications, potentially giving itself that missing signal. Agents need to test their tests, not just their code. arxiv.org/abs/2610.13065
9
2
2
30
1,371
San Diego North County sunsets – at Del Mar, CA
1
11
528
Sometimes reading the code is the fastest way to comprehend. Not prose, not architecture diagrams, but code. Can we rely solely on extensional inquiry, judging code only by what it does?
Uncle Bob says he stopped reading the code his agents write. Whether a human should still read the output is the most interesting question in software right now.
3
484
20 pictures of my dog (and my twin) – at Del Mar, CA
1
6
171
Great essay. “Nobody trusts your agent” says what most of us sense. While we were growing Alvys, customers wanted to look us in the eye: trust traces back to a person with something to lose, even with corporations. The big question is how far skin in the game can scale: How much agent work can one person take responsibility for? Does accountability add up like capital, or do many small stakes beat one big one? What backs accountability if agents sell at cost?
4
1
16
1,046
Interesting work from Jane Street on building a high-performance parallel programming DSL in OxCaml. Instead of writing hardware-specific SIMD instructions, developers express computations using the SPMD (Single Program, Multiple Data) model familiar from CUDA and other parallel programming systems. The compiler: - Builds typed computation DAGs, distinguishing uniform and per-lane values. - Lowers them into an A-Normal Form (ANF) IR. - Optimizes through hoisting, loop unrolling, and vectorization. - Uses OxCaml metaprogramming to JIT-compile specialized native code. At Cohesive, we're pursuing a similar approach with an embedded C# DSL and metaprogramming: fusing declarative definitions of relationships, state transitions, and workflows into efficient executable code. Different domain, same principle: make semantics explicit so the compiler can optimize across abstraction boundaries rather than execute each declaration independently.
1
1
217
An AI agent “fixed” a Kubernetes incident by accepting an invalid input that the system was supposed to reject. SWEOps-Bench tests agents on 55 historical bugs across 16 Kubernetes operators: - Detect the incident - Investigate the running system - Patch the code - Verify recovery after deployment. Across six agent/model configurations: - Incident detection: 73–91% - Verified recovery: 15–53% The interesting part is how they evaluate “fixed.” Clearing the fault isn’t enough. The repair must also pass checks for the affected functionality and regressions. Diagnosis and recovery only partly coincide. In 15–22% of tasks, agents recovered despite an incorrect diagnosis. More tokens didn’t reliably help. The authors recommend an explicit recovery contract: define what behavior must be restored, independently of the proposed patch. Limitations: - Size: 55 tasks, one run per task and configuration, so each task moves a score about 2 points. Some categories have only 3–5 tasks. - Task type: software defects in Kubernetes operators, not infrastructure outages or other platforms. - Exclusions: nondeterministic races and silent state drift. It turns out “make the error go away” doesn't quite cut it. arxiv.org/abs/2611.06099
1
1
4
237
Leo Gorodinski retweeted
Question to people who use AI to write, review and merge large chunks of their code base: My fear is that this leads to a system too complex to reason about, and therefore we'll end up with production issues that we cannot fix. Can you share how are things looking for you? did complexity explode, or is it well contained? how's troubleshooting production going?
154
5
5
172
26,400
I’ve been practicing DDD for many years, so this is good to hear. But I think we need to say more about what DDD means in the agentic era. Beyond telling agents to “do DDD,” we can use them to: - Formalize the model - Benchmark what works Formalization Today we guide agents mostly through skills and .md files. That’s a natural instinct, since agents follow prose instructions well in the small. But benchmarks call this into question at scale: adherence is inconsistent, especially for architectural rules. Formalization moves the domain’s rules out of prose and into structure that can be mechanically checked. Rigor that used to take too much manual effort is now within reach. What I’d try: - Define types, interfaces, and module structure first, and make "illegal states unrepresentable". - Route every change to aggregate state through operations that enforce its invariants. - Model aggregate lifecycles as explicit state machines. - Model cross-aggregate interactions as explicit, formal workflows and protocols, with sequencing, retries, and compensation spelled out. - Enforce bounded contexts as module boundaries. - Write property-based tests for domain invariants. - Apply formal methods where justified, such as model checking a protocol. Benchmarking Many claims about DDD’s benefits remain subjective. There’s now a growing set of published agent benchmarks for code, and building your own is cheaper. That evidence matters more now than it ever has. Without it, we’re producing far more code, far faster, without knowing whether we’re building better systems.
One odd quirk of AI is that if you can describe your business processes in clear, consistent language... ...you've already half-built all the software you need to automate the process DDD has never been more powerful
7
9
1
219
14,438
Coding agents follow about half the rules in your AGENTS.md every single time, and ignore more than a third of them completely. A Nov 2026 study "Do They Do as They Are Told?" checked whether agents obey the rules in real AGENTS.md (and similar): 621 context files from 268 GitHub repos, against ~6.5k commits by Copilot, Claude Code and Codex. Findings - At most 44% of the rules people write can be checked against the code at all. The rest are vague advice or process steps that leave no trace. - Compliance is bimodal: of the ~2k rules they could test, 54% were followed in every relevant commit, 37% in none, and only 8% fell in between. Be careful with spot-checking! - It depends on the rule: coding, build and documentation rules average 60–70%. Testing rules average 49%. The median architecture or process rule is never followed (samples are small: 40 and 50 rules). Limitations - Only 14% of the ~15k rules written could be tested against a commit (so the numbers describe a small slice). - The study is observational, and the typical rule was checked against only three commits. - A manual review found 71% of the automated checks correct, and the checks are lenient, so real compliance is probably lower. The authors' advice: write one concrete, imperative, checkable rule per line, and drop qualifiers like "when appropriate." My take: architecture shouldn't be a request in a .md file. It should be structure in the code that the compiler and tests enforce.
6
1
21
649
In zero-shot testing, every model failed all 30 from-scratch authoring tasks in AgentScript-Eval, a new benchmark for Salesforce’s DSL for building agents. No model compiled even half its programs. With a compile-and-edit loop and better skills, the best setup compiled 98% of programs across all 100 tasks, yet only 37% passed all behavioral checks. From-scratch authoring stayed under 17%. Failure modes - Fields that read like English but alter behavior. - Action outputs bound to variables by position, which can be swapped without breaking compilation. Compile errors get repaired because the compiler reports them. Behavioral errors persist because the rules they break live in prose, not in the compiler. My take: we need languages where behavioral requirements are explicit and mechanically checkable, rather than relying on agents to implement them correctly.
7
363
DeepMind just verified Signal's security properties on the real compiled code, not just an abstract model of the protocol. AI agents wrote 6.5M lines of machine-checked Lean proofs, against just ~26K lines of trusted specs and semantics. Contributions - Proofs of: forward secrecy, post-compromise security, handshake secrecy, and authentication for one-to-one sessions. - The proofs are about the binary, so the Rust source and the compiler drop out of the trusted base (what must be taken on faith). - Spec validation: agents inject a bug into the spec, then prove the security theorem fails. - Security reductions: machine-checked proofs that breaking the protocol (classically) means breaking an underlying cryptographic hardness assumption. Limits - The spec is trusted: agent-drafted, human-approved, and stress-tested, but nothing proves it captures the right intent. - The cryptographic primitives are assumed, not verified. - Post-quantum coverage is thin. - No concurrency (a hard part of real systems), and no group messaging yet. My take: when a system's intended behavior is well understood, agents can now write the proofs at scale. The bottleneck moves from proving to specifying the right system (and to keeping the 6.5M lines of proof in sync when the code changes). google-deepmind.github.io/co…
4
1
1
9
1,685
When agents write the code, the spec is the product. Microsoft's PDD Kit proves agent-written code against a Verus spec, but has humans review plain-language intent instead. Two hard problems remain. Validation (did we specify the right thing): - Stating properties strong enough that agents can't satisfy them trivially - Keeping specs readable, or spec review is just code review again Verification (does the code match the spec): - Proof after the fact: find abstractions that make proofs tractable, layered by refinement - Or by construction: derive the code from the spec, so alignment is a compilation problem Spec review should be replacing code review. But we're not quite there yet, because spec languages were built for proving, not for reading.
How do you solve the trust problem with AI-generated code? AI agents can generate code at a scale too great for humans to review line-by-line. That makes it difficult to ensure the code will do what the developer intended, and it leaves the final output vulnerable to bugs and security risks. Proof-Driven Development Kit turns plain-language developer intent into a specification in Verus, applying formal methods traditionally used for safety-critical systems. PDD Kit uses mathematical proofs to test agent-generated code against the spec where possible, and clearly indicates where the proofs can’t reach. The result is more reliable agentic code. Researchers @MatthaiP39719 and @alexlavaee break down Proof-Driven Development in the MSR blog: msft.it/6019af2V7
3
3
1
21
1,318
AI app builders are learning they need more than code generation. Lovable's recent acquisitions: - Sutro (September): a language for explicitly modeling app behavior and rules - Radish (this week): infrastructure for running agents reliably Acquihires or not, generation, execution, and modeling are converging. For complex, mission-critical apps, the system model, not the generated code, has to become the primary artifact.
4
266
Leo Gorodinski retweeted
What is an implicit state machine? Your apps are full of state machines, which is the logic that says "when we're in this state, and this event happens, now we're in this state" (given, when, then). Sometimes those state machines are clearly mapped out (switch statements, reducers, tables/maps, libraries like XState for the more complex ones, etc.) and the logic is explicitly enumerable. And sometimes it's not. The ad-hoc logic is spread all over the place. That's how developers typically code, and that's how agents typically vibe-code (they learned from you). Those are implicit state machines. What's wrong with implicit state machines? It's not clear what state transitions are possible, so you might have "allowed" states that should have been impossible without knowing it. Agents make the same mistakes. You can get all tests passing and your code review agents (even the well-funded ones) can say "LGTM" and you can *still* have very bad bugs because an implicit state machine allowed unexpected or illegal behavior. (Remember the Apple FaceTime bug?) So yeah, making your app logic explicit is important.
56
85
20
1,345
122,042
Leo Gorodinski retweeted
Deciding how to help non-expert humans engage with formal verification specifications seems like one of the more important non-verifiable UX problems facing the rise of formal verification. 🧵
27
34
9
364
49,411