Backend/infra engineer turned founder @hatchet_dev | Databases, Go, distributed systems, durable execution, and open source
Joined May 2025
- Tweets397
- Following671
- Followers993
- Likes343
Pinned Tweet
we're open sourcing a library we’ve been working on at Hatchet called pgoutbox. github.com/hatchet-dev/pgout…
it implements a high-performance transactional outbox (specifically, for Go applications that use postgres via the pgx driver, but we'd like to make it available in other languages soon).
we’ve been running it in production for the past 3 months at Hatchet, processing tens of thousands of messages per second. benchmarks on my local laptop have it going > 200k messages / second.
what's a transactional outbox? it's a pattern for solving a lost-write problem in distributed systems. imagine a user signs up for your app and you’d like to create a new row in your users table and send them a welcome email. if your API crashes or restarts after you commit to the users table, but before sending a welcome email, the write to the email service is lost forever. so instead, you write the message to an outbox table as part of the same tx as the user write.
obviously there’s a lot of intersection with task queues built on Postgres — transactional enqueuing is one of the main benefits! we had a few requirements that make typical task queue libraries a poor fit.
first, it needs to support hundreds of thousands of different topics with exclusive consumers. this can get a bit tricky to scale due to MVCC on Postgres; if you attach a lease with a heartbeat to each topic, you’ll perform wayy too many updates and accumulate lots of dead tuples.
second, it needs to support immediate message deletion with configurable retention policies per topic. this is quite different from a durable queue, where you’d like to preserve messages for future replays and auditing.
third, it needed a solid scaling story and APIs built around batching.
here for feedback / questions!
after a week of security incidents in the software world, we thought it'd be a good time to release something we've been working on for a little bit: FIPS-compliant hatchet images!
these basically use Tink / Go crypto primitives under the hood but disable certain features which aren't FIPS-compliant.
(it turns out bcrypt isn't FIPS-compliant, so we had to change how passwords get hashed for open source users when running the -fips build)
anyway, this removes a bunch of friction for self-hosting hatchet in really sensitive environments, which we think is a big win.
it's been a week. to end it on a high note, we're releasing FIPS-compliant images for everyone (not gated behind some enterprise sales maze).
you're either in one of two camps: you've never heard of FIPS, or you deal with it every day. this is for the second camp.
docs: docs.hatchet.run/self-hostin…
for everyone piling on to this post after the railway incident: remember that heroku had a roughly equivalent security incident in april 2022, pre-agentic coding, where a number of user environments were exfiltrated (github repos and oauth tokens, in this case, not prod runtimes, but similar impact). I don't think there's much you can extrapolate from this post unless there's a very direct RCA which says otherwise and pins the cause directly on agentic coding habits.
these platforms are fundamentally honeypots and are going to be targeted, and securing untrusted user code from deployment -> production is brutal.
these incidents obviously shouldn't happen, but it has more to do with the practices / priorities of the team than whether or not they use AI to code.
some more thoughts on this "human coherence" take since yesterday:
- agentic review doesn't solve this coherence/elegance problem, because review evaluates whether the presented diff is free of bugs, and rarely evaluates whether it's a globally optimal strategy. and if you do rely on agentic review to decide whether this is the optimal approach, you're going to waste a ton of tokens going back and forth across agents, which is just token churn handing money to the labs. this decision needs to happen earlier in the dev process.
- forcing developers to write their own PR descriptions is a great habit to ensure coherence. I've found myself opening PRs with the assistance of Claude Code where I can't distill _exactly_ what's important about the PR in my description, which forces me to go back to the code again.
- there are cases where it makes sense to trade off coherence in favor of velocity, it's ultimately a judgement call. the way that we think about it is that great software is produced by consistent, high-quality, high-velocity effort. the risk that you run with reduction of coherence is that the cost of fixing each bug or adding a new features grows exponentially with the complexity of your code (or, not even exponentially, any type of increase which is greater than the reduction in token cost is problematic). there's also the argument that "future models will get better," where you're implicitly making the bet that the rate of increase in model performance is greater than the rate of complexity in your codebase, which is not something I'd be comfortable betting on.
- our tiny human brains are really, really good at sensing that complexity is getting out of control when things become ugly, inelegant, or incoherent, and LLMs do not have this innate ability. they might in the future!
I'm an avid user of coding agents, but there's something I've been struggling to put into words which I think is encapsulated well by these screenshots. these are two PRs to fix the same bug: the first is a +627 / -291 line change, from Claude, the second is a +3 / -2 line change, from Matt, a human (we closed the former and merged the latter).
there's a certain coherence in a human's mental model of the codebase which makes some bugs trivial for humans and still really difficult (or token-expensive) for agents. this goes beyond just better memory of the codebase. humans will build a mental map and a probability matrix for which parts of the codebase are most likely to surface a particular bug, given the codebase's coherence, and start by tackling those before going another layer deeper.
without matt's contribution we might well have merged the +627 / -291 PR, in turn losing coherence, and becoming more token-dependent as a result. after you lose coherence it's game over.
i've found myself pair programming with agents on tiny PRs after a huge amount of independent research / agent swarms to tackle a given problem, and I think this is pretty effective; you can combine the things that agents are naturally good at (parallelization, verifiable problems) with your human coherence. often this results in 15-20 PRs, not all of which get merged.
I'm an avid user of coding agents, but there's something I've been struggling to put into words which I think is encapsulated well by these screenshots. these are two PRs to fix the same bug: the first is a +627 / -291 line change, from Claude, the second is a +3 / -2 line change, from Matt, a human (we closed the former and merged the latter).
there's a certain coherence in a human's mental model of the codebase which makes some bugs trivial for humans and still really difficult (or token-expensive) for agents. this goes beyond just better memory of the codebase. humans will build a mental map and a probability matrix for which parts of the codebase are most likely to surface a particular bug, given the codebase's coherence, and start by tackling those before going another layer deeper.
without matt's contribution we might well have merged the +627 / -291 PR, in turn losing coherence, and becoming more token-dependent as a result. after you lose coherence it's game over.
i've found myself pair programming with agents on tiny PRs after a huge amount of independent research / agent swarms to tackle a given problem, and I think this is pretty effective; you can combine the things that agents are naturally good at (parallelization, verifiable problems) with your human coherence. often this results in 15-20 PRs, not all of which get merged.
Our European takeover continues. We're so excited to have JB on the @hatchet_dev team!
please join us in welcoming jean-baptiste souvestre to the team! based in paris (ooh-lala), JB joins our growing engineering squad in europe. 🥖🇪🇺🍷
we asked him some questions to get to know him better, here's what he said:
➡️ light mode or dark mode?
🎤 what is a light mode?
➡️ we're told you're a cat man. tell us about your cats.
🎤 so i have 3 cats, kiki, chablis, and croche-pattes, all of whom are scoundrels in one way or the other. they love wet food, waking up at 4pm, and going to bed at 5.
➡️ how did you first come across hatchet?
🎤 i wanted to introduce hatchet to my last company because i liked the tech that much, but i figured it might just be more fun to work on it directly.
➡️ if i were in paris for one (1) day and wanted to get the best croissant, where would you suggest i go?
🎤 that's mamiche, in the 10th arrondissement, for sure.
➡️ what's something that surprised you in your first week here?
🎤 honestly can't decide between the tech maturity at such a young startup and the amount of custom frog emojis one can add to slack.
➡️ last, but not least: claude code or codex?
🎤 claude, but just because it was a french first name.
bienvenue à bord, JB! 🚀
Alexander Belanger retweeted
back in february, when we first released the @hatchet_dev CLI, we were pretty adamant that it was the CLI, and not an MCP, that would be the primary entrypoint for agents using Hatchet. and this has definitely been the case! but the rise of coding agents and the latest MCP spec are making MCPs much more usable across a wider range of services by supporting serverless request/response. and, frankly, we've been using MCP servers much more -- shoutout to @Posthog and @Figma.
so we're releasing a new and updated MCP which does everything you'd expect -- debugging runs, triggering workflows and runs, and inspecting configuration of workflows and tasks -- so you can use either the CLI or the MCP (or the regular SDKs) to accomplish what you need.
back in february, when we first released the @hatchet_dev CLI, we were pretty adamant that it was the CLI, and not an MCP, that would be the primary entrypoint for agents using Hatchet. and this has definitely been the case! but the rise of coding agents and the latest MCP spec are making MCPs much more usable across a wider range of services by supporting serverless request/response. and, frankly, we've been using MCP servers much more -- shoutout to @Posthog and @Figma.
so we're releasing a new and updated MCP which does everything you'd expect -- debugging runs, triggering workflows and runs, and inspecting configuration of workflows and tasks -- so you can use either the CLI or the MCP (or the regular SDKs) to accomplish what you need.
new: hatchet mcp
trigger, inspect, and replay runs from your coding agent
docs: docs.hatchet.run/reference/c…
Alexander Belanger retweeted
me, just now: hmm i wonder what accounts the @hatchet_dev account follows
the @hatchet_dev account:
happy friday, everyone! our low level systems person @vqtbl4q has been permitted to take over our socials for the day, i think this was a great idea
I think this is really exciting, but we've started just putting "durable" in front or behind everything without really having a precise definition. I think lots of people are confused about durable objects versus durable execution vs durable queues and statefulness and everything in between. here's an attempt to demystify things...
the first thing to understand is that AI agents are stateful by nature. when you start taking apart a coding agent like Pi or Claude Code, you’ll notice at least 3 aspects of state:
- filesystem/scratch space: Claude Code operates on files. It’s goal is to edit the filesystem to produce a diff.
- context: the sequence of tokens which have been fed to the LLM
- session: the history of individual tool calls along with session state (for example, whether the session is paused or resumed)
this shouldn't come as a huge surprise to anyone. when you run a coding agent locally, these three bits of state are also local and live on the local filesystem!
folks who are building agentic products aren’t usually building local agents. they're building agents which run as part of a software service, hosted somewhere on the cloud. even the most popular local coding agents also have a cloud agents or background agents product.
cloud agents need to run on a machine somewhere in the cloud (duh). but machines in the cloud aren’t like local machines. they sit behind many, many layers of virtualization, and the past 15 years of DevOps have taught us to treat our cloud machines like cattle, not pets. so we expect to run a fleet of machines, which should be resilient to random failures or interrupts and all manner of distributed systems problems.
Pi Durable is referencing this problem: "we need a harness that runs anywhere, can be reached from different surfaces, supports infinitely long conversations, survives catastrophic internal and external failures, and lets multiple humans steer the same agents"
this all comes down to agent statefulness.
0/
People of Pi: We've shipped Pi 1.0 with Pi Durable. Go make them yours.
earendil.com/posts/pi-1-0/
a certain class of agents are more effective when they can write to disk. coding agents are a big one, but we're seeing this more and more with general-purpose agents, too. I think of the process of writing to disk as building the "inner harness."
we'd like to preserve the state of this disk while we’re waiting for feedback, or our sessions are paused (unless your agent is able to one-shot every command, which is highly unlikely). suddenly, we need to think about how to get the state of the filesystem on one machine moved to a different machine.
remember when I mentioned that agents are like managing thousands of little databases? the natural extension to this analogy is to create incremental backups of disk as the agent makes progress.
but most filesystems don’t work this way. you'd need to mount a filesystem on each machine which is running an agent. many container runtimes barely support mounting the same disk in two places, let alone mounting many disks across many places. the disks also need to be virtualized, which can impact performance quite a bit.
the other naive solution is just to snapshot and archive the current state of the filesystem after each agent turn. There are two ways of doing this:
- Just use git. Each session turn is responsible for producing a commit, and resuming a session is just a git clone away. This breaks down for anything that’s not a coding agent, and becomes extremely slow and expensive (ever been hit with an AWS data transfer cost bill?) when running at scale.
- Just run tar and upload the archive to S3. Rolling this yourself is even slower than the git approach.
if you’re just getting started, and trying to get your MVP to production, I actually don’t think these are unreasonable approaches. they just don’t work at scale. it turns out that we need rather specialized tools to even begin solving this problem.
I’m particularly excited about how companies like @archildata are tackling this problem. they mount an “infinite,” S3-backed filesystem with strong RW consistency guarantees. but instead of each write going directly to S3, they put a bunch of fast disks close to your agents to make performance tractable. 3/
so, there's always an "outer harness" or "session manager," which is its own agent with its own statefulness requirements. in this bucket, you see durable workflows, durable objects, and Pi Durable (which can plug into anything).
and there's an "inner harness," which also has its own statefulness requirements. in this bucket, you see filesystems, sandbox drives / snapshotting, and S3 tarballs. and Pi Durable as well.
running our concurrency + fairness strategies at scale was one of the things costing us a ton of $$$, which we rewrote from scratch a few months ago. we're currently on the 4th iteration of this implementation and it's holding up well so far.
the basic idea is to move some of the load off the database and use a map-of-heaps data structure to implement fair queuing instead. for some context, fair queuing allows you to round-robin across a set of "keys" which each represent their own sub-queue.
we used to implement this purely on the Postgres level by basically doing a lateral join across each sub-queue to figure out which items to dequeue. but it doesn't work at scale, because it's almost guaranteed to seq scan at some point and eat up CPU (lots of tricks to avoiding the seq scan, but we couldn't get rid of all pathological cases).
so we built our own data structure instead for the index and moved most of the workload in-memory. the most difficult part is maintaining consistency with the underlying database, which we accomplish using a rollback log and a transactional outbox.
this has cut down massively on CPU across our instances and is one of the core reasons behind the pricing changes we announced.
for those curious, the underlying heap data structure is here: github.com/hatchet-dev/hatch…
really excited for this one. this is the result of ~2 years of obsessing about one thing: performance.
building a general-purpose workflow engine is a double-edged sword; it's great for developers and coding agents because you can use it to solve a bunch of different problems, but at the same time, users might run tasks using a mixture of features you've never seen before.
you see this all the time with general-purpose databases, btw, like postgres -- super fast in most cases but if you use just the wrong combination of features it gets really slow and non-performant.
so there were a few workloads on hatchet cloud which cost us an insane amount of $$$ to run. deeply negative margins. but rather than raise prices for these users, we basically just doubled down, then doubled down again, on improving performance.
we've actually kept our infra spend pretty much flat for ~14 months, despite increasing revenue over 4x in that time period. better performance means less load on the databases, smaller spikes, and fewer machines. which leads directly to savings for users.
really excited for this one. this is the result of ~2 years of obsessing about one thing: performance.
building a general-purpose workflow engine is a double-edged sword; it's great for developers and coding agents because you can use it to solve a bunch of different problems, but at the same time, users might run tasks using a mixture of features you've never seen before.
you see this all the time with general-purpose databases, btw, like postgres -- super fast in most cases but if you use just the wrong combination of features it gets really slow and non-performant.
so there were a few workloads on hatchet cloud which cost us an insane amount of $$$ to run. deeply negative margins. but rather than raise prices for these users, we basically just doubled down, then doubled down again, on improving performance.
we've actually kept our infra spend pretty much flat for ~14 months, despite increasing revenue over 4x in that time period. better performance means less load on the databases, smaller spikes, and fewer machines. which leads directly to savings for users.
hatchet just crossed 8k github stars!
lots of companies are pulling back from open source recently while we're doing the opposite. lots of people ask us about this: when's the rug pull?! how do you make money??
the simple answer is that we make money on hatchet cloud. we're betting on our infra chops and team's ability to run the most reliable service for async tasks in the world. in other words, the goal is to make the cloud platform performant, reliable, and cost effective at scale. so far this model has worked incredibly well for us.
but there are still tons of great reasons to self host. feedback from users who are self-hosting makes our product much, much better. and we also care deeply about open source in the first place - hence the MIT license.
