
CTO and co-founder @CockroachDB.
New York
Joined October 2011
- Tweets718
- Following124
- Followers2K
- Likes899
There were two hard problems in computer science: cache invalidation and naming things. With agents, only naming remains.
4
12
891
Work that was tedious but not hard is now accessible.
PGO+BOLT cut CPU per operation by 10% in a system I'm working on. AutoFDO on the kernel, on a network-heavy workload, cut CPU by another 10%.
AI will happily do the tedious plumbing.
Mutation Testing (en.wikipedia.org/wiki/Mutati…) appears to be turbo-powered in the AI era. Mutation testing plants bugs at scale and ensures you have testing coverage.
How to design good verification loops?
Well, that’s most of the job now, isn’t it?
Last week, I finally landed a refactor I started in Jan. It’s good I waited b/c the Jan version was buggy af.
This is the verification process I used for refactors (have different ones for perf)
2
13
1,097
Don't tempt fate. If you think your security is rock solid in the age of AI you need to get your hubris in check before the universe does it for you.
Fuck around and find out.
When Railway was breached on 21 Oct, Northflank dunked on them and urged people to migrate over to them because of that breach.
9 days they got hacked, the same way.
Zero sympathy: instead of securing you were busy dunking
10
782
I started programming around 1985. For 40 years, I built software with hand tools. I’m proud of that work, but not complacent.
In the AI era, we have modern construction equipment. Our ambitions need to rise to match.
We built houses. Time to build skyscrapers.
1/3
I see a lot of exciting innovation in the database ecosystem. Rewrites of existing systems in faster languages. Ambitious new projects.
My main reaction is that they aren’t ambitious enough. Most are rebuilding the same house with better tools.
2/3
1
44
2,071
A breach like PingCAP’s is the worst nightmare for everyone involved: customers working out what was exposed, and engineers working around the clock to make it right. I’ve lost many nights’ sleep imagining this scenario. My empathy is with all of them.
Incredible progress from the Blacksmith team. I thought the vision was a bit crazy when I first heard it, but perhaps that is the nature of the best ideas. A number of ex-CRL engineers powering this progress which brings a smile to my face.
The WiscKey paper (usenix.org/system/files/conf…) is the canonical reference for this idea for LSM trees. WiscKey = Wisconsin Key, but the whiskey allusion always brings a smile to my face.
Here's something interesting about LSM trees. Most LSM engines store keys and values side by side, which makes large values a write amplification problem.
Every compaction merges sorted files (SSTables) and rewrites their contents. This is fine when keys are around 16 bytes. But when each key has a 4 KB value, every merge rewrites those bytes again, even when the value itself has not changed.
What if we stored only the keys in the tree and wrote the values to a separate location where they could be appended once and left alone? This approach comes from the WiscKey paper, and TidesDB leverages the same technique.
The core idea is simple: any value at or above a threshold goes into the value log at commit, and the key log stores a small ID in its place. From then on, the WAL record, the memtable entry, and the key log carry only the key and the ID. Compaction moves keys and IDs, never the values themselves.
The performance difference is interesting. In a TidesDB benchmark loading 250K keys with 4 KB values and overwriting them once, separated writes ran almost 3x faster than inline writes. Compaction rewrote no values, while the inline configuration with 64 KB nodes still rewrote 5.1 GB.
We have seen this pattern in relational databases too. For example, Postgres can store large `TEXT` values out of line using TOAST when they no longer fit comfortably in a row. MySQL with InnoDB also takes a similar approach.
The broader lesson is simple - when large values are written frequently but accessed by key, separating them can make a big difference. The right threshold depends on your workload, though.
Hope this helps!
We implemented WiscKey-like value separation in Pebble (github.com/cockroachdb/pebbl…) a while back. Point your favorite coding agent at the repo if you'd like a better understanding of what it took to productionize this idea.
A reply to my B-tree thread: "assumes keys are memcomparable, which is a big restriction." It isn't a restriction on the types. It's an encoding step, and every type with a total order has one. A short thread on turning a key into bytes whose memcmp order is the key's. 🧵 1/8
The rest is bookkeeping. NULL gets a tag byte that sorts before every value. A descending column complements its bytes. Text under a collation uses the collation's sort key, order-preserving by construction. That one costs: the sort key can be several times the string. 7/8
1
2
119
B-trees kept coming up on the Pragmatic Engineer podcast this week. The simple B-tree explanation: each node packs many keys and values together, for better spatial locality than a red-black tree. True. How those keys and values are laid out inside the node matters too. 🧵 1/12
Turning down Google in 2001. @petermattis, CTO and cofounder at Cockroach Labs, on why:
“Google was three years old and I said no.
Here was the calculation in my mind: Google's down in Mountain View. I was living up in San Francisco and I didn't want to do the commute.
So I went and worked at another startup for a year and, after a year, I saw it wasn't going anywhere. Google called me back up and said, "hey, do you want to interview again, but we're not going to be able to offer you the same stock options you got before.” I was like, "Okay."
I still can't remember what they offered me the first time, but I probably would've made a lot more money if I'd taken that first. But I did all right. I'm not complaining. It's one of these things."
The fix is the page prefix. A page is bounded by two separator keys, and every key inside shares their common prefix. Take the 6 bytes past it. Fixed when the page is built, never recomputed. The narrower the page's key range, the longer the prefix and the fewer the ties. 11/12
1
1
356











