Sam Reis (@periping) header
Sam Reis (@periping) profile picture

Building programmable, open source git infrastructure at https://teapawt.net/t.co/xqsS28cHRi

Berlin
Joined July 2010
Photo by Sam Reis (@periping)Photo by Sam Reis (@periping)Photo by Sam Reis (@periping)Photo by Sam Reis (@periping)Photo by Sam Reis (@periping)Photo by Sam Reis (@periping)Photo by Sam Reis (@periping)Photo by Sam Reis (@periping)Photo by Sam Reis (@periping)Photo by Sam Reis (@periping)
We just built the most scalable Git server out there: 7,600+ concurrent pushes/s per repo at millisecond latency. Powered by Raft and S3, designed like a modern database. Today we're making it open source by releasing Enroute 0.3.0. Read the write-up: enroute.sh/blog/everyone-is-…
36
54
9
779
78,261
Have been hard at work figuring out how to make replicated state machines both simple and scalable. The result allows a single Enroute cluster to scale up to 300,000 repos, with more to come. Check out our write up! enroute.sh/blog/scaling-hori…
5
7
1
97
6,230
Just released @enroute_sh 0.4.0, and it now scales horizontally 🚀 We've been hard at work building on our new storage engine to make Enroute scale. As of this release, it supports 300k active repos on a single cluster, at 20k pushes/s. More to come! github.com/enroute-sh/enrout…
5
4
38
1,890
Previously we used Multi-Raft with a Raft group per repository. At scale this breaks down due to all the overhead. The solution is a Vertical Paxos-inspired approach where a single root log appoints and monitors log leaders rather than electing them.
1
6
142
Also has the cool side-effect that this root log can begin to act as a placement driver. We use this today for ensuring replicas spread failure domains. Very excited about this direction, and how much untapped potential there still is. Blog post on all of this is coming soon!
2
67
Pet peeve alert. People have to stop prompting blanket reliability or performance “advice”. Instrument, simulate, learn, fix. Let your agent drive the process. Also if you think adding retries everywhere improves stability, maybe stop believing you know better than your llm.
stabilitymaxxing prompt: > find every outbound call (HTTP, DB, queues, LLM APIs) with no timeout, or retries without backoff. add sane timeouts and capped exponential retries with jitter, check pool sizes match our concurrency
3
1
6
2,152
Magitcians ✨🧙
The new SCM (Source Control Mafia)
1
10
504
Sam Reis retweeted
S3-based database architectures don’t read from S3 all the time, though. Instead, they amortize the cost by caching in memory and on local disk (which can be NVMe). As a result, read performance is more or less the same as in local-disk architectures, though the tail latency is heavier; NVMe’s tail is much flatter. Writes are slower, but not by as much as people often think. You almost never want a write acknowledged by a single local disk, so a local-disk database pays a network round trip for replication anyway. That makes S3-based architectures competitive, especially with S3 Express.
TL;DR your database should be on local NVMe
25
22
6
384
44,267
Super interesting read. Love that they are doubling down on Git as well. Already going through in finer detail to see what we can learn from this to make @enroute_sh even better.
We built Pando, a new Git backend for our monorepo, to handle more pushes and fetches without changing how our engineers and agents use Git. Pando cut push latency by ~80% and now serves nearly a petabyte of Git data a day. We can grow Git capacity alongside agent usage.
1
1
11
646
Sam Reis retweeted
First @AnthropicAI, now @OpenAIDevs. Like OpenAI, we believe Git is still the right tool for the job. Very soon, many more teams will hit source control scaling limits. Code is growing exponentially, and Git needs new infrastructure to keep up with agents. If you’d rather not build your own VCS, check out @enroute_sh for fast, programmable Git infrastructure.
We built Pando, a new Git backend for our monorepo, to handle more pushes and fetches without changing how our engineers and agents use Git. Pando cut push latency by ~80% and now serves nearly a petabyte of Git data a day. We can grow Git capacity alongside agent usage.
1
1
6
703
Have been working recently on making @enroute_sh scale better for large numbers of repositories. Part of that is a new per-repo consensus impl that’s inspired by Vertical Paxos. Anyway, seems like this improves single repo throughout and latency as well? 🤔
1
1
10
449
Sam Reis retweeted
This kind of feedback makes my day! ♥️ Super cool to see people building their own code forges on top of @enroute_sh. We’ve also built our own forge for dogfooding Enroute, so it works exactly the way we want. The UI is super clean, and instead of PRs we have a simple breakdown and summary of everything your agent changed. Fun fact: It runs on a single @flydotio machine and is still the fastest thing out there.
2
2
12
405
Congrats @tangled_org on the launch! 🚀
We spent the last few months building the first ever truly federated code forge. Tangled is self-hostable, hackable & extensible. The future of software engineering is a moving target -- our bet on open protocols gives us the agility to move with it. tangled.org/blog/forge
5
239
Sam Reis retweeted
Nice comeback from @github after the (re-) launch of @GoEntire this week. Seems like 7k+ pushes/s is how you build Git infra these days. Go check out @enroute_sh if you want to build on open-source git infra with that kind of performance already today!
1
1
16
981
Welcome @github to the 7000+ pushes/s club.
2
3
9
646
Just released Sirocco 0.3.0: no longer just a decompressor, it now also supports fast, no-unsafe streaming DEFLATE compression. In fact, it surpasses almost all existing compressors on the Pareto frontier on my M4. The final boss libdeflate is still better in some cases. WIP.
Whenever you‘re dealing with Git objects, (de)compression shows up big time in cpu profiles. I just open sourced sirocco (github.com/srijs/sirocco) which for avg git pack parsing is now the fastest streaming inflate lib that I know of. Ships in our next release.
1
3
23
792
Update: Now at least on-par with deflate everywhere.
26
Fast writes to disk quorum with delayed S3 persistence is where it's at. If you want the same for your Git hosting, check out @enroute_sh.
Rivet 3.0 is built on S3 But we did something completely different... And oohhhh boy: - 6.9 ms actor cold starts (durable) - 0.69 ms SQLite writes (3 availability zone) - 0.20 ms SQLite reads Here's how we built our new storage engine, WarpDrive. ~~~ Most S3-backed database read & write straight to S3. That means: - Every commit & cache miss is an S3 request ($$$) - S3 downtime takes you down (SLA allows 8 hours downtime/yr!) - S3 Express is limited to one availability zone This usually ends up slow, expensive, and has poor tail latencies. For Rivet 3.0, we decided we could do better. We wanted 3.0 to ship with: - SSD speeds with storage on S3 - Stateless nodes - No dependencies ~~~ To do that, we built WarpDrive to durably write your data on fast disks *before* we write it to S3. Every write on WarpDrive is: 1. Written to disk in 3 availability zones 2. Acked once a majority has them 3. Then batched to S3 in the background That means that Rivet has: - Writes that get durably acknowledged insanely fast - Fewer S3 operations by batching in the background - Survives S3 outages for new & warmed actors - Nodes remain stateless - S3 is the only dependency ~~~ WarpDrive is something we're incredibly proud of and can't wait to have you try for yourself. Rivet 3.0 with WarpDrive is in preview today. It's available on Rivet Cloud or for self-hosting today. Please send feedback on X, Discord, or DMs -- we want to hear anything and everything you have to say (please bump me if I missed a message!) There should be no reason to vendor lock yourself to Cloudflare Durable Objects if you can have better performance, better DX/AX, lower cost, and more flexibility on software you own. And of course, it's all open source.
2
23
1,133
Did you know @enroute_sh is the fastest git server, no matter how you run it? Our internal dogfooding forge runs on a single Fly.io machine and still beats any other Git hosting solution.
The @enroute_sh Git server is not only extremely fast but also fully programmable. Start it with a single command on your machine and build against it. Self-host it on your own VPS or deploy it as a multi-Raft cluster for high availability. Any setup is just a config change on the same binary. Read our blog post to get the details! enroute.sh/blog/four-ways-to…
10
398
Sam Reis retweeted
so many gems and great tips in this talk from @yminsky about how to improve your codebase for agents • agents thrive on feedback, and types are one of the best kinds • make illegal states unrepresentable with types, and it enforces global rules across your codebase • property based tests, fuzzing, and even full-system deterministic simulation testing • expect tests: just print stuff out and capture it in the test. makes behavior legible. it's very similar to how compilers have fixture tests, it reduces the review surface area • formal methods used to be very expensive: seL4 is about 8,700 lines of C and took around 25 person-years, with 23 lines of proof for every line of C code, but agents change the roi youtube.com/watch?v=rUYP4C29…
129
181
35
2,692
375,183