Valentyn Kit 🦀 | Rust · Solana (@valentynkit) header
Valentyn Kit 🦀 | Rust · Solana (@valentynkit) profile picture

Rust · Solana · distributed systems. First principles - from primitives to distributed systems. Lead platform eng @smpl_app · ex UN · ex https://teapawt.net/t.co/gukOMIEqng

Ukraine
Joined January 2024
Photo by Valentyn Kit 🦀 | Rust · Solana (@valentynkit)Photo by Valentyn Kit 🦀 | Rust · Solana (@valentynkit)Photo by Valentyn Kit 🦀 | Rust · Solana (@valentynkit)Photo by Valentyn Kit 🦀 | Rust · Solana (@valentynkit)Photo by Valentyn Kit 🦀 | Rust · Solana (@valentynkit)Photo by Valentyn Kit 🦀 | Rust · Solana (@valentynkit)Photo by Valentyn Kit 🦀 | Rust · Solana (@valentynkit)Photo by Valentyn Kit 🦀 | Rust · Solana (@valentynkit)Photo by Valentyn Kit 🦀 | Rust · Solana (@valentynkit)Photo by Valentyn Kit 🦀 | Rust · Solana (@valentynkit)
I learned async three times before it stuck: C# Tasks, Go goroutines, tokio. Productive in a week each time, starting over by the next job. Fixed it by building a TCP server seven times in C, blocking sockets up to epoll, until I saw what every runtime is actually doing underneath.
1
2
8
2,148
Solana privacy, without mixers, shared pools, preserving amount and recipient privacy at the same time? Presenting you Scope Protocol for @solanaturbine Capstone Project. It was a fan Pet Project, to play around and really dive deeper into Solana Privacy, and how it works. Thanks: @AbhineetBiju, jmkoy
2
1
4
178
You map a file as shared in one process and write one byte. Nobody else has the file open. The kernel keeps a shared-dirty and a private-dirty count for that mapping. Which one says 4 kB?
1
2
139
It prints Shared_Dirty: 0 kB and Private_Dirty: 4 kB, in debug and release alike. smaps counts a page as private when it is mapped exactly once and shared when it is mapped more than once, and the kernel docs say that accounting is independent of MAP_SHARED.
1
10
Three containers on my Mac said DNS was broken. DNS was fine. Almost nothing was leaving the Mac at all. If your containers sit on a Docker network, their lookups ride the host's uplink, and when it dies the error comes back labelled DNS. Each container asks Docker's built-in resolver, which forwards anything outside to the host. When the host can't reach the internet, the failure still comes back from the resolver. - what the logs said: lookup failed, server misbehaving - how many: 235 error lines across three containers - what the Mac's Wi-Fi did: next to nothing, in and out - for how long: 32 minutes, until the reboot The graph that said it plainly was the Wi-Fi receive rate, flat near zero. I had alerts on every feed going silent and none on the link they all ride.
1
43
Indexing a byte slice in safe Rust can kill your process, and no panic hook will ever run. If your service maps a file and hands out a &[u8], this is your read path. My mmap bench has a mapping struct with a safe method that returns the whole file as a slice. One test walks it off the edge: - map a two-page file full of 7s, read byte 0 through the slice: 7 - shrink the file to zero bytes, from the same process - read byte 0 again, same safe method, no unsafe at the call - the process dies on signal 7, SIGBUS, and the next line never prints Rust checked the index against the slice's length. The slice still says two pages. The file says zero. A load past the end of a mapped file has no page behind it, so the kernel answers with a signal instead of an error. That is why memmap2 makes every file-backed map unsafe to create. Its docs say the danger is the file being modified "in or out of process", and the defenses are "platform specific and limited". My bench made both calls safe because it owns its file. That is a promise about every other process on the machine, written as a function signature. Who else can open the file your service maps?
28
Every stop-when-nothing-changed loop I have written trusted one empty round. UW-Madison proved it can give a wrong answer. Engines like Feldera, in Rust, must decide when a recursive query is done. Their counterexample to the original paper's rule: six nodes, one edit. In a plain loop, an empty round means done. To update fast, these engines diff each round against the same round before your edit. There a zero can mean "changed exactly like last time", and the real change lands one round later. Stopping at that first zero kept a path the edit had deleted. What got me: Feldera had already sidestepped it. It waits for its internal state to settle, and the paper proves that rule sound, in Lean.
1
2
180
I have load-tested event loops at a steady rate and called the tail done. Real arrivals come in bursts; random ones at the same rate hide them. A researcher at M2 Tech showed it on a year of exchange data. If your loop is slower than the packet gap, bursts set your tail. He put the oldest rule, one thread per hot path, through a simulated receiver on both. Packets come in bursts, a few microseconds apart. A loop slower than that falls behind on each one, and the waits stack. - one thread, 32 microsecond task: the slowest 1% took over four times the task - same load at random times: a far smaller tail - split over two threads: most of the tail gone, one hop added to the median One thread still wins wherever each packet is handled inside the gap. He says to measure that gap on recorded traffic, because a random load at the same rate hides most of the tail.
1
2
102
The CPU graph in my menu bar said 12% while macmon said 37%. Any script that reads Mac CPU from the first number on top's CPU line has the same gap. top's CPU line goes user, then sys, then idle. My graph's script takes the third word of that line, and the third word is the user share alone. Time spent in the kernel is the next number, and it never reaches the graph. - the kernel share it left out, 30 samples on my Mac today: 13.92% to 26.81% What got me: the obvious fix, swapping in macmon's number, would have changed what the graph means without a single error. It is the dashboard panel called "CPU" that two exporters fill differently.
4
182
A small Rust server with a low file descriptor limit took eleven clients, and the twelfth one took it down. If your accept loop returns early with ?, this is your server. It was the loop everyone writes first: take each incoming stream, unwrap it with ?, hand it to a thread. - clients 1 to 11 -> accepted, one descriptor each - client 12 -> accept fails: too many open files - the ? -> returns from main, and the server process exits Same limit, but match the error, log it, sleep a moment, continue: the server kept running and went on accepting. std's docs for accept now say a file descriptor limit error can be retried once descriptors close. I wrote that, merged in June.
3
81
The fans on my Mac were loud at night, and the cause was a job an AI agent had started hours earlier, in a session that had since moved on. If you spawn background work from a request handler, this is that task. The request ends; nothing tells the task. - the process: a DuckDB scan over every parquet file in a dataset - CPU: 200% - running for: 7 h 38 min - package power: 18.8 W, where it idles at 4 to 7 - the containers I suspected: under 1% CPU combined Killing it and putting the display to sleep took package power to 3.85 W. Now, before blaming anything heavy, I sort processes by CPU and read how long each has been running. Who owns the work your agent started after the conversation ends?
1
1
49
Delete the folder a docker run -v container mounts, restart the container, and Docker makes the folder again. Empty. If you start containers with docker run -v and a folder from the host, this is your restart too. My Temporal container ran into it. Its project folder was gone; the containers that used it were not. - on restart, Docker created the missing folder as an empty directory - that empty directory mounted over the path where Temporal reads its config - Temporal found no config file and exited - 51 restarts in 42 minutes Nothing in the log said mount. It said no such file or directory, about every 50 seconds. The rule is in Docker's docs: with -v, a host path that does not exist gets created for you, always as a directory. With --mount, the same missing path is an error and the container never starts. I would rather a container refuse to start than start on an empty folder.
2
268
For months, npm on my Mac had no gate against brand-new package versions, and I thought it did. Most of my package managers refuse any version younger than seven days. Hijacked packages usually get caught within a day or two, so a week of waiting skips most of them. npm's gate lived in a settings file, and a config sync had emptied it. bun had a different problem: it reads its settings from another file, so the one I set never did anything. I only found both by forcing each gate to refuse a normal install, then setting it back. Now a script asks each tool what it actually sees, instead of trusting my config files.
1
92
Wrap a File in a progress counter and Rust's io::copy quietly drops its fast path. Same 256 MiB file on a Linux VM, one wrapper apart: - io::copy(file, out) -> 7 syscalls, under 20 ms - io::copy(Progress(file), out) -> 65,542 syscalls, 188 to 386 ms When std can see both ends are files, it hands the copy to the kernel and the bytes never come up into your process. A reader std can't see through gets the generic loop: read a small chunk into a buffer, write it back out, until the file is done. The source lists this under drawbacks: wrap the file "to report progress" and you "can hit a performance cliff." I like a std comment that admits where its own fast path stops.
31
With io_uring, three file reads cost one system call, and picking up the results costs none. Normally every read is its own system call: ask the kernel, wait, ask again. io_uring sets up two rings in memory that your program and the kernel can both see. You write three read requests into the first ring. That is just writing to memory; the kernel has not been asked anything yet. Then one system call says: take these, and wake me when they are done. The kernel does the reads, and each result lands in the second ring as it finishes, in whatever order that happens. You read the results straight out of that memory. No second call. The catch: it only pays when you have a pile of reads. Sending one request at a time, my own reader did 12% fewer reads a second than plain pread.
1
3
92
My terminal froze in the middle of a debugging session, and the thing holding it up was a script I had written. LuLu, a macOS outbound firewall, asks about every new connection a program makes. My script sends each prompt, the process and its destination, to the Jev classifier with a written policy. At 90% sure or more, Jev allows or blocks on its own. Anything less reaches me; nothing fails toward allow. It is there for poisoned packages phoning home: node, python and curl get rules pinned to one host, so a new destination is a new prompt. In a test, no fake exfiltration host scored high enough to slip through. But asking Jev is itself a new connection. LuLu held it and prompted, and only that call could answer. Every prompt behind it waited, my terminal included. It is the auth service that has to call itself for a token before it can hand one out. Now the script lets its own traffic through by process path. Any gate I build gets its way past itself before I switch it on.
1
101
Valentyn Kit 🦀 | Rust · Solana retweeted
I read 8 MB of a file through mmap, and Linux pulled a whole gigabyte off the disk to serve it. Same reads with plain pread: exactly 8 MB. My first guess was a bug in my benchmark. A miss through a mapping is a page fault, and the fault doesn't fetch one page. It reads a window of the file around it, 4 MiB on this VM, betting you'll walk the neighbours next. On a scan, that bet pays. At random, every miss drags in a window you never touch. 2,000 random reads of a cold file: - pread: 8 MB from disk - mmap: a whole gigabyte, each read about 18x slower - mmap after madvise(MADV_RANDOM): 8 MB again Nothing in your code tells the kernel whether you scan or jump. It guesses, and a cold start pays for the guess.
2
2
1
9
654
I read 8 MB of a file through mmap, and Linux pulled a whole gigabyte off the disk to serve it. Same reads with plain pread: exactly 8 MB. My first guess was a bug in my benchmark. A miss through a mapping is a page fault, and the fault doesn't fetch one page. It reads a window of the file around it, 4 MiB on this VM, betting you'll walk the neighbours next. On a scan, that bet pays. At random, every miss drags in a window you never touch. 2,000 random reads of a cold file: - pread: 8 MB from disk - mmap: a whole gigabyte, each read about 18x slower - mmap after madvise(MADV_RANDOM): 8 MB again Nothing in your code tells the kernel whether you scan or jump. It guesses, and a cold start pays for the guess.
2
2
1
9
654
Harvard researchers gave caches more memory and watched some miss more. ARC, 2Q, S3-FIFO, SIEVE and TinyLFU all do it. If you run MySQL, its buffer pool is one of the caches they rebuilt. The trouble is a scan that repeats. Later pages push out earlier ones, so the next pass finds nothing until the whole scan fits. And on some traces a bigger cache missed more. The fix: pages that arrive while the cache fills stay put, and an evicted page comes back fast only if it was used recently. On MySQL's buffer pool, one TPC-H query missed 41.8% less. If you fight scans by shrinking MySQL's old-page share, they tried that alone too, and four other queries got worse. What got me: plain LRU provably never misses more with more memory. The caches built to beat it gave that up. I have fixed a slow cache by adding memory and never once drawn the curve.
2
4
284