Eric Fredine (@fredine) header
Eric Fredine (@fredine) profile picture

Principal engineer writing code for fun and profit. Lapsed photographer. Husband. Father. Aspiring fly fisher.

Vancouver, British Columbia
Joined July 2008
Photo by Eric Fredine (@fredine)Photo by Eric Fredine (@fredine)Photo by Eric Fredine (@fredine)Photo by Eric Fredine (@fredine)Photo by Eric Fredine (@fredine)Photo by Eric Fredine (@fredine)Photo by Eric Fredine (@fredine)Photo by Eric Fredine (@fredine)Photo by Eric Fredine (@fredine)Photo by Eric Fredine (@fredine)
Eric Fredine retweeted
"The question is not whether there will be iron, but what kind of Irontown we build with it. I want our conversation about AI to make room for nuance and genuine understanding, for us to look at it and each other with eyes unclouded by hate. We should take advantage of its real benefits, but also recognize the damage it can do, and the choices we still have about what we build and what we’re willing to sacrifice for it." so good @mitsuhiko lucumr.pocoo.org/2026/11/9/i…
2
65
4,856
Nice explainer of the Riemann Hypothesis
Guys, this is very serious and you have to read this right now. The Riemann Hypothesis may have just been solved, and if the claim holds, it's an insane achievement. In mathematics there is a famous problem called the Riemann Hypothesis, and it's about prime numbers. Primes are numbers like 2, 3, 5 and 7 that can only be divided by 1 and themselves. Every other number is built by multiplying them together, so they're the building blocks of all numbers. You can never tell exactly where the next prime will show up, but if you step back, there's a pattern. Around 1,000, about 1 in 7 numbers is prime. Around a million, about 1 in 14. Around a billion, about 1 in 21. Primes get rarer at a steady, predictable pace. Because of that pattern, mathematicians can estimate how many primes there are below any number. But the estimate is never exactly right. It's always a little off, and the big question is how far off it can get. Think of flipping a coin 1,000 times. You expect 500 heads, but you'll get something like 480 or 520. It's a little off, but never wildly, and you can predict how far off it will be. In 1859 a mathematician named Bernhard Riemann found that the prime estimate's error isn't random. It's controlled by a hidden set of special numbers. Where those numbers sit decides how big the error gets. Riemann guessed that every single one of them sits on one straight line. If he's right, the error always stays small, just like the coin, and primes are as orderly as they could possibly be. If even one of those numbers is off the line, the error can grow much bigger, and primes are far messier than we think. So far computers have found trillions of these special numbers, and every single one is on the line. But there are infinitely many, so checking will never be enough. You need a proof that covers all of them at once. For 167 years nobody has found one. It's one of the seven hardest unsolved problems in mathematics, with a million dollar prize attached. Now a mathematician has posted a note saying that, working with Claude, he believes he has that proof. He says a computer has already checked every step, and that Claude raced ahead of him near the end. Experts still need to confirm it, but this isn't a random internet claim. If it holds, it doesn't mean we can predict every prime. It means we finally know the pattern can never go wildly wrong. Thousands of math papers that begin "assuming the Riemann Hypothesis is true" become proven fact overnight. We'd also get firm limits on how far apart primes can be, and a guarantee that a fast method for testing whether a huge number is prime always works. And no, your bank account isn't in danger. Online security depends on how hard it is to break a huge number into its prime factors, and this proof doesn't make that any easier. If anything, it gives the math behind encryption a stronger foundation. If it holds, this is bigger than one math problem. A question that beat the greatest minds in history for 167 years would have been cracked by a human and an AI in a matter of weeks. If AI can find answers and computers can check them, the limit on science is no longer how smart we are. It's how many good questions we can ask.
3
345
We’re now trying to do root cause analysis with a group of journalists and political appointees.
Well that's what I thought watching in real time, that they were just filibustering, but Mullin eventually did provide a relatively coherent explanation ~15 minutes in, and Duffy was clearly fuming about it because it was a) off-message and b) blamed Duffy!
1
151
Why the flight stoppage got more coverage than the MSFT outage
Replying to @fredine
I think news editors were wary to universalize their own office hell to every working person in the country suffering alone and disconnected, whereas at an airport that universality is visible and manifest and can be photographed
140
Great post on scheduling parquet on S3 reads
Story time: How I set out to build a parser and ended up implementing a full-blown scheduler. When I started Hardwood, a Parquet reader (and writer) for Java, I expected the hard part to be decoding: types, encodings, nesting, SIMD. All this is real work, but for remote files (S3), the hardest part is deciding what to do next. Depending on region and distance, every request can cost tens of milliseconds before the first byte arrives. A reader that fetches a column chunk, decodes it, then fetches the next one spends most of its time waiting. To hide that latency you need many requests in flight. Then you need to decide when to issue which ones, how many, and what to do with the CPU while they're outstanding. For Hardwood, the goal is that a read should either run at the speed of its slowest resource (network, cores, memory) or of the code consuming the rows, but never slower due to scheduling. That means hiding fetch latency as much as possible, and no core idle while there are pages ready to decode. It also means not reading ahead without limit, because memory is finite and one read must never starve another. Parquet makes this harder than it sounds. The metadata sits at the end of the file. A page can't be decoded until its bytes have arrived, its column chunk's dictionary has been decoded, and its position in the output batch is known. Some files have no page index, so you only find the pages by scanning headers as the bytes come in. Every one of those steps finishes in its own order, on its own thread. So Hardwood now has a complete work and IO scheduler. All the work, from fetching a byte range to parsing a footer, planning a row group or decoding a page, is a job. Each job is keyed by the first row it serves, and the lowest key runs first. IO jobs (fetching bytes) run on Java 21 virtual threads, while CPU jobs (decoding) run on a worker pool with a configurable size. Optionally, CPU jobs can run on the caller's thread instead, should you want no extra threads competing for the CPU (e.g. in a single-core K8s pod). Separate bounds govern how far fetching and decoding may run ahead of the consumer. The two are bounded apart because their costs differ: request latency calls for many bytes in flight, while decoded values are several times larger than compressed ones and need only a few batches of head start. To guarantee progress, whatever the consumer needs next always runs, no matter what the bounds say. Pages are decoded straight from the fetched buffers into output batches, so the engine also tracks who still reads each buffer and frees it once the last page is done. None of this is specific to Parquet. Any format read over a high-latency link, with dependencies between its parts, ends up here. I'm planning to write up the details of Hardwood's scheduling engine in a longer post at some point. For now, the short version: if you're building a remote reader, you're building a scheduler. If only someone had told me that upfront ;)
1
13
1,326
Eric Fredine retweeted
I urge all to read "No Silver Bullet - Essence and Accident in Software Engineering" by Brooks. TLDR: shrinking all the "accidental" complexities in software engineering (semantics, boilerplate, dep management, awkward syntax, composition of code and packages, etc.) does NOT reduce the "essence" of complexity. And often, out of accidental complexities emerge new essential complexities. Shrinking down all the accidental bits to zero does not inherently and instantly result in the core complexities disappearing: if so, with the convention of ergonomic languages like Python and Lua and TypeScript, or better tools like IaC, we would have seen leaps and bounds in software capabilities and productivity. But, what emerges and transforms, like a werewolf, is increasing essential complexities that are, at the heart of it all, bound to the domain of the software and abstractions of the real world. What happens when abstractions and ergonomics and new ways of working arrive is that the inherent complexity transforms and re-appears: > "The complexity of software is an essential property, not an accidental one. Hence descriptions of a software entity that abstract away its complexity often abstract away its essence." AI and agents have changed software development. But the hard essential bits in software remain (if not are 10x) - today, instead of focusing on syntax, focus on how to build stable, complex, interlocking systems where the complexity is well managed.
Replying to @thdxr
i think the confusion is the urge to have pre-ai style opinions on programming but the entire point here is a lot of these things stopped mattering because ai can deal with it either way
15
29
2
258
14,631
Eric Fredine retweeted
Replying to @Paddys_posts
At this stage in a startup's life, revenue has little momentum. It goes up because the founders make it go up. So if they have to focus on something else, it stops going up.
13
13
1
523
40,691
it's not in the press release, but Canada actually reuse the heat from GPUs to boil maple sap into maple syrup 😋
We're proud to announce the initial close of our Series E, targeting up to CAD $7 billion at a CAD $35.7 billion valuation; marking the largest private financing round in Canadian startup history.
1
1
17
830
Eric Fredine retweeted
when nick told me that he had some interp breakthroughs, he was uh, underselling it. i think this may be remembered as a major moment in scientific history. news so good even eliezer will have to lower his pdoom. we can start making alignment into a 'normal science'
interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works already-clean.goodfire.com/
163
395
84
6,619
440,160
A laudable left-liberal impulse toward inclusion became a vehicle for all kinds of deeply illiberal anti-individualistic identity politics concepts to enter mainstream Democratic politics. theargumentmag.com/p/the-fox…
30
89
16
798
57,984
Eric Fredine retweeted
The real world abounds with recursively self-improving systems, but one in particular deserves attention: Science, modeled as a system (perhaps even as an agent, with goals and resources). If you want to really understand AI RSI, science should be your reference point. Science is an intelligent system, and it is obviously recursively self-improving: 1. Scientific discoveries unlock new technology that helps build better experimental tools. This is a top driver of progress in nearly all fields. 2. They unlock new conceptual advances (ideas, theories) that help solve more problems. 3. They increase society's economic output, leading to more resources flowing into science. 4. They unlock better faster tooling (e.g. more compute via better chip & networking technology). As a result, many measures of scientific *input* grow exponentially: 1. Headcount (doubles every ~15 years) 2. Global R&D spending (doubles a bit faster, every ~13 years) 3. Papers and patents (technically this is a measure of headcount) 4. Compute dedicated to science (doubles every ~2 years) But is scientific progress exponential? Historically, the rate of scientific impact over time has remained roughly constant since the start of the industrial revolution (i.e. scientific progress is *linear*). 1850-1900 was about as dramatic as 1900-1950 or 1950-2000. 1850–1900: Evolution, germ theory & antiseptic surgery, thermodynamics, electromagnetic field equations, the periodic table, pharmaceuticals, electricity, telegraph and telephone, internal combustion engine, skyscrapers, mechanized agriculture... 1900–1950: special and general relativity, quantum mechanics, nuclear fission & atomic energy, antibiotics, genetic theory, electronic computers, information theory, synthetic polymers and plastics, the transistor, aviation... 1950–2000: DNA, genetic engineering, integrated circuits, microprocessors & personal computing, the Internet, crewed spaceflight, moon landing, satellite communications & GPS, standard model of particle physics... In real terms, like life expectancy, which has increased in a remarkably linear fashion of roughly 3 months per year since 1840, progress is a straight line. This is especially apparent for fields where impact is easy to measure, like biology, medicine, and agriculture. I first wrote about this phenomenon and its causes in 2012, and a steady stream of research has confirmed it in the years since. Examples include the 2018 paper by Nielsen and Collison, "Science Is Getting Less Bang for Its Buck," and the 2020 economic paper, "Are Ideas Getting Harder to Find?" (In fact, I believe the Nielsen paper stemmed from a conversation I had with him about this exact idea six months earlier) In short, the primary cause is that research solves the highest-impact, easiest problems first, and every subsequent problem is either harder or lower-impact. Exponentially so. The paper that presented information theory wasn't very hard to write (single author!) but you'd have a hard time ever writing a CS paper that beats it in impact. This is why science as a system requires exponential resources (input) to produce linear impact (output). It gets exponentially harder over time. Worth thinking about if you're pondering RSI for AI. I fully believe AI RSI is already happening and will accelerate in the future. But I do not believe this leads to an "intelligence explosion" -- that would fly in the face of everything I know about intelligence and everything I know about recursively self-improving systems.
166
213
44
1,812
176,617
Eric Fredine retweeted
Goose: my latest programming language. Safer, faster and less memory than Rust/C++ for algorithms where tightly packed, flat data structures matter. It can represent more kinds of data structures as flat, pointer-free bytes than any language I know. github.com/aardappel/goose 1/
114
159
62
1,397
144,346
Eric Fredine retweeted
"I spent a lot of time going over the design decisions involved here, trying my best to complain about async Rust, only to end up going “Oh. Yeah. That’s reasonable. That makes sense. I can see why they did it that way.” every step of the way." herecomesthemoon.net/2026/10…
7
12
1
145
6,137
Eric Fredine retweeted
The critical distinction between base LLMs (2024 and earlier) and modern LRMs is not symbolic tool use. It's the switch from a transductive paradigm (intuit the answer to the query) to an inductive paradigm (intuit the program/instructions that produce the answer to the query). They're trained to be inductive, and they perform test-time induction, i.e. test-time prediction of a NL program / reasoning chain. This unlocks entirely new capabilities -- in particular fluid intelligence. Base LLMs, to this day, have ~0 fluid intelligence. LRMs have substantial levels of fluid intelligence. The performance of LLMs on ARC 1 (a benchmark from 2019) remains ~10-15% today. Scaling them up by a factor ~100,000x got them from 0% to 10%. Meanwhile LRMs the same size or smaller saturated ARC 1 in 2025.
123
145
27
1,349
91,095
Eric Fredine retweeted
Meanwhile, at a dollar store in Canada
2
2
309
Eric Fredine retweeted
The more time I spend working with coding agents, the more convinced I am that they make software engineering even harder We can do amazing things with them, but unlocking their full potential requires extraordinary discipline and knowledge
Replying to @GergelyOrosz
Bury your head in the ground at your own risk. I aim to not jump on any hype trains, but since Opus 4.6 and GPT-5.2 + the harnesses it was clear that these things can write code nearly as good as I can in my best language; better in other languages. But we won't see non-devs push production code for a long time; possibly forever, if you ask me. Software still needs to be built in robust ways, and it's our profession to do this well, and use the new tools we have, which create new and interesting (+tough!) challenges
381
389
109
4,466
626,715
Eric Fredine retweeted
The right architecture for durable execution depends on the size of the state, the cost of unloading and reloading it, the type of state, the duty cycle, and the mutation rate. The right answer could be long-running VMs, local snapshots, remote snapshots, S3, a DB, etc.
I used to think durable execution required a server and workers. Now I think it's just a library and an S3 bucket. Change my mind.
9
31
1
307
25,797
New approach. I can think of a lot of possible applications.
After co-inventing ChatGPT, I kept asking myself: why have superhuman chat models not led to AGI? I’ve spent the last 2 years in stealth building a new way to train models (RLCD), and a new type of frontier AI model that we are releasing today: Jev • 20-200x faster • 40-400x cheaper (w/ output tokens free) • Frontier composable intelligence optimized for decisions AFAICT the shortest path to AI-based economic revolution
8
182
Eric Fredine retweeted
I must be among an extremely small group of people (n=1?) that have both 1) trained a frontier LLM and 2) designed and synthesized custom viruses in a lab with my own two hands. And I think that the takes on AI killing us all by creating dangerous viruses is total bogus.
567
1,887
436
14,353
1,958,070