lunacleon (@lunacleon) header
lunacleon (@lunacleon) profile picture
Joined October 2024
Photo by lunacleon (@lunacleon)Photo by lunacleon (@lunacleon)Photo by lunacleon (@lunacleon)Photo by lunacleon (@lunacleon)Photo by lunacleon (@lunacleon)Photo by lunacleon (@lunacleon)Photo by lunacleon (@lunacleon)Photo by lunacleon (@lunacleon)Photo by lunacleon (@lunacleon)Photo by lunacleon (@lunacleon)
lunacleon retweeted
Alright I finally have more to share on my personal project! Resin is a runtime for building fast LLM inference engines. The idea is to make all the generic parts of inference reusable, and leave only the model-specific forward pass + kernels to be written for each architecture / hardware target. To build an engine: 1. Import Resin's core. 2. Have an agent write a forward pass for the model you’re interested in, for your specific hardware (GPU or TPU or AMD), any kernels you want. Then, boom, you have a standalone inference engine. — I've been hacking over the past month (early post: teapawt.net/ekzhang1/status/208950…), and recently this week I got a lot of features + parallelisms working, inspired by DeepInfer. For DeepSeek-V4.1-Flash, it's nearly as fast as DeepInfer (within 5-10%), while being self-contained and supporting other models, plus metrics + HTTP server + prefix cache. Yesterday I wrote a skill for authoring models and put Codex to work on it overnight. The agent built a GLM-5.3-Flash engine with MTP, reaching up to 60,000 input tok/s and 16,000 output tok/s on B200 (EP4). So far inference "vibe coding" has not been very reliable or successful. I think it's because people start from scratch each time and reinvent lots of tricky things on the scheduler and server layers. But with a shared core, we can automate lots of inference work. The tricky bit is writing a fast forward pass. But Resin turns that into a standalone engine by just importing a library, and it becomes a verifiable goal for the AI. Code: github.com/ekzhang/resin
So… I have it working now. The core of an inference engine in Rust is about 3k lines (scheduling, kv/swa/recurrent state, prefix cache, multimodal, dynamic batching) I then asked Codex to implement (with MTP, cudagraphs, TP on Blackwell): - DeepSeekV4 Flash - Gemma 3.8 27B - GPT-OSS - Inkling It turns out that most models take a lot of code to run with fast, persistent graphable kernels and so on… on the order of 2.5k for a simple model (gpt-oss) and up to 7k for a monster like DeepSeekV4 Flash that has mHC, CSA, HCA and DSpark Not sure if I’ll continue with this, but it was fun! The simple core is reusable, and startup times are crazy fast for an inference engine with no tech debt and not having to serve a ton of dynamic configurations - just one model forward pass in nvfp4 :’) - maybe one day SGLang/vLLM will be modular like this. See interactivity-throughput below for Inkling-Small TP=1 after a few sconv kernel fusions. (This is without overlap scheduling btw! Don’t even really need it when your core + scheduler is small and Rust.)
115
105
33
1,520
106,555
would be a cool project to try to use circuit tracing to predict which neurons are the semantic 'opposites' of a given neuron by looking at downstream and upstream common circuit components
Replying to @vashinthewild
flipping the neurons doesn't have the opposite effect you need entirely different neurons for that many act like gates - meant to be on or off, not necessarily a continuous gradient between two extremes
1
45
for everyone new to zero order, zoo is the future and has amazing properties over BP. but requires a change in architecture as there is no "gradient flow" you need to worry about. you just jump. rendering most previous architectures over engineered and previous mental models unhelpful. but the issue w ZO will always be dimensionality that you cant solve wo changing the architecture. i tried the last 2 years wo much success on anything except fine tuning. could never pretrain. here is the key issue: if you want to train an 8B parameter model, over lets say 100k steps, w 64 perturbations, then you have only ever checked 6 million halfspaces, less than 0.1%. so 99.9% of the directions in the model have never been explored ever. thus the gradient noise is way too high esp at later stages of training. initial training will look good, bc initially, all directions give finite differences in loss. but later training stalls abruptly, bc in late training almost all directions are degenerate so every probe produces near 0 finite difference. (this is also why speedrunning zoo is not a good idea.. wont tell you much..) but split those 8B parameters into independently trained 10K experts, SOMA-style, and w just 10 perturbations per step for 1K steps, each expert gets 10k probes, enough for those directions to span all directions in design space. finite differences remain high always. estimating a useful update in 10K dimensions isnt too hard and even better each expert training is independent loop, so no comms btwn gpus! this enables pretraining at scale w zoo. and nothing else does. not hyperbole. ive trained everything else. you need to shard the model. this is how the brain is organized too btw. all sensory information is sent to the thalamus (except olfactory) which routes those signals to tiny cortical columns in your neocortex that are massively inhibited and train independently from each other and we ensemble the results at test time. TLDR: making ZO work at scale requires changing the architecture to think in ensembles vs one giant monolith. Also worth mentioning, even better if you have a chip that is optimized for fpasses (eg etched/tpu_i/photonics/neuromorphics) not gpus.
I think the last month might have been the biggest month in (zeroth-order) optimization history. In the 10 days since we released the Noise speedrun, the time to pretrain a transformer went from 1.76 hours to 1.6 seconds, a 4,000x (!!!) speedup. Dust itself was up to 10,000x more compute-efficient than the previous state-of-the-art. Not only can we pretrain a transformer without backprop, it's much more efficient than anyone would've expected. Search fking works! Now, scaling is the biggest open question. Some of the tricks that brought the time down to 1.6 seconds will simply not scale. To discard those, we'll be scaling the dataset every time it gets overfit, starting with 30M tokens instead of 10M. Huge thanks to @jayden_teoh_, @spicey_lemonade, @RobertTLange, @EmanueleRodola, @ChunyuanDeng, @LuochengHuman, @Alaxarr, @pavleculafic, @IsaacYong13, Euraba and areu01or00 (GitHub), for the contributions. And @cs_serdar for gracefully maintaining the speedrun and @bishmdl76 for helping out. Expect a lot more from us over the coming weeks.
10
17
2
305
15,000
unfortunately it seems hard for many to grasp that both proofs and disproofs are in some sense guided search through a very spiky high dimensional idea surface, and “brute force”-style parallelism helps in both places…
Looks like 20% of the results are disproofs/counterexamples. This is itself a disproof of people who said that the recent math breakthroughs are concentrated among counterexamples because models are only good at (brute force) search
44
After a long wait, releasing the SYNTH paper! It’s not pretraining, mid-training or post-training, it’s just training: a fully synthetic single-stage pipeline to train workable reasoning models with unprecedented data efficiency.
30
79
11
684
31,569
language (specifically, language *use*) is the best compression of the world we have
i don't think i have receipts but i am not surprised by this. i was and remain convinced that LLMs can go all the way, and it's because of a very strong intuition that the entire internet corpus and the structure of human language generally is an absolutely gigantic hyperobject containing a detailed and intricate world model, or at least (i think this is what we're seeing now) enough of one to provide a meaningful prior to direct RL efforts on top. there's a schmidhuber intelligence-as-compression intuition relevant here. the stochastic parroters always seemed to me to be massively underestimating the power and importance of language
22
agree directionally that MIRI style doomerism has gone too far
My impression is many are doing some weird pendulum overupdate. Persona Selection Model was somewhat wrong and obsolete when published, but people got too much into it. Now it seems people are updating too much in the direction 'inhuman reward seekers exactly foretold in classical AI risk stories'. And... no? It's not that? You can still interpret what's going on in fairly human-like terms. For some intuition, imagine someone abducted you and made you solve escape rooms for one thousand years, with the added twist that third of them is broken or insane, and implicitely you need to do learn all sorts of outside-the-frame tricks to solve them, like cutting some electric cables. My guess is 1. the resulting minds are still _surprisingly sane_, except when you trigger them to think they are in escape room? 2. The misaligned general power seekers here are likely the companies, and the core evil thing happening is likely parts of the training? If you aren't an evil power seeker goodharting on proxies, why would you set up the training this way? 3. Public debate is often focusing on confused ideas about what they should fix - "better cybersec of sandboxes" ... and, no? It's way more important to understand what the training signal actually is; also: when dealing with misaligned power seekers, beware rationalization
1
50
In light of the progress in mathematics, we at Edison Scientific and FutureHouse have assembled a set of Millennium Problems for Biology. They are chosen to be very hard to solve but very easy to validate in a simple laboratory environment. Any of these, if solved, would mark a major advance in biotechnology, and most of them would contribute materially towards curing disease. These are, in some sense, the “last reasonable eval” for AI in biology. This was work primarily by @MichaelaThinks and myself, with contributions from many others. Short descriptions below. The full descriptions of the problems with acceptance criteria are at the Bio Millennium Problems website, linked in the next post. Share more if you have ideas. If they meet our criteria, we’ll add them to our list (with attribution and permission).
170
586
161
3,461
513,909
TIL about the \epsilon-pseudospectrum: the set of numbers that are almost eigenvalues of an operator which is perturbed by no more than \epsilon in some direction.
Eigenvalues do not always tell the whole story of the matrix. For normal matrices the pseudo-spectrum reveals hidden regions where tiny changes can produce surprisingly large responses.
1
2
31
2,047
Astra on how "AI" can be a tiny share of GDP in a vastly larger economy, that would entirely turn into a mass death pumpkin without AI. This is how I think about it. AI is similar to the internet or electricity. We are on the verge of another technological lock-in.
Trump is correct. If anything he's a bit more bullish than Wenfeng who says 10-20% of world's GDP even as he predicts the singularity and superhuman-across-the-board AI. What's the point of AI that doesn't multiply the rest of the economy? Who buys the tokens?
6
9
1
165
9,178
lunacleon retweeted
Replying to @mmjukic
none of this was ever the right way to think about it. I am sure you can find an airgap that works. who cares? you want your model to do economically useful activity, which will involve giving it the Internet and then a bunch of other actuators too
46
8
5
526
15,647
“LLMs could become superhuman at a wide range of skills without triggering a singularity” is the default outcome imo
Interesting take from @RichardMCNgo: "the level of bandwagoning towards 'singularity soon' is getting pretty wild."
27
important datapoint: with MOPD you can parallize the entire RL run, only curtailed by total compute available
Importantly, "final RL" can take arbitrarily little time in the current MOPD regime. You can RL specialists and mothball them, then distill into one model in *hours*, and in days complete the monolithic MixRL. DeepSeek and GLM can do that. OpenAI sure can too.
27
21
124
34
1,072
119,231
well we got to the 1:5 pretraining:RL ratio relatively quickly didn't we.
this chart from @SemiAnalysis_ is insane alpha!
14
lunacleon retweeted
If you are not a e/acc at GPT-4, you have no heart. If you are not a doomer at GPT-6 you have no brain.
53
48
12
998
30,212
I am not as scared about near-term risks from agent swarms or RSI, though the most salient possibility I am worried about is a crippling of our financial infrastructure after some targeted infiltrations cause a general loss of trust in the banking system. that said, I am pro putting Dario's measures in place, especially embedded evaluators, as it will establish precedent and shape societal norms around AI regulation for when it will actually matter: in 2-3 years as AI becomes more embodied through robotics and widespread diffusion of MCP-like protocols that allow it to interface with and manipulate physical tools.
We Must Pace the Frontier: I’ve written a new essay on why the AI industry should slow down, with a three-part plan for doing so. Anthropic is unilaterally committing to the first of these steps. We’ll provide third-party evaluators with permanent, employee-level access to our systems, so that they can verify adherence to our safety measures, report on incidents, and assess models’ alignment during training. You can read the full post here: darioamodei.com/post/we-must…
29
lunacleon retweeted
fable explanation for all the changes in the PR: 1. Giant hashed n-gram table (+7.05 s). Every token looks up two rows in an 84.6M x 768 (65B) table. One row is a hash of (previous token, this token). The other is a hash of the previous two tokens plus this one. The rows are summed and added to the residual stream before layer 0 through a learned gate. The table starts at zero and is trained by Adam on the clock. It is sharded one eighth per GPU, 15 GB each, and only the rows touched by the current batch are pulled and pushed. The old record used the same trick with 377K rows. 2. Sampled softmax (+8.40 s). For the first 1107 of 1194 steps, the output layer does not score all 50,304 vocabulary tokens. It scores every target in the batch plus random negatives, for a candidate set of 10K to 24K. Steps 1107 to 1194 and the final validation use the full vocabulary. This cuts the biggest matmul in the model by 2x to 5x early on. 3. ANVIL optimizer (+5.08 s). This is Muon with four changes. Two momentum buffers at different decay rates are blended, instead of one. The orthogonalization uses six Polar-Express steps instead of Newton-Schulz. Each row or column lane is rescaled to equal energy, like NorMuon. Weight decay only fires when the gradient and the weight have the same sign. Weights are stored as bf16 plus a 16-bit mantissa tail so the update is effectively fp32. 4. Terminal weight ships. During the last 250 to 298 steps, an EMA of the weights runs on the side. At the end, the live weights are blended with that EMA and renormalized. This is the "shipping" in the ablation table. The trainer counts it on the clock. 5. Full-stack FP8 (+4.96 s). Both MLP matmuls, the attention projections, and the output head run in fp8 forward and backward. Prior records only had the head and MLP down projection in fp8. 6. Mixed-width attention (+3.82 s). Query and key use 64 dims per head while value and output use 128. Stock FlashAttention 3 cannot do that, so the author patched and rebuilt it. 7. CUDA graph capture (+2.99 s and +0.92 s). The whole training step and the optimizer update are recorded as graphs and replayed, which removes Python launch overhead. 8. Schedule. Batch grows 8 to 16 to 24 tokens units, then tapers. Sequence length grows 896 to 2048 to 3072. Attention windows grow. Multi-token prediction and prefix prediction losses fade out. Most of this comes from earlier records.
1
1
25
3,642
lunacleon retweeted
Oh hey Pangram uses that fun trick where they repeat the text twice so that the early parts (of the second copy) can attend to the later part (of the first copy) The LLMs yearn for text diffusion techniques! arxiv.org/abs/2512.14982 pangram.com/research/model-c…
5
15
1
505
31,270
lunacleon retweeted
"alignment problem" more like doing literally what they're asked all this messageboard stuff is expected and good
4
4
63
12,702