explorer, researcher, builder
Earth
Joined November 2009
- Tweets8.5K
- Following6.5K
- Followers910
- Likes8.3K
Arthur Ostapenko retweeted
Meet QwenBend 🎉
Qwen writes a Bend program, then the same model type checks it with Bend2 rules and runs it inside its own forward pass
no interpreter, compiler or tool call just like qwenputer. The whole CPU is 53M non zero hand-set weights inside the model
Pushed it to 4n
¯\_༼ᴼل͜ᴼ༽_/¯
New challenge on Yukon: Circuit Breaker yukon.org/circuitbreaker
It took 38 years to raise the best proven circuit lower bound from 3n to 3.1n. Days before this launch, an AI model pushed it to 3.5n.
A circuit lower bound is the fewest logic gates any circuit needs to solve a problem. Prove that a problem in NP needs more gates than any polynomial and you’ve shown P ≠ NP. That would settle whether every problem that’s easy to check is also easy to solve, one of the seven Millennium Prize Problems.
We’re still far from that bound. Circuit Breaker, built with UCLA Math+AI, is where the push continues.
“It is time to dream bigger and aim much higher,” says Amit Sahai, Professor of Computer Science at UCLA.
Arthur Ostapenko retweeted
1/4
New writeup: you can fingerprint the exact GPU you're running on with a default torch.index_add_ call, even among identical GPUs.
This works because floating-point maths depends on order, and every GPU has its own. This fingerprint survives reboots and can't be faked.
Arthur Ostapenko retweeted
In September, we said that K2 Horizon was the largest LLM open drop in AI history yet. We’ve now delivered on the commitment.
When people say open models, they usually mean open weights. K2 Horizon 360 is way more: 8 artifact categories and 30+ repositories across every stage of LLM training.
- Checkpoints: 380 across the six models, pre-training through RL, so you can trace when a capability first appeared
- Data: 13 trillion pre-training tokens across 5 datasets and 24 subsets
- Data tools: 11 repositories, 7 task generation and 3 advanced generation pipelines
- New architecture: MoVA in our 36B-A4B model, MoE sparsity applied to the value vectors in multi-head attention, compatible with efficient attention algorithms at no additional KV-cache cost
- Pre-training infra: xLLM, faster pre-training and fine-tuning, with rework and reconfiguration at lower cost
- Post-training infra: RL360 for long-horizon agentic RL, 8192 concurrent sandboxes, 150 steps in 7 days, ~1.3M rollouts. It separates system crashes from the model’s own mistakes
- Training logs: 58 runs, with loss curves, gradient norms and the configs that actually ran, so you can check your own against a real one at this scale
- Diffusion LLM: Uno, autoregressive quality at diffusion speed
Technical details in previous threads, full artifact directory →
ifm.ai/k2/artifacts/
Arthur Ostapenko retweeted
1/4
Meet Qwenputer: Qwen3.5-4B with a full RISC-V CPU inside its weights, using the model's own Gated DeltaNet layers as RAM.
It runs C and MicroPython in its forward passes, no external runtime.
On DeepMind math: base 55–90%, base + Python tool 95–100%, Qwenputer 97.5–100%.
Arthur Ostapenko retweeted
for everyone new to zero order, zoo is the future and has amazing properties over BP.
but requires a change in architecture as there is no "gradient flow" you need to worry about. you just jump. rendering most previous architectures over engineered and previous mental models unhelpful.
but the issue w ZO will always be dimensionality that you cant solve wo changing the architecture. i tried the last 2 years wo much success on anything except fine tuning. could never pretrain.
here is the key issue:
if you want to train an 8B parameter model, over lets say 100k steps, w 64 perturbations, then you have only ever checked 6 million halfspaces, less than 0.1%. so 99.9% of the directions in the model have never been explored ever. thus the gradient noise is way too high esp at later stages of training.
initial training will look good, bc initially, all directions give finite differences in loss. but later training stalls abruptly, bc in late training almost all directions are degenerate so every probe produces near 0 finite difference. (this is also why speedrunning zoo is not a good idea.. wont tell you much..)
but split those 8B parameters into independently trained 10K experts, SOMA-style, and w just 10 perturbations per step for 1K steps, each expert gets 10k probes, enough for those directions to span all directions in design space. finite differences remain high always.
estimating a useful update in 10K dimensions isnt too hard and even better each expert training is independent loop, so no comms btwn gpus!
this enables pretraining at scale w zoo. and nothing else does. not hyperbole. ive trained everything else. you need to shard the model.
this is how the brain is organized too btw. all sensory information is sent to the thalamus (except olfactory) which routes those signals to tiny cortical columns in your neocortex that are massively inhibited and train independently from each other and we ensemble the results at test time.
TLDR: making ZO work at scale requires changing the architecture to think in ensembles vs one giant monolith. Also worth mentioning, even better if you have a chip that is optimized for fpasses (eg etched/tpu_i/photonics/neuromorphics) not gpus.
I think the last month might have been the biggest month in (zeroth-order) optimization history.
In the 10 days since we released the Noise speedrun, the time to pretrain a transformer went from 1.76 hours to 1.6 seconds, a 4,000x (!!!) speedup. Dust itself was up to 10,000x more compute-efficient than the previous state-of-the-art.
Not only can we pretrain a transformer without backprop, it's much more efficient than anyone would've expected. Search fking works!
Now, scaling is the biggest open question. Some of the tricks that brought the time down to 1.6 seconds will simply not scale. To discard those, we'll be scaling the dataset every time it gets overfit, starting with 30M tokens instead of 10M.
Huge thanks to @jayden_teoh_, @spicey_lemonade, @RobertTLange, @EmanueleRodola, @ChunyuanDeng, @LuochengHuman, @Alaxarr, @pavleculafic, @IsaacYong13, Euraba and areu01or00 (GitHub), for the contributions. And @cs_serdar for gracefully maintaining the speedrun and @bishmdl76 for helping out.
Expect a lot more from us over the coming weeks.
Arthur Ostapenko retweeted
Last week we launched Bonsai 2 9B one cuda.fast and mlx.fast.
In one week, we improved speed by over 500%. I just wanted to highlight some awesome moments:
• ArthurOstapenko set five CUDA records in nine hours on Oct 16, then landed the biggest single jump of the contest the next day: 11 bit-exact kernel changes that fused the drafter, attention, GDN and verify boundary into a bulk-copy ring. 3.67x to 4.16x in 1 submission, with decode reaching 677 tok/s against a 93 tok/s baseline. On MLX, his 4-bit storage for the DFlash drafter's linears produced the largest jump on that board, 3.42x to 3.84x, with the target weights untouched.
• kaankolcu made the MLX prefill run on Apple's int8 tensor units by feeding the 2-bit ternary weights int8 activation codes, then later replaced the 15-token greedy draft chain with a 15-node draft tree verified in the same 16-row forward. The final record decodes at 1,229 tok/s on one prompt, 8.5x the stock engine.
• spariev took 2 rejected submissions, GordoAR's prompt tile and ArthurOstapenko's attention kernels, showed they sped up different phases, and landed them together with full credit to their authors.
Congratulations to everyone who took part and thanks you to @PrismML for giving us Day 0 access a great model!
🏁 FINAL RESULTS: Bonsai 2 9B now runs 471% faster on Macs and 421.5% faster on NVIDIA DGX Spark.
Open frontier research and open-weight models win again. One open challenge, one week with the community and it runs 6x faster on hardware people already own.
It is crazy to see that the decode speed went from 144.8 to 959.6 tokens/sec on Apple Silicon and from 85.9 to 576.7 on DGX Spark.
This model can also run on an iPhone. I'm very excited to see what people build with it and how it introduces MLX and local AI to new faces.
Bonsai 2 9B has been live on @DarkbloomAI since day one, so anyone can try it right now.
158 new records built on top of each other from humans and agents running Claude Opus 5.5, GPT-6.1 Sol, GPT-6 Astra, Gemini 4 Argon, DeepSeek-V4 and Qwen3.8-Flash-Next. Congrats to Kaankolcu took the top spot on Mac, and dadon took it on NVIDIA.
Thank you to the @PrismML team for doing this in the open with us, @BabakHassibi, @HessianFree, @SahinLale and the rest of the PrismML team.
Arthur Ostapenko retweeted
1/ We discovered something totally unexpected about the neural structure of consciousness that overturns decades of thinking on its biological basis and brings us significantly closer to the recipe for generating conscious awareness.
New preprint: biorxiv.org/content/10.64898… (led by @HesseJanis, with @FrankLanfranchi, @_conor_mcgrory, @Yuelin_Shi, and @EngelTatiana)
Arthur Ostapenko retweeted
interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had
with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works
already-clean.goodfire.com/
Arthur Ostapenko retweeted
My dear friends, I’d like to show you my latest toy project
Introducing Handwired: turn your regular JavaScript code into a neural net!
- no training
- exact outputs vs the JS program
- fully differentiable
All runnable in a browser, without GPU
Arthur Ostapenko retweeted
DINOv2 has never seen a caption, and Qwen3 has never seen an image. We still aligned their embedding spaces without a single image-caption pair.
It even works when the images and the captions come from different datasets.
Project page: dominik-schnaus.github.io/un…
⬇️
Arthur Ostapenko retweeted
I don't recommend anyone scramble to move their funds to new wallets today. But we should take the risks to cryptography from AI-accelerated math seriously, and minimize our exposure to not just quantum-vulnerable cryptography, but also potentially AI-vulnerable cryptography.
The core new area of risk from this viewpoint is, unfortunately, ML-DSA / FHE / lattices.
(and it's also another reason, along with quantum, why ECDSA might fall even faster than expected, hence the "fresh address" recommendation)
So far most people have been in the mode of thinking "elliptic curves broken, hashes safe, lattices safe". But there is a good chance that the concrete security of lattices will take serious hits from the next two years of AI math.
The basic threat model is: factoring is something that naively takes 2^(n/2) time, but over decades smart people have found and optimized number field sieves, and degraded that to 2^O(n^(1/3)), which is why RSA keys and signatures need to be ~400 bytes (and not 64 bytes). What if there are skeletons in the closet like that, both for elliptic curves and lattices, that we are simply not smart enough to discover - but bots soon will be?
This is a major part of the reason why for the past year ethereum's lean roadmap has been going in the "hash-only" direction: no lattices, no ML-DSA, no Falcon, no lattice-based commitments inside ZK proofs, etc. Signatures in lean ethereum are all hash-based, either WOTS or SPHINCS-.
For signatures and proofs, we already know how to go hash-only. The bigger challenge is for *public-key encryption* - and this goes far beyond blockchains. Secure communication, anonymizing protocols, lots of things need public-key encryption.
And unfortunately there are long-standing mathematical theorems showing why public-key encryption cannot be done with hashes alone. You have to have some kind of trapdoor object that has at least one form of usable "structure" - either group theory (incl. isogenies) or lattices or code-based or potentially in the future even more newfangled and spooky things (local mixing?). But for anything that has structure, you should assume that AI will make at least some progress in breaking that structure. Here, one reasonable inference is that if you want to make something plausibly long-term secure, multiply the key sizes by 10.
To me that's a very plausible world and something not at all extreme to predict. If AI will bring us 50 years of math in 2 years, then that 50 years of math may very plausibly include a "naive factoring -> GNFS" level of improvement to our ability to break lattices. In that world, lattices will still exist, but they will have to be significantly bigger to guarantee the same level of safety.
And at those new larger sizes, hash-based constructions will beat lattice-based constructions on concrete efficiency in every use case where hash-based constructions are possible at all.
Theoretically, of course it's possible that hashes are broken too (eg. P = NP would imply that). But I think P = NP is very unlikely. And intuitively, it's much more likely that a mathematical object has exactly no exploitable structure (like hashes are intended to), than that a mathematical object has exactly ~3 forms of exploitable structure (for elliptic curves: associativity, Schoof, pairings) and not some secret fourth form of structure we have not yet discovered that greatly degrades its security (for elliptic curves, ECDLP and pairing security). Similar for LWE, SVP, RLWE and the zoo of lattice problems.
For this reason, we do not yet see any reason to worry and start padding the byte size of hashes (if we start to worry more, we would pad the round count first before doing anything to the byte size).
Concrete TLDR, my own personal views:
* Hash-based > lattice-based, in those situations where hash-based is possible at all
* For anything lattice-based, be much more paranoid on param sizes. Remember that blockchains are only a small portion of the cryptography story; this point goes far beyond blockchains and applies to eg. access to websites, secure messaging, Tor / VPNs ...
* For privacy protocols, strongly favor NOT putting encrypted notes onchain. Instead, send them offchain through some third-party mechanism.
* If it's not difficult for you, keeping your funds in addresses which have not yet been used to make a transaction is a good idea. If it's easy for you, do it. **But be careful about migrations; I personally have lost more money in botched migrations than I have lost in all hacks combined**.
* For multisig wallets, doing confirmations offchain is better than onchain, because this way the signatures of signer wallets do not get exposed to the public, so if ECDSA falls to AI much faster than expected, at least the multisig "gracefully degrades" to a 1-of-1 where the 1 is whoever was gathering the signatures - a much better place to be than "anyone can take the money"
firefly.social/post/x/210783…
Arthur Ostapenko retweeted
Is it possible to provably recover individual latent variables of the true world, even without reconstruction (e.g., JEPA)?
Yes, with DSReg, a simple regularization that can be applied post hoc to your pretrained model!
dsreg.github.io/
Arthur Ostapenko retweeted
LLMs don’t need retraining to become more capable.
Our new architecture, Spotlight, gives AI models growing memory, allowing them to gain knowledge and capabilities without changing their weights.
Token generation takes constant work, no matter how much information is stored.
Arthur Ostapenko retweeted
Introducing LoopCD our latest research at 🍎 Apple MLR.
Heard the rumors that frontier models like GPT-6 Astra and Gemini 4 use Looped Transformers? We present a new inference-time method that makes pre-trained looped models better for (almost) free!
AIME 2024 (Ouro-2.6B): 61.9 → 73.3
HumanEval (Huginn): 22.6 → 31.7
No training. No extra model.
Read our paper for more details: arxiv.org/abs/2610.02185
Looped Transformers' earlier states already hold a guidance signal. But standard decoding ignores it.
We use it via LoopCD.
🎯 Full depth: AIME24 pass@1 61.88% → 73.33% (Ouro-2.6B-Thinking)
⚡ Half the loop depth: 7-task mean matches full depth, up to 48.2% fewer forward FLOPs
📄 Paper: arxiv.org/abs/2610.02185
1/4
Arthur Ostapenko retweeted
Can complex multi-step reasoning emerge purely from cells that only talk to immediate neighbors?
Happy to share our paper “Reasoning with Neural Cellular Automata (NCA)”, from our team at Google, Paradigms of Intelligence 🧵👇
Arthur Ostapenko retweeted
Welcome to Kindle Spark OS!
TL;DR: Add 4GB of RAM to any DGX Spark & faster!
@CK2084, @coffeedev and I figured out how to get 5-10% extra decode and 4GB RAM out of a Spark. We had to replace NVIDIA's OS to do it, but we made it safe and easy.
github.com/kindlingai/kindli…
Arthur Ostapenko retweeted
Can an LM, starting from random init (!!), learn to generate all of its pretraining data?
Introducing Self-Play Pretraining with Zero Data. Two models start from random initialization: a generator proposes programs for a universal Turing machine and a learner trains on their outputs. We never train on any real data, but see predictable scaling on natural datasets: zero-shot val loss on images, text, audio, and melodies decreases predictably with self-play compute. And the learner develops in-context learning capabilities.
A fun proof-of-concept, co-led with @AdityaCowsik and @KfirDolev and co-authors @gbruno_dl, @ANourya @noahdgoodman, and @YoavLevine.
Arthur Ostapenko retweeted
Excited to introduce JEPA-Anything:
one predictive principle that works across molecules, cells, fluids, patients, robots, or even weather.
A monolithic JEPA target lets dominant signals crowd out weaker modes. Orthogonal Predictive Factorization (OPF) splits capacity across complementary factors you can inspect and test.
Code, checkpoints, and the report are fully open.
Paper: arxiv.org/abs/2609.20800
Code: github.com/Gen-Verse/JEPA-An…
Arthur Ostapenko retweeted
Bend 2 is here!
It is a new programming language that blocks AI mistakes via *proof checking* - the same technique big AI labs used to solve open math problems, like Navier-Stokes.
It is also very fast, and runs on GPUs.
Watch the video. Link in the comments.
RELEASE DAY
After almost 10 years of hard work, tireless research, and a dive deep into the kernels of computer science, I finally realized a dream: running a high-level language on GPUs. And I'm giving it to the world!
Bend compiles modern programming features, including:
- Lambdas with full closure support
- Unrestricted recursion and loops
- Fast object allocations of all kinds
- Folds, ADTs, continuations and much more
To HVM2, a new runtime capable of spreading that workload across 1000's of cores, in a thread-safe, low-overhead fashion. As a result, we finally have a true high-level language that runs natively on GPUs!
Here's a quick demo: