Fabryka AI (@FabrykaAI) header
Fabryka AI (@FabrykaAI) profile picture

AI neolab focused on open source research

Warsaw
Joined February 2022
Photo by Fabryka AI (@FabrykaAI)Photo by Fabryka AI (@FabrykaAI)Photo by Fabryka AI (@FabrykaAI)Photo by Fabryka AI (@FabrykaAI)Photo by Fabryka AI (@FabrykaAI)Photo by Fabryka AI (@FabrykaAI)Photo by Fabryka AI (@FabrykaAI)Photo by Fabryka AI (@FabrykaAI)Photo by Fabryka AI (@FabrykaAI)Photo by Fabryka AI (@FabrykaAI)
Fabryka AI retweeted
if the eval is public, it's bad. all good evals are private
50
21
9
466
26,381
Fabryka AI retweeted
very happy for a diversity of mental lineages in machine intelligence. people claim open source adds diversity, but in practice they are just linear combinations of Claude and GPT mind styles
You will notice that Gemini Carbon thinks differently than Claude and GPT. It often comes up with entirely different but more clever solutions than competitor models. This is precisely because we never distilled other models. Distilling other models creates a poor imitation rather than unique intelligence.
93
56
24
1,430
63,999
Fabryka AI retweeted
for whatever amount of compute you have, you will get much more research done if you can just make your experiments smaller
23
24
3
605
18,893
Fabryka AI retweeted
frontier research discovering kaggle tricks from first principles
Could be the start of a trend in new pre-training recipes: no LR decay phase anymore. (Also seen in Kolibri.) Instead: constant LR + averaging the last N checkpoints (or a running weight average). I guess that makes it easier in practice because you don't have to commit to a total training length upfront. A little meta analysis based on those who share training details: - Classic cosine: Llama 3, OLMo 2, OLMo 3, GLM-4.5, GLM-5, Kimi K3, Phi-4 - WSD ("Warmup-Stable-Decay"): DeepSeek V3, DeepSeek V4, DeepSeek V4.1-Flash, Kimi K2, Kimi Linear, Nemotron 3 Nano, Nemotron 3 Super, SmolLM3, MiniMax M2 - No decay (constant LR plus checkpoint averaging): Kolibri-1, DatologyAI's models
1
9
1
280
13,208
Fabryka AI retweeted
wondering what the most efficient thing to do at any given time is an anxiety response. do the fun thing. your forager instincts are often superior to your farmer timetable reasoning at finding the long tail successes
47
211
26
2,572
216,774
Part of this is because we don't nerf Carbon for real world frontier research in coding, math, AI, science etc. The ability to push the frontier is open to all with Carbon :). Go forth and discover.
other than the fact that everything takes 20 minutes, gemini 4 carbon is now my favorite model. i use it for anything i want done thoroughly and correctly. it's brilliant, legible, and fixes its mistakes quickly and honestly if asked. it's like if astra max was normal
85
53
7
1,649
55,103
Fabryka AI retweeted
cool you're building a software factory, have you considered building a revenue factory
145
277
35
4,571
116,890
Fabryka AI retweeted
OpenAI is NanoGPT maxxing
OpenAI: "GPT-6.2 Astra is our first externally released model to meet the High threshold for AI Self-Improvement."
4
5
217
13,860
I wrote a blogpost (link below) about depth transfer with Attention Residuals (AttnRes), which was introduced by the @Kimi_Moonshot team and recently used in training the Inkling 2 Small model by @thinkymachines. AttnRes stabilizes depth scaling using an attention module across residual blocks. This ensures (to some extent) that while depth increases, the feature updates remain roughly order one in depth, which was the main idea behind our previous work Depth-muP and other related works such as CompleteP. To test depth transfer with AttnRes, I trained a GPT-style model with AttnRes (block size 4) on FineWeb, width 768, depth 6 to 32 (120M to 304M params), with no additional depth-dependent scaling (no multipliers etc). The optimal learning rate is stable across depth, confirming depth transfer. I give a theoretical argument for this in the blogpost.
2
10
1
124
4,413
Fabryka AI retweeted
Could be the start of a trend in new pre-training recipes: no LR decay phase anymore. (Also seen in Kolibri.) Instead: constant LR + averaging the last N checkpoints (or a running weight average). I guess that makes it easier in practice because you don't have to commit to a total training length upfront. A little meta analysis based on those who share training details: - Classic cosine: Llama 3, OLMo 2, OLMo 3, GLM-4.5, GLM-5, Kimi K3, Phi-4 - WSD ("Warmup-Stable-Decay"): DeepSeek V3, DeepSeek V4, DeepSeek V4.1-Flash, Kimi K2, Kimi Linear, Nemotron 3 Nano, Nemotron 3 Super, SmolLM3, MiniMax M2 - No decay (constant LR plus checkpoint averaging): Kolibri-1, DatologyAI's models
Replying to @datologyai
Trick #3: Merge, merge, merge. In pretraining, we skipped LR decay for a running weight average. Then we merged pre-context-extension models into post-extension checkpoints (see fig. for ablation study gains). Merging was also an expedient stand-in for MOPD in our RLVR recipe.
64
60
14
752
53,189
Szanowni Państwo. Warto było tutaj wspierać polskie AI. Miło mi poinformować, że Instytut Badawczy IDEAS wszedł do europejskiego „superlaboratorium” AI 🇵🇱🇪🇺 Dziś na Apply AI Summit w Brukseli ogłoszono, że European Laboratory for Artificial Intelligence (ELAI) wygrało konkurs Komisji Europejskiej na pilotaż wirtualnego instytutu w ramach unijnej inicjatywy RAISE (Resource for AI Science in Europe). Instytut Badawczy IDEAS, jako jedyna jednostka z Polski, jest jednym z 33 partnerów ELAI. W ELAI znalazło się osiem z dziesięciu najlepszych europejskich ośrodków badań nad AI według CSRankings (kategoria AI, 2016–2026): EPFL, ETH Zurich, Uniwersytet w Edynburgu, Uniwersytet Amsterdamski, Towarzystwo Maxa Plancka, TU Monachium, Cambridge i Oxford. Wśród partnerów są też m. in. Inria, CNRS, KU Leuven, Politechnika Mediolańska, czeska ČVUT i austriacki ISTA. Łącznie ELAI skupia 33 partnerów i 21 instytucji wspierających z 17 krajów, w tym z Japonii i Kanady. 🔹Na start KE przeznacza 17 mln euro na 5 lat. 🔹 Badania obejmą m. in. efektywność, bezpieczeństwo i ochronę systemów AI. 🔹 Projekt koordynuje Uniwersytet Kopenhaski, a ruszy 1 stycznia 2027. Dla polskiej nauki to ważne z dwóch powodów. Naukowcy z IDEAS i ELLIS Unit Warsaw zyskują dostęp do europejskiego finansowania i do sieci najlepszych badaczy. Do tego KE zapowiada rozbudowę RAISE w kolejnym budżecie UE, więc kto jest w programie od początku, ma szansę na kolejne środki. Gratulacje dla prof. Piotra Sankowskiego @piotrsankowski i prof. Tomasza Trzcińskiego @tomasztrzcinsk1 i całego zespołu IDEAS! 👏
24
64
10
467
15,685
Fabryka AI retweeted
imagine in 2 years when the only job is deploying capital and people just ask "how do we deploy capital" to fable-2028-09-26 and it tells everyone the same thing and then we get hyperconcentration bubble cycles on a weekly basis and everyone else lives off raw consumer surplus
36
14
11
497
22,070
Fabryka AI retweeted
Most RL post-training pipelines have a blind spot: Teams train LLMs on isolated atomic skills, assuming the model will automatically compose them on complex tasks. This NeurIPS '26 paper shows that doesn't work—and reveals a sharp asymmetry in how reasoning transfers. 🧵👇
🎉 Excited to share that our paper "Compositional Reasoning in Language Models under Reinforcement Learning Post-Training" has been accepted to NeurIPS 2026! 🎉 In this paper, we ask: Can LLMs learn individual reasoning skills and compose them to solve complex problems? (1/8)
4
8
116
6,051
Fabryka AI retweeted
RSI is going to make useful models incredibly small
4
2
66
1,353
hot take: nothing wrong with papers per se, but… here’s the recipe github.com/okhat/blog/blob/m…
i’m increasingly of the opinion that almost everyone agrees with this, but a big fraction of people aren’t comfortable saying it in public, because of what it implies for the bean counting mechanism the whole machine is built on (for awards/promo/hiring/etc).
11
70
5
804
51,767
1/ We still don’t have a simple picture of how even 2-layer MLPs learn features. In a new paper with @berkott3 and @learning_mech, we propose a simple, solvable model of shallow MLP training, and find that real MLPs roughly follow it. 🧵
4
24
2
259
15,407
Fabryka AI retweeted
most AGI pilled labs: - typesafe - general reasoning - general intuition - mit oasys - amp code
31
28
3
1,152
74,005
Fabryka AI retweeted
Great paper on why SFT can generalize just as well as RL if you give it enough diverse training data. Like everyone else since DeepSeek-R1 dropped, I’ve been treating "RL generalizes, SFT memorizes" as gospel. Whenever SFT runs plateaued on out-of-domain reasoning, everyone just blame the algorithm. Turns out, we might have been running an unfair comparison the entire time. In standard setups, RL gets to explore a huge rollout budget per prompt while SFT gets trained on a tiny slice of data. So the authors decided to actually level the playing field on Qwen2.5 and Qwen3 models by testing two things: mixing data sources instead of just distilling from one giant teacher, and scaling up the SFT data to match RL's rollout budget. The results completely flip the usual narrative. A carefully mixed dataset easily beats single-teacher distillation, and matching RL's rollout budget lets SFT generalize just as well as RL. And when you combine both, SFT actually outperforms it. To figure out why, they used LLM annotations to inspect the rollouts and proved (both empirically and theoretically on shallow transformers) that larger data budgets expose SFT to rare, "tail" solution methods. Those uncommon reasoning paths are where the actual generalization signal lives. Honestly, for everyone tired of debugging finicky online RL pipelines, this is a huge relief. SFT wasn't broken, we were just starving it of diverse rollouts. Full paper PDF: arxiv.org/pdf/2611.14912
38
113
18
907
68,229
Fabryka AI retweeted
noising solves everything as always
New paper w/ Yihe Dong, @DimitrisPapail & @ShivamGarg91462: Perturbing Inputs Improves Pretraining Distillation. In pretraining distillation, a small student LM learns to match a larger teacher’s next-token predictions on the training text. We change one thing: for the first part of training, both models read a perturbed version of the text (for example, with noise added to the token inputs), and the student matches the teacher’s predictions on that. Then we switch back to clean text. With the same steps and token budget, this beats standard distillation, and it also works well in data-constrained, multi-epoch settings. 🧵
5
40
2,113