Fabryka AI (@FabrykaAI) header
Fabryka AI (@FabrykaAI) profile picture

AI neolab focused on open source research

Warsaw
Joined February 2022
Photo by Fabryka AI (@FabrykaAI)Photo by Fabryka AI (@FabrykaAI)Photo by Fabryka AI (@FabrykaAI)Photo by Fabryka AI (@FabrykaAI)Photo by Fabryka AI (@FabrykaAI)Photo by Fabryka AI (@FabrykaAI)Photo by Fabryka AI (@FabrykaAI)Photo by Fabryka AI (@FabrykaAI)Photo by Fabryka AI (@FabrykaAI)Photo by Fabryka AI (@FabrykaAI)
Fabryka AI retweeted
if the eval is public, it's bad. all good evals are private
49
21
9
458
24,973
Fabryka AI retweeted
very happy for a diversity of mental lineages in machine intelligence. people claim open source adds diversity, but in practice they are just linear combinations of Claude and GPT mind styles
You will notice that Gemini Carbon thinks differently than Claude and GPT. It often comes up with entirely different but more clever solutions than competitor models. This is precisely because we never distilled other models. Distilling other models creates a poor imitation rather than unique intelligence.
93
56
24
1,423
63,492
Fabryka AI retweeted
for whatever amount of compute you have, you will get much more research done if you can just make your experiments smaller
23
24
3
605
18,875
Fabryka AI retweeted
frontier research discovering kaggle tricks from first principles
Could be the start of a trend in new pre-training recipes: no LR decay phase anymore. (Also seen in Kolibri.) Instead: constant LR + averaging the last N checkpoints (or a running weight average). I guess that makes it easier in practice because you don't have to commit to a total training length upfront. A little meta analysis based on those who share training details: - Classic cosine: Llama 3, OLMo 2, OLMo 3, GLM-4.5, GLM-5, Kimi K3, Phi-4 - WSD ("Warmup-Stable-Decay"): DeepSeek V3, DeepSeek V4, DeepSeek V4.1-Flash, Kimi K2, Kimi Linear, Nemotron 3 Nano, Nemotron 3 Super, SmolLM3, MiniMax M2 - No decay (constant LR plus checkpoint averaging): Kolibri-1, DatologyAI's models
1
9
1
279
13,199
Fabryka AI retweeted
wondering what the most efficient thing to do at any given time is an anxiety response. do the fun thing. your forager instincts are often superior to your farmer timetable reasoning at finding the long tail successes
47
211
26
2,572
216,719
Part of this is because we don't nerf Carbon for real world frontier research in coding, math, AI, science etc. The ability to push the frontier is open to all with Carbon :). Go forth and discover.
other than the fact that everything takes 20 minutes, gemini 4 carbon is now my favorite model. i use it for anything i want done thoroughly and correctly. it's brilliant, legible, and fixes its mistakes quickly and honestly if asked. it's like if astra max was normal
85
53
7
1,648
54,954
Fabryka AI retweeted
cool you're building a software factory, have you considered building a revenue factory
145
277
34
4,539
115,505
Fabryka AI retweeted
OpenAI is NanoGPT maxxing
OpenAI: "GPT-6.2 Astra is our first externally released model to meet the High threshold for AI Self-Improvement."
4
5
217
13,842
I wrote a blogpost (link below) about depth transfer with Attention Residuals (AttnRes), which was introduced by the @Kimi_Moonshot team and recently used in training the Inkling 2 Small model by @thinkymachines. AttnRes stabilizes depth scaling using an attention module across residual blocks. This ensures (to some extent) that while depth increases, the feature updates remain roughly order one in depth, which was the main idea behind our previous work Depth-muP and other related works such as CompleteP. To test depth transfer with AttnRes, I trained a GPT-style model with AttnRes (block size 4) on FineWeb, width 768, depth 6 to 32 (120M to 304M params), with no additional depth-dependent scaling (no multipliers etc). The optimal learning rate is stable across depth, confirming depth transfer. I give a theoretical argument for this in the blogpost.
1
10
1
124
4,377
Fabryka AI retweeted
Could be the start of a trend in new pre-training recipes: no LR decay phase anymore. (Also seen in Kolibri.) Instead: constant LR + averaging the last N checkpoints (or a running weight average). I guess that makes it easier in practice because you don't have to commit to a total training length upfront. A little meta analysis based on those who share training details: - Classic cosine: Llama 3, OLMo 2, OLMo 3, GLM-4.5, GLM-5, Kimi K3, Phi-4 - WSD ("Warmup-Stable-Decay"): DeepSeek V3, DeepSeek V4, DeepSeek V4.1-Flash, Kimi K2, Kimi Linear, Nemotron 3 Nano, Nemotron 3 Super, SmolLM3, MiniMax M2 - No decay (constant LR plus checkpoint averaging): Kolibri-1, DatologyAI's models
Replying to @datologyai
Trick #3: Merge, merge, merge. In pretraining, we skipped LR decay for a running weight average. Then we merged pre-context-extension models into post-extension checkpoints (see fig. for ablation study gains). Merging was also an expedient stand-in for MOPD in our RLVR recipe.
64
60
14
751
52,988
Looped transformers are becoming common knowledge now
we're getting loops for dinner SenseNova-Loop-9B is on par with Qwen 3.8-27B with a looped-architecture and mHC
1
137
Szanowni Państwo. Warto było tutaj wspierać polskie AI. Miło mi poinformować, że Instytut Badawczy IDEAS wszedł do europejskiego „superlaboratorium” AI 🇵🇱🇪🇺 Dziś na Apply AI Summit w Brukseli ogłoszono, że European Laboratory for Artificial Intelligence (ELAI) wygrało konkurs Komisji Europejskiej na pilotaż wirtualnego instytutu w ramach unijnej inicjatywy RAISE (Resource for AI Science in Europe). Instytut Badawczy IDEAS, jako jedyna jednostka z Polski, jest jednym z 33 partnerów ELAI. W ELAI znalazło się osiem z dziesięciu najlepszych europejskich ośrodków badań nad AI według CSRankings (kategoria AI, 2016–2026): EPFL, ETH Zurich, Uniwersytet w Edynburgu, Uniwersytet Amsterdamski, Towarzystwo Maxa Plancka, TU Monachium, Cambridge i Oxford. Wśród partnerów są też m. in. Inria, CNRS, KU Leuven, Politechnika Mediolańska, czeska ČVUT i austriacki ISTA. Łącznie ELAI skupia 33 partnerów i 21 instytucji wspierających z 17 krajów, w tym z Japonii i Kanady. 🔹Na start KE przeznacza 17 mln euro na 5 lat. 🔹 Badania obejmą m. in. efektywność, bezpieczeństwo i ochronę systemów AI. 🔹 Projekt koordynuje Uniwersytet Kopenhaski, a ruszy 1 stycznia 2027. Dla polskiej nauki to ważne z dwóch powodów. Naukowcy z IDEAS i ELLIS Unit Warsaw zyskują dostęp do europejskiego finansowania i do sieci najlepszych badaczy. Do tego KE zapowiada rozbudowę RAISE w kolejnym budżecie UE, więc kto jest w programie od początku, ma szansę na kolejne środki. Gratulacje dla prof. Piotra Sankowskiego @piotrsankowski i prof. Tomasza Trzcińskiego @tomasztrzcinsk1 i całego zespołu IDEAS! 👏
24
64
10
467
15,637
Fabryka AI retweeted
imagine in 2 years when the only job is deploying capital and people just ask "how do we deploy capital" to fable-2028-09-26 and it tells everyone the same thing and then we get hyperconcentration bubble cycles on a weekly basis and everyone else lives off raw consumer surplus
36
14
11
497
22,005
Fabryka AI retweeted
Most RL post-training pipelines have a blind spot: Teams train LLMs on isolated atomic skills, assuming the model will automatically compose them on complex tasks. This NeurIPS '26 paper shows that doesn't work—and reveals a sharp asymmetry in how reasoning transfers. 🧵👇
🎉 Excited to share that our paper "Compositional Reasoning in Language Models under Reinforcement Learning Post-Training" has been accepted to NeurIPS 2026! 🎉 In this paper, we ask: Can LLMs learn individual reasoning skills and compose them to solve complex problems? (1/8)
4
8
116
6,051
compute market is a joke currently ( no stability it seems ) - it looks like ex-crypto people are speculating on it or something
2
92
Fabryka AI retweeted
RSI is going to make useful models incredibly small
4
2
66
1,353
training 1B model for fun - wondering how fast I can do this
2
2
163
that's how I feel
14