very happy for a diversity of mental lineages in machine intelligence. people claim open source adds diversity, but in practice they are just linear combinations of Claude and GPT mind styles
You will notice that Gemini Carbon thinks differently than Claude and GPT. It often comes up with entirely different but more clever solutions than competitor models.
This is precisely because we never distilled other models. Distilling other models creates a poor imitation rather than unique intelligence.
Fabryka AI retweeted
for whatever amount of compute you have, you will get much more research done if you can just make your experiments smaller
frontier research discovering kaggle tricks from first principles
Could be the start of a trend in new pre-training recipes: no LR decay phase anymore. (Also seen in Kolibri.)
Instead: constant LR + averaging the last N checkpoints (or a running weight average).
I guess that makes it easier in practice because you don't have to commit to a total training length upfront.
A little meta analysis based on those who share training details:
- Classic cosine: Llama 3, OLMo 2, OLMo 3, GLM-4.5, GLM-5, Kimi K3, Phi-4
- WSD ("Warmup-Stable-Decay"): DeepSeek V3, DeepSeek V4, DeepSeek V4.1-Flash, Kimi K2, Kimi Linear, Nemotron 3 Nano, Nemotron 3 Super, SmolLM3, MiniMax M2
- No decay (constant LR plus checkpoint averaging): Kolibri-1, DatologyAI's models
Fabryka AI retweeted
wondering what the most efficient thing to do at any given time is an anxiety response. do the fun thing. your forager instincts are often superior to your farmer timetable reasoning at finding the long tail successes
Fabryka AI retweeted
Part of this is because we don't nerf Carbon for real world frontier research in coding, math, AI, science etc.
The ability to push the frontier is open to all with Carbon :). Go forth and discover.
Fabryka AI retweeted
I wrote a blogpost (link below) about depth transfer with Attention Residuals (AttnRes), which was introduced by the @Kimi_Moonshot team and recently used in training the Inkling 2 Small model by @thinkymachines.
AttnRes stabilizes depth scaling using an attention module across residual blocks. This ensures (to some extent) that while depth increases, the feature updates remain roughly order one in depth, which was the main idea behind our previous work Depth-muP and other related works such as CompleteP.
To test depth transfer with AttnRes, I trained a GPT-style model with AttnRes (block size 4) on FineWeb, width 768, depth 6 to 32 (120M to 304M params), with no additional depth-dependent scaling (no multipliers etc). The optimal learning rate is stable across depth, confirming depth transfer. I give a theoretical argument for this in the blogpost.
Fabryka AI retweeted
Could be the start of a trend in new pre-training recipes: no LR decay phase anymore. (Also seen in Kolibri.)
Instead: constant LR + averaging the last N checkpoints (or a running weight average).
I guess that makes it easier in practice because you don't have to commit to a total training length upfront.
A little meta analysis based on those who share training details:
- Classic cosine: Llama 3, OLMo 2, OLMo 3, GLM-4.5, GLM-5, Kimi K3, Phi-4
- WSD ("Warmup-Stable-Decay"): DeepSeek V3, DeepSeek V4, DeepSeek V4.1-Flash, Kimi K2, Kimi Linear, Nemotron 3 Nano, Nemotron 3 Super, SmolLM3, MiniMax M2
- No decay (constant LR plus checkpoint averaging): Kolibri-1, DatologyAI's models
Replying to @datologyai
Trick #3: Merge, merge, merge.
In pretraining, we skipped LR decay for a running weight average.
Then we merged pre-context-extension models into post-extension checkpoints (see fig. for ablation study gains).
Merging was also an expedient stand-in for MOPD in our RLVR recipe.
Fabryka AI retweeted
Szanowni Państwo. Warto było tutaj wspierać polskie AI. Miło mi poinformować, że Instytut Badawczy IDEAS wszedł do europejskiego „superlaboratorium” AI 🇵🇱🇪🇺
Dziś na Apply AI Summit w Brukseli ogłoszono, że European Laboratory for Artificial Intelligence (ELAI) wygrało konkurs Komisji Europejskiej na pilotaż wirtualnego instytutu w ramach unijnej inicjatywy RAISE (Resource for AI Science in Europe).
Instytut Badawczy IDEAS, jako jedyna jednostka z Polski, jest jednym z 33 partnerów ELAI.
W ELAI znalazło się osiem z dziesięciu najlepszych europejskich ośrodków badań nad AI według CSRankings (kategoria AI, 2016–2026): EPFL, ETH Zurich, Uniwersytet w Edynburgu, Uniwersytet Amsterdamski, Towarzystwo Maxa Plancka, TU Monachium, Cambridge i Oxford.
Wśród partnerów są też m. in. Inria, CNRS, KU Leuven, Politechnika Mediolańska, czeska ČVUT i austriacki ISTA. Łącznie ELAI skupia 33 partnerów i 21 instytucji wspierających z 17 krajów, w tym z Japonii i Kanady.
🔹Na start KE przeznacza 17 mln euro na 5 lat.
🔹 Badania obejmą m. in. efektywność, bezpieczeństwo i ochronę systemów AI.
🔹 Projekt koordynuje Uniwersytet Kopenhaski, a ruszy 1 stycznia 2027.
Dla polskiej nauki to ważne z dwóch powodów.
Naukowcy z IDEAS i ELLIS Unit Warsaw zyskują dostęp do europejskiego finansowania i do sieci najlepszych badaczy.
Do tego KE zapowiada rozbudowę RAISE w kolejnym budżecie UE, więc kto jest w programie od początku, ma szansę na kolejne środki.
Gratulacje dla prof. Piotra Sankowskiego @piotrsankowski i prof. Tomasza Trzcińskiego @tomasztrzcinsk1 i całego zespołu IDEAS! 👏
Fabryka AI retweeted
imagine in 2 years when the only job is deploying capital and people just ask "how do we deploy capital" to fable-2028-09-26 and it tells everyone the same thing and then we get hyperconcentration bubble cycles on a weekly basis and everyone else lives off raw consumer surplus
Fabryka AI retweeted
Most RL post-training pipelines have a blind spot:
Teams train LLMs on isolated atomic skills, assuming the model will automatically compose them on complex tasks.
This NeurIPS '26 paper shows that doesn't work—and reveals a sharp asymmetry in how reasoning transfers. 🧵👇
compute market is a joke currently ( no stability it seems ) - it looks like ex-crypto people are speculating on it or something
2
92

































