Currently: Training Mellum models @jetbrains , Previously: PhD @dcucomputing, research @GoogleAI
∆
Joined May 2020
- Tweets741
- Following600
- Followers220
- Likes4.3K
Mohammed Sabry retweeted
One thing I hate about training frameworks (Megatron specifically but not limited to it) is it's annoying to run arch ablations on them. You have a very rigid structure that you need your model to fit in, like "Qwen3.5" or whatever else is implemented in it.
If you come up with something that doesn't fit into anything, you need to implement it practically everywhere: the framework itself, transformers, vllm/sglang. If only there would be a nice way to express an arbitrary arch and then "compile" it into the actual framework code.
I started writing my own "arch language" (and a training framework) for fun. Now, obviously, the hardest part is to actually compile it into the stuff I want and I'm seriously considering it shouldn't be hardcoded but be a fast agent that will quickly draft the model implementation from building blocks instead.
Mohammed Sabry retweeted
it's fast
Meet Mellum2.1 – JetBrains' open-source 12B MoE model built for coding agents and local execution.
🚀 140+ T/S in local testing scenarios
🇪🇺 Ethically trained European open model
💻 Efficient worker model inside IDE agent workflows
Try it locally: huggingface.co/collections/J…
Mohammed Sabry retweeted
Long-term AI adoption depends on driving down token costs. Mellum2.1 is our contribution to that shift – bringing an open-source 12B MoE model into local PyCharm workflows to deliver 140+ T/S without cloud overhead.
Learn more and see it in action:
Meet Mellum2.1 – JetBrains' open-source 12B MoE model built for coding agents and local execution.
🚀 140+ T/S in local testing scenarios
🇪🇺 Ethically trained European open model
💻 Efficient worker model inside IDE agent workflows
Try it locally: huggingface.co/collections/J…
Mohammed Sabry retweeted
Meet Mellum2.1 – JetBrains' open-source 12B MoE model built for coding agents and local execution.
🚀 140+ T/S in local testing scenarios
🇪🇺 Ethically trained European open model
💻 Efficient worker model inside IDE agent workflows
Try it locally: huggingface.co/collections/J…
Mohammed Sabry retweeted
We're doing a meetup during GTC Berlin at the JetBrains office in Berlin! It's a great chance to talk to people from Mellum and ML teams. We'll also have friends from NVIDIA joining.
Everyone's welcome!
Wednesday, 21 October.
Mohammed Sabry retweeted
We observed a lot of similar behaviours in Mellum2 pre-training run but it smoothed out in the lr decay stage
This needs a mechanistic look...the shorter recovery period points to a circuit accessibility problem....also neural nets develop many redundant circuits, which points more toward the circuits being "harder to activate" than being damaged
Mohammed Sabry retweeted
Today we're releasing Mellum2.1: a major update to the Mellum2 that we open-sourced in June. This release is the result of a significant scaling of our RL post-training to enable agentic coding workflows.
Mellum2.1 achieves top scores on the agentic coding benchmarks relative to its speed.
We're releasing the weights in both HF and GGUF formats, and MTP support is coming in the following days.
🤗 collection: huggingface.co/collections/J…
Blog post: blog.jetbrains.com/ai/2026/1…
ALT Mellum2.1 performance
Mohammed Sabry retweeted
Yesterday I got two useful pieces of feedback on the recirculation paper:
1. the training-free effect might grow with model size, as @xidulu suggested;
2. @qubitium also observed gains on Llama.
So I tested three ~30B models, together with my Dense and MoE models, on much larger MMLU/GSM8K test sets (14,042 / 1,319 examples) and added 95% bootstrap CIs to reduce evaluation noise.
Unfortunately, the picture is much weaker than I expected: most MMLU gains are below 1 pp (Qwen 27B is an outlier, explained in the thread below), all GSM8K confidence intervals cross zero, and I see no clear trend with model size.
The training-free recirculation story is looking less robust than I initially thought.
Recirculation is coool, but the evidence seems narrower.
PPL looks great: −23.0% adaptive, −8.5% training-free. Yet across 8 standard single-token benchmarks, every training-free change is <0.6pp, with 2 regressions. This looks like benchmark noise to me.
Table 2: arxiv.org/pdf/2608.17981#pag…
Mohammed Sabry retweeted
We will have an architecture that obsoletes the transformer within the next three years.
A good question from here is whether there is an optimal context length, and whether it follows a predictable scaling law (figure 2 and 9 do not hint at that in any way, which is reasonable given the contribution's angle)
arxiv.org/abs/2608.12218
During pretraining if long context gives enough information for tasks then the model could learn to utilize context instead of internalizing tasks. Maybe internalizing tasks could be suppressed and it could be useful to build a pretraining curriculum?
This feels like a new tension for the field: what capabilities should stay internal to the model, given that capacity is expensive, and what should be delegated to external modules.
The older version of this tension was on inductive bias, and it helped lead us toward Transformers with less built-in structural bias.
I want to add two points:
* We were developing this thread of work as my PhD topic under the terms external modularity (modules outside the model that collaborate with the model) and internal modularity (circuits responsible for built-in capabilities).
* The second point is that most people think of external modularity mostly in terms of tools (programmed modules), but it can involve neural modules too. A simple example is PEFT modules. hence one of my first works was developing a module-card so they could be included in such discussions: PEFT-Ref, arxiv.org/abs/2304.12410
I started this around two months ago, and I’d love to collaborate with the community on pushing it further.
My broader view is that, if we want more robust synthetic data pipelines, we should rely more on explicit rubrics and correctness-by-construction checks.
That is the motivation behind FORTGEN: a coverage-driven synthetic benchmark for Fortran modernization with executable semantic oracles.
GitHub: github.com/mhmdsabry/FORTG
Slides: docs.google.com/presentation…
Mohammed Sabry retweeted
In an era where many frontier labs are converging towards conservative incremental approaches, it's heartening to see resources being directed into ambitious efforts and fresh ideas!
Unveiling our new startup Advanced Machine Intelligence (AMI Labs).
We just completed our seed round: $1.03B / 890M€, one the largest seeds ever, probably the largest for a European company.
We're hiring!
[the background image is the Veil Nebula - a picture I took from my backyard, most appropriate for an unveiling]
More details here:
techcrunch.com/2026/03/09/ya…
The problem isn’t treating scaling curves like physics; it’s declaring regime changes without confidence intervals on the fit...
Mohammed Sabry retweeted
Continual learning has always been important, but people have not found effective ways. Tweaking loss objectives brings tons of ml papers a few year's back but cannot really achieve inf long context. Maybe we need more arch innovation like efficient attn and learnable memory.
Continual learning is the next big thing in AI.
Today’s deep networks show catastrophic forgetting where new data points cause old knowledge to get erased because backpropagation updates all weights in a network, causing old useful info to be overwritten.
Contrast this with brain - it doesn’t do backpropagation. no global updates, only local updates with each new data point.
The day we have a credible alternative to backpropagation is the day we will solve continual learning.
It’s an exciting time to be doing AI research!