Paul Janson (@janson002) header
Paul Janson (@janson002) profile picture

Ph.D. Student @Mila_Quebec and Concordia University, Working in Deep learning optimization and efficiency. Previously RS Intern @Microsoft Research & @Kaust

Montréal, Québec
Joined September 2013
Photo by Paul Janson (@janson002)Photo by Paul Janson (@janson002)Photo by Paul Janson (@janson002)Photo by Paul Janson (@janson002)Photo by Paul Janson (@janson002)Photo by Paul Janson (@janson002)Photo by Paul Janson (@janson002)Photo by Paul Janson (@janson002)Photo by Paul Janson (@janson002)Photo by Paul Janson (@janson002)
Paul Janson retweeted
We suddenly became twice as interested in the "appendix"... 🙃 #ICLR2027
Replying to @TuhinChakr
2 out of 3 talk about appendix experiments, I don't even need Pangram to know no person would do that
20
37
11
701
45,921
Your neo-lab wants to pre-train a frontier model, but no provider will sell you enough co-located chips? We got you. Our new work shows that SOAP & SPlus achieve strong worker scaling to 96+ DiLoCo workers (e.g., 96 providers) by whitening the update’s covariance. 🧵1/9
7
26
7
100
8,419
RL post-training grades a language model on its finished answer. One reward, and every choice that led to it is pushed up or down together. That works when an answer is written left to right. For diffusion language models it falls short, and here is why. 🧵 1/N
1
3
1
18
1,034
Paul Janson retweeted
Mila’s annual supervision request process is open for MSc and PhD applications for Fall 2027 admission. While I won’t be taking on new students for supervision this cycle, I encourage you to apply to join Mila’s brilliant deep learning research community. For more information and guidelines, you can visit the link below or contact the @Mila_Quebec team directly. mila.quebec/en/prospective-s…
4
22
1
172
8,558
[Updated schedule] Don't miss our COLM workshop happening today (Oct 9th)! Starting with a talk from @Yoshua_Bengio at 9 am, we have a stacked lineup of speakers, including @_albertgu @lateinteraction @achowdhery @agarwl_ @EyubogluSabri ! Happening in Continental Ballroom 5
4
19
1,900
Paul Janson retweeted
Paper rejected from arxiv but accepted at neurips. Can’t make this stuff up
update: our arXiv rejection made it to NeurIPS!!!! hal.science/hal-05708820v1/d… “08/26: Thank you for submitting your work to arXiv. We regret to inform you that arXiv's moderators have determined that your submission will not be accepted and made public on arXiv.org.”
1
13
1,333
Paul Janson retweeted
We’ve been researching fault tolerance in Crucible, Templar’s pre-training platform. The goal: keep training through node failures and make better use of unreliable workers and spot instances, without idling an entire model replica when one stage goes down. 1/n
3
10
1
12
4,021
Call for Papers — CODEC-FM @ NeurIPS 2026 (Sydney) Frontier-scale training shouldn't require a hyperscaler's cluster. We're bringing together the open-model and decentralized-training communities to address the core technical challenges of this paradigm, extending foundation model training beyond a single datacenter and supporting more scalable, collaborative open-model development. :push_pin: Topics: communication-efficient & async optimization · heterogeneous & fault-tolerant systems · federated learning at scale · security in trustless settings · open model infra & governance · benchmarks :alarm_clock: Deadline: Aug 29, 2026 Details & CFP → collaborative-open-decentral… @SameeraRamasin1 @benjamintherien @Ana_koloskova @ebelilov @niclane #kaja #Aakanksha @Pluralis
10
4
24
5,963
Paul Janson retweeted
Pluralis-8B is done, the first pipeline-parallel training over WAN with our new Agora system. We maintained approximately 20% MFU throughout the run, with a completely dynamic pool of devices utilising networking speeds 100-1000x slower than a datacenter setup
6
17
3
111
11,681
Paul Janson retweeted
We’re excited to share our new work : “Stabilizing Native Low-Rank LLM Pretraining.” (arxiv.org/abs/2602.12429) 🚀 Can we train LLMs from scratch using only low-rank factorized weights and still match dense performance? Short answer: yes (with care).
4
43
6
207
34,780
Paul Janson retweeted
Training LLMs over low-bandwidth networks is hard but its necessary in the current landscape. In pipeline parallelism, every stage sends its activations to the next stage, and as models scale, this communication becomes the main bottleneck. Our new work 🍁MAPL asks: Can we cut this communication sharply without hurting model quality and token efficiency ? 🧵 1/N
1
15
1
22
1,649
Paul Janson retweeted
🎉 Our paper "𝗙𝗿𝗼𝗺 𝗠𝗲𝗺𝗼𝗿𝗶𝘇𝗮𝘁𝗶𝗼𝗻 𝘁𝗼 𝗣𝗮𝗿𝗮𝗺𝗲𝘁𝗲𝗿 𝗜𝗻𝘁𝗲𝗿𝗳𝗲𝗿𝗲𝗻𝗰𝗲: 𝗛𝗼𝘄 𝗢𝘃𝗲𝗿𝘁𝗿𝗮𝗶𝗻𝗶𝗻𝗴 𝗘𝘅𝗽𝗲𝗿𝘁𝘀 𝗛𝗮𝗿𝗺𝘀 𝗠𝗼𝗱𝗲𝗹 𝗠𝗲𝗿𝗴𝗶𝗻𝗴" was accepted at ICML 2026! 🔎 Do better expert models always lead to better merged models? Not necessarily! 📜Read the paper: arxiv.org/abs/2506.14126v2 🧵 1/9
1
12
1
35
7,279
Training big models gets painful once a full replica won't fit on one accelerator. You end up with model-parallel methods or techniques like FSDP that are communication-heavy and limited in how far they parallelize. We tried a new axis that lets you split the model the way model parallelism does, but communicate gradients instead of activations. 🧵 1/N
1
4
1
12
1,664
Paul Janson retweeted
🚨Excited to announce our workshop Context Beyond the Window hosted at COLM in SF! 🚨 LLMs have finite context windows, yet real-world tasks demand absorbing, retaining, and acting on information that far exceeds any single prompt. 1/3 We're looking for submissions across: context-beyond-window.github… • Context compression 🧃 — token compaction, recursive subagent calls, and external memory for storing and retrieving information • Efficient architectures 🚀 — sub-quadratic attention variants that make extremely long context computationally feasible • Continual training 🌱 — test-time training on streaming data, context distillation, and knowledge accumulation through continued pre-training • Agentic memory systems 🐘 — scaffolds and test-time scaling techniques that improve knowledge retention and acquisition in LLMs • Evaluation 🎯 — benchmarking models on increasingly long-horizon tasks
6
29
8
104
37,068
I’ll be at #MLSys2026 this week to present “PyLO: Towards Accessible Learned Optimizers in PyTorch”! Come listen to @janson002’s presentation today from 3:45PM – 4:00PM in room 2 or join us at poster 29. If you work on similar topics or just want to chat — DM me. 1/2
1
3
15
802
Paul Janson retweeted
This is like a good stress test for optimizers. Kaon is basically Muon/lmo + spectral noise. It preserves the singular vectors of the gradient and randomizes only the positive singular weights. For exchangeable noise, the conditional expectation is the spectral-norm-ball lmo direction up to scale. Individual draws are not necessarily lmos tho. Freon’s map for c>1/2 is decreasing on the singular values, so the operator is non-monotone. Exact fixed-step Freon can fail even on a simple convex quadratic minimization near rank deficiency. Freon’s map for c<=1/2 (i.e., the monotone case) can also be analyzed using phi-convexity. Shameless plug: arxiv.org/abs/2605.11850
Finally got you, damned lich king @tensorqt
12
72
11,055
Paul Janson retweeted
New paper! We analyze proximal preconditioned gradient methods that extend Muon/Scion to handle nonconvex constraints (Stiefel manifold, spectral sphere, norm balls, ...) with convergence guarantees under heavy-tailed noise + variance reduction w/ STORM! arxiv.org/abs/2605.11850
1
29
3
180
21,212
Paul Janson retweeted
Modded-NanoGPT optimization result #13: @benjamintherien has achieved a new record of 3210 steps (-15), by wrapping NorMuonH in a MuLoCo-style outer Nesterov SGD. Compared to the target loss, this result has a p-value of p=1.3e-4. Compared to result #11, it has p=0.099.
3
11
2
83
8,445
I’ll be at #ICLR2026 🇧🇷 this week to present “μLO: Compute-Efficient Meta-Generalization of Learned Optimizers” and give a talk about SparseLoCo at the Protocol Learning Workshop! If you work on these topics or just want to chat — DM me. 🧵1/3
1
4
38
2,138