Martin Müller (@marmuel_) header
Martin Müller (@marmuel_) profile picture

Posttraining @poolsideai

Zurich, Switzerland
Joined June 2016
Photo by Martin Müller (@marmuel_)Photo by Martin Müller (@marmuel_)Photo by Martin Müller (@marmuel_)Photo by Martin Müller (@marmuel_)Photo by Martin Müller (@marmuel_)Photo by Martin Müller (@marmuel_)Photo by Martin Müller (@marmuel_)Photo by Martin Müller (@marmuel_)Photo by Martin Müller (@marmuel_)Photo by Martin Müller (@marmuel_)
Martin Müller retweeted
This is the thing with internal tools: over time, they almost always become subpar vs a tool a vendor builds At Large Company that have their own, custom o11y, talked with an eng... who said how much they'd love to use Datadog (or similar) because that would "just" work instead of "kind of" work
We replaced a SaaS product with an internally built tool. Anyone from the company can improve it by submitting code. I give it 2 more months until it is an unusable mess thanks to all the “improvements” people are making.
72
27
14
761
108,756
Martin Müller retweeted
Today we’re publishing the technical report behind Laguna M.1 and Laguna XS.2. This report opens up more of what went into them: Model Factory, pre-training data, distributed training, post-training, agent RL, quantization, and evaluation. poolside.ai/assets/laguna/la…
15
87
34
433
335,471
A ridiculous amount of work has gone into this from pretraining, synthetic data, long-horizon/async RL, and product. Super happy to see it all come together 🏭🚀
Today we’re releasing Laguna XS.2, Poolside’s first open-weight model. It’s a 33B total / 3B active MoE model built for agentic coding and long-horizon tasks. Trained fully in-house on our own stack. Runs on a single GPU. Released under Apache 2.0. Links 👇 Weights: huggingface.co/poolside/Lagu… API: platform.poolside.ai/ Blog: poolside.ai/blog/laguna-a-de…
7
135
Martin Müller retweeted
We believe that to compete at the frontier, you have to own the full stack: from dirt to intelligence. Today we’re announcing two major unlocks for our mission to AGI: 1. We're partnering with @CoreWeave and have 40,000+ NVIDIA GB300s secured. First capacity comes online starting Dec ’25. 2. Project Horizon: Poolside is developing a vertically integrated 2GW AI campus in West Texas to secure our medium term scale. On this site @CoreWeave will be our anchor tenant for the first 250MW phase. Learn more about it on our blog: poolside.ai/blog/announcing-… Or on WSJ: wsj.com/tech/ai/a-giant-new-…
34
49
34
414
457,389
how i feel about vibe coding
Modern medicine is a gift.
2
1
363
New blog post on how modularity evolves under changing environments, with some cool simulations and a repo to play around with 🤓. Deep learning models optimize for static objectives, get very very good at a very particular problem, by getting stuck in very successful but uninterpretable local minima. Biological systems evolve to solve problems under constantly changing environments - creating inherently modular, adaptable solutions. Based on the work on Nadav Kashtan and ​​@UriAlonWeizmann​, it is possible to simulate a caricature version of this phenomenon, and show how logic circuits naturally evolve modularity when adapting to modularly varying goals. This emergent modularity not only makes the system more interpretable but also potentially speeds up evolution toward higher complexity. Full blog post here: smallscaled.com/2025/02/happ… Github repo here: github.com/burcutepekule/mod… Enjoy! 🤓
1
4
386
Martin Müller retweeted
⚡️ Excited to share that I am starting an AI+Education company called Eureka Labs. The announcement: --- We are Eureka Labs and we are building a new kind of school that is AI native. How can we approach an ideal experience for learning something new? For example, in the case of physics one could imagine working through very high quality course materials together with Feynman, who is there to guide you every step of the way. Unfortunately, subject matter experts who are deeply passionate, great at teaching, infinitely patient and fluent in all of the world's languages are also very scarce and cannot personally tutor all 8 billion of us on demand. However, with recent progress in generative AI, this learning experience feels tractable. The teacher still designs the course materials, but they are supported, leveraged and scaled with an AI Teaching Assistant who is optimized to help guide the students through them. This Teacher + AI symbiosis could run an entire curriculum of courses on a common platform. If we are successful, it will be easy for anyone to learn anything, expanding education in both reach (a large number of people learning something) and extent (any one person learning a large amount of subjects, beyond what may be possible today unassisted). Our first product will be the world's obviously best AI course, LLM101n. This is an undergraduate-level class that guides the student through training their own AI, very similar to a smaller version of the AI Teaching Assistant itself. The course materials will be available online, but we also plan to run both digital and physical cohorts of people going through it together. Today, we are heads down building LLM101n, but we look forward to a future where AI is a key technology for increasing human potential. What would you like to learn? --- @EurekaLabsAI is the culmination of my passion in both AI and education over ~2 decades. My interest in education took me from YouTube tutorials on Rubik's cubes to starting CS231n at Stanford, to my more recent Zero-to-Hero AI series. While my work in AI took me from academic research at Stanford to real-world products at Tesla and AGI research at OpenAI. All of my work combining the two so far has only been part-time, as side quests to my "real job", so I am quite excited to dive in and build something great, professionally and full-time. It's still early days but I wanted to announce the company so that I can build publicly instead of keeping a secret that isn't. Outbound links with a bit more info in the reply!
1,494
3,553
1,073
27,451
2,552,943
Exciting news!🤓Our paper on the ontogeny of immune tolerance with @CJEMetcalf and @AiingLim is out as a preprint! We mathematically model how the gut microbiome and adaptive immunity interact in early life to establish immune and microbial homeostasis: doi.org/10.1101/2024.05.20.5…
1
6
8
1,442
Martin Müller retweeted
Super excited to be releasing AlphaFold 3 today, developed by @IsomorphicLabs and @GoogleDeepMind: our next generation AI model for predicting the biomolecular structures and interactions of proteins, DNA, RNA, small molecules, and more: bit.ly/44yfaCw 1/
25
262
47
1,108
215,496
Martin Müller retweeted
This take on the FineWeb release is one of the most interesting feedback and also a reason FineWeb is very different from even larger datasets like RedPajama-V2 (which is double its size!) Surprisingly, the size of the dataset of 15T tokens is not very important, what is much more important is why we spend ~120k GPU hours on the H100 cluster to prepare/share a ... dataset? Let's take a moment to dive in it! First, where can you get data at scale for web-scale llm pretraining? Well we're all lucky that Common Crawl is there and has been doing in the open a large chunk of the work of crawling/archiving the web for years, work that otherwise only private teams like Google/Bing would have access to. Next question: can you just train directly on the petabytes common crawl corpus, maybe just extract the text from the html pages and train on it, the model will figure it out? You would have one of the largest possible dataset for sure! The answer me and a part of the Mistral team who participated in BigScience/Bloom training learned over time is: No! Like Sergey says below, you actually want a dataset which is both large but also high quality What is high quality for a web-scale LLM pretraining dataset and how do you know you have it? Maybe I should then just train on, like, wikipedia only! The answer to this is also no. First because wikipedia is too small ofc but more important because our intuitive notion of data quality is not always reflected in the performance of the models and some aspects of data filtering can be counter-intuitive. Before I dive more in this let me give you an example of unintuitive behavior. Between 2022 and 2023 the "LLM quality" of Common Crawl dropped significantly as in "training a LLM on the crawls btw 2022-2023 will give you lower performances on a set of evals". What happened? Well it turns out the Common Crawl team has been filtering more strongly domains with adult content. Not really the cause you'd be intuitively thinking about, right? So how do you know you have good quality data? Well the simple, kinda circular, answer is: you just train on it. You train smaller models so it's not (too) expensive but still models that are big enough and on sensitive enough evaluations to give you signal about the quality of a larger model trained on the same dataset: this is what we call ablation models in FineWeb. Which ablation models did we used? We settled on two ways of training ablation models: in the first option we trained a 1.8B parameters model on 28B tokens which took about 5h on 64 H100. In the second option we trained the same 1.8B params models but much longer, for 350B tokens (which took about 2.5 days on 64 H100). Note that in this second, larger, ablations we trained on more tokens than GPT3 or Bloom were trained for instance. For each of many options for the filters we explored (heuristics and ML models), we trained some of these ablation models and compared the performance of the models to see if we saw an improvement or regression. Overall we trained about 200 small ablation and 15 larger ones for a total of more than 120k GPU hours. You can see the results of these evaluations for instance in the performance plots we include in the dataset card where we train on 350B tokens (160k steps). Overall this is the main difference with a simple raw Common Crawl and RedPajama-V2. In these later case you still need to do the work of selecting how to filter the data yourself and it's the work we wanted to provide the community with in FineWeb. At least a first version of it since we see FineWeb as a ressource we will improve over time along various direction. Even better filtering for sure and also multilinguality which is very interesting as well. To finish, an important parallel work I really need to mention is the work of the Dolma team which have been working tirelessly on improving the filtering of Dolma and, in our on-going training we see that the very latest release of Dolma from last week is very promising. We'll include these numbers in the coming blog post that we are currently writing and we are super happy about the work of the AllenAI team. Excited to dive in your dataset as well, friends! Stay tuned for the blog post detailing all I mentioned above and other anecdotes in greater details! Cheers
People seem to over-index on the 15T number after Llama 3. While the number matters, what is even more important is the quality and diversity of those tokens. If there was a good way to measure those, that would have been an impressive result to report.
15
128
31
800
328,218
We're looking for a backend engineer for @neuralframes. Needs to be a good python developer and have experience with GPU stuff / stable diffusion. Has a unique opportunity to shape AI music video generation for years to come. DMs are open. Thanks 🙏
14
11
2
28
2,012
Martin Müller retweeted
magnet:?xt=urn:btih:5546272da9065eddeb6fcd7ffddeef5b75be79a7&dn=mixtral-8x7b-32kseqlen&tr=udp%3A%2F%2Fopentracker.i2p.rocks%3A6969%2Fannounce&tr=http%3A%2F%https://teapawt.net/t.co/g0m9cEUz0T%3A80%2Fannounce RELEASE a6bbd9affe0c2725c1b7410d66833e24
503
1,490
961
9,701
4,637,946
Martin Müller retweeted
I created my first art style (XenMon) 🥳 I am heavily influenced by the Half-Life Xen Level because I am playing it right now 😋 Thanks to dreamlook.ai ❤ for the GPU credits 🔥 📦model: huggingface.co/camenduru/xen… 🦒colab: please try it 🐣 github.com/camenduru/xenmon-…
Let the fine-tuning begin 🏭🏭
3
8
1
52
8,063
If you ever wanted to try out Stable Diffusion training - now is the time 🤗 We've been working on dreamlook.ai since the beginning of the year, leveraging TPUs to offer very fast and affordable training The result: faster than on A100 & insanely scalable ⚡️
🔥 𝗧𝗿𝗮𝗶𝗻 𝟮𝟬 𝗦𝘁𝗮𝗯𝗹𝗲 𝗗𝗶𝗳𝗳𝘂𝘀𝗶𝗼𝗻 𝗺𝗼𝗱𝗲𝗹𝘀 𝗳𝗼𝗿 𝗳𝗿𝗲𝗲 🧑‍🔬 All our users can experiment freely until Sunday 🛠️ Train SD1.5 and SDXL models 🖼️ Generate high-res pics with ControlNet Available until Sunday midnight PST! Link below ⬇️
2
5
869
Let the fine-tuning begin 🏭🏭
🔥 𝗧𝗿𝗮𝗶𝗻 𝟮𝟬 𝗦𝘁𝗮𝗯𝗹𝗲 𝗗𝗶𝗳𝗳𝘂𝘀𝗶𝗼𝗻 𝗺𝗼𝗱𝗲𝗹𝘀 𝗳𝗼𝗿 𝗳𝗿𝗲𝗲 🧑‍🔬 All our users can experiment freely until Sunday 🛠️ Train SD1.5 and SDXL models 🖼️ Generate high-res pics with ControlNet Available until Sunday midnight PST! Link below ⬇️
1
4
7,480
Martin Müller retweeted
The cat is out of the bag! 🙀 TPUs are AMAZING at running Stable Diffusion 🚀 That's what we've been doing from the start at @dreamlookai to get crazy fast training times ⚡️
Super fast SDXL on TPU v5e with JAX! ⛓️ JAX works great and is super easy for parallel computation ➕ 💵 TPU v5e is very cost efficient ➕ 🧨 Diffusers supports JAX for SDXL Read the details or play with the demo in our blog post! huggingface.co/spaces/google…
2
2
21
2,783
Cool to see this out! Happy to have contributed a bit to it as well :) We've been working & optimizing this stack for the last 9 months at @dreamlookai and are now training full SD1.5 1200 steps in under 2min30s😛
Super fast SDXL on TPU v5e with JAX! ⛓️ JAX works great and is super easy for parallel computation ➕ 💵 TPU v5e is very cost efficient ➕ 🧨 Diffusers supports JAX for SDXL Read the details or play with the demo in our blog post! huggingface.co/spaces/google…
1
1
5
665
🔊 Juggernaut XL v2 model is out! 🏭 Trained on @dreamlookai 🤩 One of the most powerful SDXL model so far civitai.com/models/133005?mo…
2
4
22
1,811
We're starting to get SDXL training under control 😻 That's full UNet + Text Encoder training, not LoRA 💪 Coming verrry soon to @dreamlookai
3
3
1
18
1,974