How small can a strong retriever get? Today we release three answers: SauerkrautLM-MoEdernColBERT 1.5M, 27M and 115M, late-interaction MoE retrievers under Apache 2.0. The idea: take ModernBERT and replace the dense feed-forward block in every transformer layer with a mixture of experts. Mo(E)dernBERT. Why MoE for a retriever? Retrieval cost is dominated by encoding every token of every document. MoE decouples capacity from compute: the model stores more than it computes per token. More capacity for multilingual and domain-specific relevance, without paying for it across a million-document index. What's inside: 1/ MoE layers: a router picks 2 of 8 GeGLU experts per token, plus one shared expert. 115M: 67.4M of 114.6M parameters active. 2/ Hash embeddings as a factorization: a 192-value row per token id, projected to the hidden size. 115M: 5.96M instead of 30.7M embedding parameters. 3/ Above 513 tokens, a ±256-token sliding window: trained on 512 tokens, works up to 32,768. 4/ ColBERT head with 128-dim token vectors and MaxSim. A fused Triton MoE kernel ships with the models. How the three differ: → 1.5M: hidden 128, 2 transformer layers. 43,255 docs/s on one RTX 6000 Ada, 0.87 MB at 4 bit, 0.13 ms per query on an Apple M5 Max → 27M: hidden 640, 2 transformer layers. 6,708 docs/s, 13.8 MB at 4 bit → 115M: hidden 1,024, 6 gated short-convolution plus 4 transformer layers. LegalQuAD 0.851 vs. 0.772 for Nemotron-3-Embed-8B (public MTEB result), LongEmbed 90.1 All three were pretrained by us from random initialization: Stella-style embedding distillation, knowledge-distillation pre-training, fine-tuning. To our knowledge, these are the first ModernBERT-style MoE encoders with hash embeddings. More benchmark results and models here: huggingface.co/collections/V… #lateinteraction #ColBERT #MixtureOfExperts #ModernBERT #OpenSource