krishna

Well under 1% of the common corpus used to train frontier language models has Indian-language representation, according to a published language distribution study. @fewshotlearner, Head of Vision Models at Sarvam, takes on that gap in "From Scratch to SOTA: Training a 3B State-Space Vision Model," which @aiDotEngineer published on YouTube. The talk walks through how a 3 billion parameter model, small enough to run on a single GPU, got to state of the art in document OCR and beat models 100 times its size, with the data, training, and compute all done in India. - State-space, not transformer. Krishna lays out the tradeoff: a transformer's attention cost grows with the square of sequence length, while a state-space model updates a single running state token by token, so compute scales linearly and memory stays constant, which matters when a page can produce 5,000 to 10,000 visual tokens. - A contrarian bet on block-level OCR. Where most VLMs in late 2025 did page-level OCR, Sarvam Vision does OCR at the block level instead, with two dedicated modules, one for layout and one for reading. - A four-stage curriculum. Text-only pretraining on 13 trillion tokens across English, Indian languages, math, and code builds the language backbone first. Then 300 million image-text pairs teach it to see, 100 million OCR samples take it through supervised fine-tuning, and a reinforcement learning stage (sample a group, score against unit tests, reinforce toward the average to beat) pushes past what fine-tuning alone reaches. - The numbers. At launch it scored 84.3 on the Olmo OCR Bench and 93.2 on Omnidoc Bench, and it holds an unbeaten lead across 22 Indian languages, ahead of frontier models including Gemini, ChatGPT, and Opus. - Already in production. The model powers Akshar, an agentic document intelligence workbench with confidence scores, block-level grounding, and agentic proofreading, and has been used to digitize more than 35 million pages across English and 22 Indian languages. I'm working through the published talks from AI Engineer World's Fair sharing summaries and takeaways. Follow for more!