Abhinandan retweeted
hi everyone,
i am an inference engineer, proactively looking for the right team to join.
Things i have done
+ contributed to vLLM, mooncake, dynamo and more
+ have 2.5+ yoe building at startups
+ received direct offer from yc-backed startup through oss
Can specifically help with
+ KV
+ RL rollouts &
+ CUDA kernels
Currently
+ deepening, making more & more contributions to @vllm_project
+ spending ~$25 on Hoppers & Amperes every week
here is my work - abhinandan.one
resume screenshot is attached.
lmk if you/your team/known is looking for a similar person. pls DM me.
if u have read so much, pls like, repost, quote & cfbr ๐.
will be grateful for ur support.
Abhinandan retweeted
EAGLE-4 is here.
From EAGLE-1 to EAGLE-3, weโve kept making inference faster.
EAGLE-4 is our fastest yet.
961.6 tokens/s on @Alibaba_Qwenโs Qwen3.8-27B with TokenSpeed (BF16, 4ร @NVIDIA B300, TP4, one request, T=0).
Github: github.com/eagle-hq/EAGLE-4
Abhinandan retweeted
deepseek has given a lot
+ the first* oss reasoning model (the viral launch of R1, jan 2025)
+ MLA
+ DSA
+ MTP (spec-decoding)
+ FlashMLA
+ DeepGEMM
+ DeepEP
+ EPLB
+ 3FS
+ @DeepSeekHarness
+ DeepInfer
and its all open-source.
thank u so much @deepseek_ai for this all ๐
๐ Introducing DeepInfer, our in-house inference framework, now fully open source!
Powering DeepSeek-V4.1-Flash at scale with high throughput and low latency. Weโre sharing our production stack to help the community lower compute costs, reduce energy use, and make AI more affordable.
๐ github.com/deepseek-ai/DeepIโฆ
1/4
Abhinandan retweeted
๐ Introducing DeepInfer, our in-house inference framework, now fully open source!
Powering DeepSeek-V4.1-Flash at scale with high throughput and low latency. Weโre sharing our production stack to help the community lower compute costs, reduce energy use, and make AI more affordable.
๐ github.com/deepseek-ai/DeepIโฆ
1/4
Abhinandan retweeted
Today, Dream Labs is coming out of stealth with Dream-1, a robot foundation model that controls bimanual robots it has never seen, on tasks it was never shown, in places it has never been, from $3/hour.
Dream API is live for everyone, today. ๐งต
Abhinandan retweeted
Spent years watching models ace a single image and completely whiff on anything that moves or changes angle. Today weโre fixing that.
Try Glimpse now, our first visual thinking model with competitive frontier performance compared to Astra on key public visual reasoning benchmarks.
Abhinandan retweeted
Attention is the first step in understanding the models.
But generally, it is not easy to understand it in one definition or directly jumping to QKV & softmax.
So I tried to make it easy and simple to understand it from the first principles intuition.
Abhinandan retweeted
oss update:
+ 1 PR merged to @nvidia 's dynamo
+ 1 PR merged to @vllm_project /vllm-ascend
more on the go.... ๐
oss update:
+ 7 PRs merged to @vllm_project /semantic-router
+ 1 PR merged to @KVCache_AI /Mooncake
Abhinandan retweeted
oss update:
+ 7 PRs merged to @vllm_project /semantic-router
+ 1 PR merged to @KVCache_AI /Mooncake
Abhinandan retweeted
i am an inference engineering unit, aggressively looking for a relevant great team to join. i have
+ been digging deep into transformers, scheduling, caching, quantization every new day
+ been continuously contributing to @vllm_project , @nvidia's TensorRT-LLM, dynamo, @KVCache_AI 's mooncake, @sgl_project , @_llm_d_ and more
+ built projects on scheduling, batching, KV & prefix caching, quantization, HiCache, RL rollouts
+ paying 20+ dollars to @runpod almost every week
besides
+ been founding engineer at fast-paced ai startups, overall 2.5+ yoe
+ received direct offer from a yc-backed startup through open-source contributions
+ been recognised as top 1% typeScript engineer globally on Algora
and
> have built cdp-native browser-use agents
> auto demo video generation system using @Remotion
> proactive DevRel system for content/docs
> process graphs for hr automation workflows
plus
^ i work 12+ hours almost everyday
^ generally 12pm to 4am
^ i work in silent room with no headphone
open to
> remote
> relocation
> onsite
> give you provide enough value
no
- girlfriend
- parties
- smartphone distraction( avg 12 mins/day)
โ here is my work - abhinandan.one
โ github - githhub.com/awesome-pro
if you feel that I can create a worth value for your team - lmk, i am available. DMs are open
or if someone you know might need me, please refer - will be highly grateful
and if you have read so long, please like, quote/repost, cfbr - your small help may give me reach my next team.
finally thanks for reading.
Abhinandan retweeted
got my first contribution merged to @vllm_project
so happy - nothing to say more.
Abhinandan retweeted
Reasoning from scratch, round number 5! This time, talking about log-probability scoring (also a great fundamental concept for loss functions like cross-entropy in pre-training and distillation) and self-refinement.
00:00 Introduction and inference-time scaling recap
05:02 Loading the pretrained LLM
08:00 Comparing and scoring model answers
10:18 Building a rule-based scorer
17:53 Token probabilities and sequence likelihood
26:47 Computing token probabilities in PyTorch
30:12 Token indexing and shifted targets
37:27 Log probabilities and numerical stability
45:57 Scoring answers with average log probabilities
56:24 How self-refinement works
59:07 Generating critiques and revised answers
1:01:00 Implementing the self-refinement loop
1:05:57 MATH-500 evaluation results
1:07:35 Takeaways and next steps
Abhinandan retweeted
Wake up, ppl ๐
Xiaomi just open-sourced ~7K of the RL environments it used to train MiMo on HuggingFace, across domains like code, cyber, general, music and web dev
I think this is one of the biggest things to happen in frontier open-source RL environment data.
Indepth analysis and learnings coming soon ๐
Abhinandan retweeted
here is an overview of the 4 most advanced efficient architectures: Deepseek V4.1 Flash, MiMo V3, Qwen 3.8 Next Flash and GLM 5.3 Flash
Deepseek and MiMo are quite similar, they both use YOCO - only the first part of the network is active during prefill to build the KV cache representation - no linear attention, and token level indexer
Qwen and GLM both use a more standard interleaving 3:1 (like Kimi K3 as well) between sparse attention and linear attention (GDN vs KDA)
both deepseek and qwen use Engram, they all use a gate or sink except GLM 5.3 Flash, they also all use no or partial RoPE on the full/sparse attention layers
they also all have some sophisticated residual network, either mHC (simplified or not) or gated residual and they are all trained using Muon
visualization by opus 5.5 and me :)
Abhinandan retweeted
Why do LLMs not generate same response every time, for the exact same prompt provided.
The reason is sampling.
Sampling makes the output generation probablistic unlike the Greedy generation.
Temperature determines the entropy or creativity of the output tokens. While topK and topP control the selectivity of the tokens.
So in this video I have explained how does the sampling based generation work. How the temperature, topK & topP matter in this generation of output.
Everything from first principles in easy to understand way.
This is the Part 2 of Building an LLM Inference Engine from Scratch - Sampling Inference Generation
Complete Video & Logic Code in first reply
What happens between your prompt and the tokens you see on screen?
๐๐ป ๐๐๐ ๐ฑ๐ผ๐ฒ๐๐ป'๐ ๐ด๐ฒ๐ป๐ฒ๐ฟ๐ฎ๐๐ฒ ๐ฎ๐ป ๐ฎ๐ป๐๐๐ฒ๐ฟ ๐ฎ๐น๐น ๐ฎ๐ ๐ผ๐ป๐ฐ๐ฒ. It predicts one token, appends that token to the sequence, runs the model again, predicts the next token. And keeps repeating this process.
Most of us know this concept, but I wanted to understand what the inference loop actually looks like I build it myself.
So I have started building a mini LLM inference engine from scratch.
In this Part 1, I implemented greedy autoregressive generation:
input_ids [B,T]
โ model forward pass
โ logits [B,T,V]
โ take the last position [B,V]
โ choose the highest-scoring token
โ append it
โ repeat
Building such a generation loop, makes you to properly understand
- what logits represent,
- why the output has shape [B,T,V],
- why generation only uses the last position,
- and how the model's own prediction becomes the input for the next step.
All this question I have tried to answer in this video in easy to understand way.
This is the first part of a series Building Inference Engine where I'm progressively adding:
๐๐ฎ๐บ๐ฝ๐น๐ถ๐ป๐ด โ ๐๐ฉ ๐ฐ๐ฎ๐ฐ๐ต๐ฒ โ ๐ฏ๐ฎ๐๐ฐ๐ต๐ถ๐ป๐ด โ ๐ฝ๐ฎ๐ฑ๐ฑ๐ถ๐ป๐ด/๐บ๐ฎ๐๐ธ๐ โ ๐ง๐ง๐๐ง/๐๐ง๐

























