David Wang retweeted
Today we're announcing that Modal has crossed $1B in annualized run rate, as well as raised a $750M Series D at a $15.75B valuation, led by @Accel.
Since our Series C, usage has tripled and we’ve grown more than 15x year-over-year.
We are deeply grateful to the developers and companies building on Modal. There's still so much more to build, and we can't think of a better time to be building software.
I GET TO WORK WITH THIS GUY @nanjiangwill 🐐
Training large models with RL typically runs on a fixed homogeneous cluster of the latest accelerators for numerical stability and faster throughput. This throws away inference optimization work done for other vendors and generations, limits the capacity you can utilize, and is unnecessarily expensive.
So we trained a model on agentic coding tasks using fully asynchronous RL and mixed-precision, mixed-accelerator samplers. With off-policy correction, training stayed stable through convergence, on par with homogeneous samplers, at a lower cost.
David Wang retweeted
DFlash: Keep Drafting Parallel
My talk at @modal Runtime in SF on Oct 1. The drafter was the serial bottleneck, so DFlash made it parallel. That moved the bottleneck to selection, so DFlash 2 fixed that.
DFlash 2 is here! Qwen3.8-27B at 70 tok/s on an M5 Max MacBook Pro.
⚡ Up to 4.6× the speed of autoregressive decoding, with the same output.
This is the next generation of DFlash, seeded at Z Lab and upgraded at Inco AI. Get one more accepted token on every pass, for free!
inco.ai/blog/dflash2/
apparently I have 8 commits in sglang but I’ve been slacking since my last one was mid June
this is what the Mandate of Heaven looks like
DFlash 2 is here! Qwen3.8-27B at 70 tok/s on an M5 Max MacBook Pro.
⚡ Up to 4.6× the speed of autoregressive decoding, with the same output.
This is the next generation of DFlash, seeded at Z Lab and upgraded at Inco AI. Get one more accepted token on every pass, for free!
inco.ai/blog/dflash2/
modal 💚 sglang! grateful for our collaborations so far, and many more to come!
It was a pleasure to work through the nights together with @modal on the Kimi K3 support! Now it's time to try it out 😁
David Wang retweeted
DFlash to the moon(shot)!
David Wang retweeted
Frontier models require frontier speculators. Our team moved mountains to achieve 460 tok/s with our custom DFlash model.
Try it day 0 at modal.com/endpoints.
David Wang retweeted
Kimi K3 is live on Modal.
Moonshot has shipped the world's first open 3T-class model, and we're a day zero launch partner.
We trained a custom DFlash speculator for K3's novel architecture so you can run it faster, losslessly. The most capable open model we've worked with by far.
me n bro @_gongy
yes, working at @modal is as fun as it seems.
we are casting 35 more roles across new york, san francisco, and stockholm. one could be yours !
modal.jobs
David Wang retweeted
Inkling by @thinkymachines is now available on Modal, backed by a custom DFlash speculator for 67% higher throughput and interactivity.
Running on Modal Auto Endpoints with SGLang today.
Today, we are introducing Inkling.
Inkling reasons efficiently across text, image, and audio modalities. We are making the full weights available.
thinkingmachines.ai/news/int…
Available today for fine-tuning on Tinker. Play with it in the Inkling Playground. 🧵
As always, it’s a lot more nuanced than this. In spec decoding, the predominant cost is target verification, the chain vs tree tradeoff is well known, and at production concurrencies you’re often better off training a better draft model than doing tree drafting.
vLLM has even recently removed tree attention support github.com/vllm-project/vllm…
David Wang retweeted
Speculation Is All You Need.
In this blog post, we announce the co-release (w/ Z Lab) of six more state-of-the-art DFlash speculators for @Alibaba_Qwen 3.x.
Over 1k output tps for 3.5 122B-A10B on a B200.
Read the blog for why we're all-in on spec dec.
modal.com/blog/spec-is-all-u…
David Wang retweeted
Try them out! 🚀 Really enjoyed the amazing collaboration with the Modal team. The retrained DFlash draft models now support longer contexts, making them a better fit for agentic workloads.
Speculation Is All You Need.
In this blog post, we announce the co-release (w/ Z Lab) of six more state-of-the-art DFlash speculators for @Alibaba_Qwen 3.x.
Over 1k output tps for 3.5 122B-A10B on a B200.
Read the blog for why we're all-in on spec dec.
modal.com/blog/spec-is-all-u…
David Wang retweeted
🚀 New blog: The next generation of speculative decoding: DFlash and Spec V2
DFlash + Spec V2 hit >4.3X baseline throughput for LLM inference, now the default speculative decoding engine in SGLang! Together with @modal and z-lab.ai, our jointly-released DFlash drafter for Qwen 3.5 397B-A17B beats both baseline and native MTP in every setting we benchmarked:
1️⃣ >4.3X baseline & 1.5X native MTP throughput (concurrency 1, HumanEval, 8xB200)
2️⃣ Block diffusion drafter: a full token block in one forward pass
3️⃣ KV injection: target-model features fed into every draft layer’s KV cache for higher acceptance
4️⃣ Spec V2 overlap scheduler: +33% end-to-end
Read the code, deploy a DFlash server, and start experimenting!
David Wang retweeted
🚀 DFlash now runs on SGLang's new default speculative-decoding engine, Spec V2.
⚡️ Hitting >4.3× baseline throughput (1.5× over native MTP) on Qwen 3.5 397B-A17B. Same quality, more speed!
⭐ github.com/z-lab/dflash
