The 867 number is PP4 on GLM 5.3 also? So just x4 vs x16 is the difference there? Seems like it’s very model-dependent. I’ve been working on GLM 5.3 Flash on an EXL3 quant and it’s pretty challenging. ~750 tok/s prefill PP2 after some optimizations, x4.

Do it

No it’s not! We’re getting 6-7k tok/s prefill PP2 on Qwen Flash Next W4A16. EXL3 is slower but up to 4k now - better than 3090s.

Looking great! Did you have to make any breakthroughs with EXL3 for this one?

Join us. True 64GB Ampere class NVIDIA cards. PCIE 2.0 x4 link sounds limiting but we’re getting great speeds with pipeline parallelism anyways. 6,000 tok/s prefill and 200+ tok/s decode is pretty nice. github.com/AlexBrandes/cmp17… Or you can mod them to x16 and do tensor parallelism. Suggest checking out @Morrowmake’s work on that.

You’re a bit ahead of me but I’m still throwing some compute at EXL3. Basically at half the prefill speed of W4A16, like you said. I’m hoping there’s more juice to squeeze though still.

New Qwen models for the people 🙌

Disclosed vulnerabilities are already starting to slope towards infinity. Not really the singularity I had hoped for, to be honest.

Not sure what we’re talking about but these speeds are unacceptable. Back to the lab until we get over 40 tok/s. You got this 🤘

Did you fill out the form on pacaswap.com? If so and you haven’t heard back, send me a DM and I’ll make sure we get it sorted out for you.

I wonder if you could use some of the tricks that cmpunlocker uses on CMP 170hx mining cards to achieve PCIE 2.0. You might even do ok with pipeline parallelism on 1.0 since you’d be doing just a few layers per card. github.com/amoghmunikote/cmp…

Yo, WTF?

That makes sense. I was playing around with developing some EXL3 kernels last week focused on 27b initially but the implementation is still pretty rough. Maybe I'll just adopt yours and try to add optimizations. I'll make a PR if I find anything worth sharing.

Nice! We need a more active community branch for CMP 170hx. I’ll see what I can contribute. On your Flash Next implementation, do you see lots of remaining upside with TP2? I’m seeing great results with PP2, like prefill speeds over 6,000 tok/s and decode speeds over 200 tok/s at c=1. I wonder if PP is actually more optimal given the PCIE constraints. github.com/AlexBrandes/cmp17…

@seanphan Let me know if you’d be open to a PR on your repo. I believe this is the fastest public recipe for Qwen Flash Next on 2x 170hxs.

New recipe from the home lab on 2x CMP 170HXs 👨‍🍳 Qwen 3.8 Flash Next Single stream: up to 226 tok/s 8 concurrent: up to 840 tok/s Prefill: 6000+ tok/s Running W4A16 quant - PP2 - PCIe Gen2 x4 links These cards are putting in work. github.com/AlexBrandes/cmp17…

🏒🚀

We have over 300 paid models available also! Free models rotate every few weeks to give everyone a chance to try them out.

The state of GPU market right now.