VPetar

A 12GB RTX 3060 is reportedly generating around 50 tokens per second with Qwen3.8-35B-A3B. The same setup reaches approximately 550 tok/s in prompt processing, using only 16GB of system RAM alongside the GPU. No 24GB graphics card. No multi-GPU server. Just a consumer GPU, limited system memory and a carefully optimised inference stack. The experiment uses Qwen3.8-35B-A3B, an MoE model, with aggressive quantisation and CPU/GPU hybrid inference. The hardware and configuration The reported setup looks like this: GPU: NVIDIA RTX 3060, 12GB VRAM System RAM: 16GB Model: Qwen3.8-35B-A3B Quantisation: Q4_K_M Runtime: llama.cpp Generation speed: ~50 tok/s Prompt processing: ~550 tok/s Configured context: 262K tokens Optimisations: Flash Attention and Q8 KV cache The two speed figures measure different stages of inference. Prompt processing, also called prefill, measures how quickly the model processes incoming text. Generation, or decode, measures how quickly it produces new tokens. So, 550 tok/s does not mean the model generates answers at that speed. The reported generation rate is around 50 tok/s. How does a 35B model fit into 12GB of VRAM? The main advantage comes from its Mixture-of-Experts architecture. Qwen3.8-35B-A3B has approximately 35 billion parameters in total, but only around 3 billion are activated for each token. Rather than sending every token through all the model’s experts, a routing mechanism selects a smaller group of experts to perform the computation. This reduces the amount of computation required per token compared with activating the entire network. However, there is an important distinction: 3B active parameters does not mean the entire model only needs memory for 3B parameters. The full set of weights still needs to be accessible during inference. That’s where hybrid inference comes in. Splitting the workload between GPU and system RAM The developer uses llama.cpp to distribute model components across the RTX 3060’s 12GB VRAM and the machine’s 16GB system RAM. The reported configuration offloads 26 MoE layers to the CPU. This allows the system to access model components that cannot fit entirely inside the GPU’s memory. The trade-off is data movement between system RAM and GPU memory. CPU offloading can make larger models possible, but transfers over PCIe introduce overhead and can become a performance bottleneck. The MoE architecture helps reduce computation, while hybrid inference makes the full model accessible. Three optimisations help manage memory 1. Q4_K_M quantisation The model weights use a 4-bit GGUF quantisation format. Reducing weight precision significantly lowers storage requirements, making the model easier to accommodate across limited GPU and system memory. There is a quality trade-off, though. Quantisation introduces approximation error, and its effect can vary across coding, reasoning and other tasks. 2. Flash Attention Attention calculations can generate substantial intermediate data and require significant memory movement. Flash Attention improves efficiency by reducing unnecessary memory transfers and avoiding the need to store certain large intermediate attention matrices in full. This helps reduce memory pressure and can improve inference performance. 3. Q8 KV cache The KV cache stores key and value representations from previously processed tokens so the model can reuse them during subsequent generation. As context grows, the cache becomes increasingly important to memory consumption. Using an 8-bit KV cache instead of a higher-precision format reduces the memory required to retain previous context, leaving more room for the model and longer conversations. Together, these optimisations address different parts of the memory problem: model weights, attention computation and cached context. Can it actually handle 262K context? The configuration reportedly supports a context window of up to 262K tokens.

One RTX 3090 with 24GB VRAM is reportedly delivering 1,035 tokens/s across 64 concurrent Qwen3.8-27B sessions. The setup runs at a 250W power limit using HyperQwen, an open-source inference stack built around vLLM. Reported performance: • Single session: ~127 tok/s • 64 concurrent sessions: ~1,035 tok/s aggregate • Context: Up to 150K tokens • GPU: RTX 3090, 24GB VRAM The key is vLLM’s continuous batching and PagedAttention. Continuous batching keeps new requests moving through the GPU as others finish, while PagedAttention manages KV-cache memory more efficiently across sessions. The 1,035 tok/s figure is total server throughput, not the speed each user receives. Individual session speeds can vary as more requests compete for GPU resources. HyperQwen also provides Docker profiles for single-session and batch serving. Project: syv-ai/HyperQwen on GitHub. It’s an interesting demonstration of how much more a consumer GPU can do when the inference software is optimised for concurrent workloads, not just one user.