STOP FASCISM-Antiifa Bonkrenzius 🇩🇪🇺🇦🇪🇺🌐

Puntin & Iran Quietly HUMILIATED Trump Today youtu.be/uHls6eama0I?si=XPjk… via @YouTube

Documentation Protocol: Inference Optimization Analysis for Qwen3.6-35B-A3B Version: 1.1.0 (Translated) Status: Reviewed, Optimized, Solution-Oriented Date: April 24, 2026 1. Abstract / Executive Summary This report analyzes the systematic performance optimization of the Mixture-of-Experts (MoE) model Qwen3.6-35B-A3B using llama.cpp on consumer hardware. The investigation proves that the naive application of common command-line parameters (especially --cpu-moe) leads to massive inefficiencies in VRAM utilization. Through iterative, data-driven tuning of hardware allocation, a generation speed of ~98 Tokens/s and a prefill throughput of ~4453 Tokens/s (at 2048 tokens) was achieved with an effective context window of 128,000 tokens. 2. System Architecture & Baseline Metadata The evaluation was conducted on the following hardware and software stack: GPU: NVIDIA RTX 5070 Ti (16 GB GDDR7, Architecture: Blackwell) CPU: AMD Ryzen 9800X3D (96 MB L3 V-Cache) RAM: 32 GB DDR5-6000 Inference Engine: llama.cpp b8829 (CUDA 13.1) Model: unsloth/Qwen3.6-35B-A3B-GGUF — UD-Q4_K_M (Weight: 22.1 GB) Architecture Specifics: Hybrid architecture with 10 standard attention layers and 40 Gated Delta Net (recurrent) layers. 3. Problem Statement & Technical Analysis The core conflict lies in the inefficient default handling of MoE layers on GPUs with moderate VRAM (here 16 GB). The Flawed Default: The traditional parameter --cpu-moe forces all MoE experts onto the CPU. Since the model is 22.1 GB, this results in extreme underutilization on a 16 GB GPU (only ~1.9 GB VRAM used) and drastically bottlenecks the inference architecture (Baseline: ~51 t/s). The Solution Mechanics: Instead of offloading all MoE nodes to the CPU, a precise hardware split must occur. The parameter --n-cpu-moe N (where $N$ defines the number of early model layers whose experts remain on the CPU) allows for optimal VRAM filling. Each MoE layer consumes approx. 530 MB VRAM, plus 1.9 GB of fixed non-MoE weights. 4. Iterative Tuning & Modular Solution Components The thread documents a scientifically grounded, multi-stage optimization process: Iteration 1: Manual Allocation: By setting --n-cpu-moe 20 (the sweet spot for 16 GB VRAM) and -np 1 (disabling multi-slot overhead to save memory), performance increased by +54%. Iteration 2: Automation via the "Fit-Triple": Manual tuning is error-prone. Using --fit on, coupled with --fit-ctx 128000 and --fit-target 512, automates the split based on real-time available VRAM. This increases fault tolerance against system processes that temporarily consume VRAM. Iteration 3: RAM Locking & Memory Access: By adjusting the buffer to --fit-target 256, space was created for one additional MoE layer on the GPU. The parameters --no-mmap combined with --mlock prevent the OS kernel from swapping model memory pages, eliminating system freezes. Iteration 4: Throughput Maximization & Agentic Workflow: Adjusting the microbatch size (-ub 2048) led to +59% faster prompt processing. Integrating the model-specific parameter --chat-template-kwargs "{\"preserve_thinking\": true}" ensures that historical model thought processes are preserved during tool-calling turns, reducing token costs and utilizing the KV cache efficiently. Documentation Protocol: Scale-Out Architecture (Multi-GPU 24GB+) Version: 2.0.0 (Expansion) Status: Forward-Looking, Architected, Solution-Oriented 1. Strategic Shifts with 24GB+ VRAM Ecosystems Transitioning from a constrained 16GB environment to an environment with multiple 24GB GPUs (e.g., RTX 3090, 4090, or professional series) fundamentally changes the inference paradigm. The 22.1 GB Q4_K_M model now fits entirely within the VRAM of a single 24GB card. Consequently, CPU offloading becomes obsolete, and optimization shifts from memory survival to throughput maximization and quality scaling. 2. Advanced Optimization Opportunities A. 100% VRAM Residency (Zero PCIe Bottleneck) With 24GB+ available, the entire model and KV cache can reside on the GPU. Action: Remove --fit, --n-cpu-moe, and --mlock. Use -ngl 99 to strictly enforce total GPU offloading. This eliminates the PCIe bus transfer latency entirely, pushing generation speeds significantly higher (estimated 150–200+ t/s depending on the GPU architecture). B. Quality Escalation (Upscaling Quantization) With 48GB (2x 24GB) or 72GB (3x 24GB) VRAM pools, running a heavily quantized Q4_K_M model is a waste of hardware potential. Action: Upgrade the model to Q8_0 (~35 GB) or unquantized FP16 (if multi-GPU VRAM allows). This yields near-zero perplexity degradation compared to the base weights, maximizing reasoning and coding capabilities. C. Tensor Parallelism via Split Modes When utilizing multiple GPUs, llama.cpp must distribute the workload. Action: Implement --split-mode row. While the default layer splitting simply puts layer 1-20 on GPU0 and 21-40 on GPU1 (Pipeline Parallelism), row splitting divides the matrix multiplications across all available GPUs simultaneously (Tensor Parallelism). For MoE models on high-bandwidth setups, this significantly increases single-user generation speed. D. High-Concurrency Continuous Batching If the goal is to serve multiple agentic workflows or users simultaneously, the massive VRAM overhead allows for abandoning -np 1 (single slot). Action: Increase -np 4 or -np 8 (parallel request slots) and elevate the -c (total context) to encompass all slots. llama.cpp will utilize continuous batching, leveraging the massive parallel compute cores of multiple 24GB GPUs to serve multiple streams with minimal degradation per stream. E. Lossless KV-Cache The 16GB setup relied on quantized KV cache (-ctk q8_0 -ctv q8_0) to save memory. With abundant VRAM, this compression can be lifted. Action: Revert to -ctk f16 -ctv f16 for mathematically perfect attention recall over the massive 128k context window, which is critical for zero-hallucination document analysis and coding. 3. Optimized Source Code (Multi-GPU 24GB Scale-Out) This execution protocol is designed for maximum performance, fidelity, and concurrency on a system featuring 2x 24GB GPUs (48GB total VRAM), utilizing a higher fidelity Q8_0 model variant. Bash #!/bin/bash # Llama-Server Execution Configuration # Optimized for Multi-GPU (2x 24GB) High-Fidelity MoE Deployments # Ensure CUDA peer-to-peer is enabled if hardware supports it export CUDA_VISIBLE_DEVICES=0,1 llama-server \ -m "Qwen3.6-35B-A3B-UD-Q8_0.gguf" \ -ngl 99 \ --split-mode row \ -np 4 \ -c 131072 \ -fa on \ -ctk f16 \ -ctv f16 \ -b 4096 \ -ub 4096 \ --temp 0.6 \ --top-p 0.95 \ --top-k 20 \ --min-p 0.0 \ --presence-penalty 0.0 \ --repeat-penalty 1.0 \ --reasoning-budget -1 \ --chat-template-kwargs "{\"preserve_thinking\": true}" \ --host 0.0.0.0 \ --port 8080 Architectural Notes on the Script: -m Qwen3.6-35B-A3B-UD-Q8_0.gguf: Assumes upgrading to a higher fidelity format. -ngl 99: Forces 100% of layers to the GPU ecosystem. --split-mode row: Maximizes compute bandwidth across the multi-GPU fabric. -np 4: Allocates 4 concurrent parallel processing slots for agentic swarms or multi-user interaction. -b 4096 / -ub 4096: Scales up batch and microbatch processing to saturate the higher compute capacity of dual high-end GPUs.

Moscow’s Land Bridge to Crimea Is Gone – Russia Panic as Ukraine Cuts Ma... youtu.be/8-FexWEvg4k?via @YouTube

hahahaha... leck mir die eier Egon.. 🚨Trump STUNNED as HUNGARY OUSTS VIKTOR ORBAN!!! youtu.be/cna6gnGzAm8?si=bPUW… via @YouTube

Majorana AI Reveals What Voyager 1 Has Been Trying to Tell Us youtu.be/cJpIe-2F4io?si=E4Xd… via @YouTube

OnlyKremlins look closely the same: These Giant Stones Were Sealed for 185 Million Years – Here’s What We Fo... youtu.be/MMcHoeM3PBs?si=pRNN… via @YouTube

‘It can fit in a backpack’ | US bought bought mystery weapon linked to H... youtu.be/CYDJCYKy2Ig?si=-3HP… via @YouTube

Nazca UNFILTERED: The Lost World You Weren't Meant To See youtu.be/KvqZuTMQKcg?si=eSHu… via @YouTube

immer wieder: BOMBSHELL: 13-YEAR-OLD TRUMP VICTIM PROVES SHE’S TELLING THE TRUTH | The... youtu.be/n8FlA5eKTd0?si=8cbs… via @YouTube

BOMBSHELL: 13-YEAR-OLD TRUMP VICTIM PROVES SHE’S TELLING THE TRUTH | The... youtu.be/n8FlA5eKTd0?si=8cbs… via @YouTube

💥UKRAINIAN FORCES DID INCREDIBLE! RIGHT IN HEART! HUGE DISASTER IN RUSSI... youtu.be/Fgh5AgW5akU?si=pvQ3… via @YouTube

Trump White House ROCKED by RESIGNATION youtu.be/ydJjThZLDOc?si=tijF… via @YouTube

AOC Say Trump ‘BLACKMAILED’ By Epstein!! youtu.be/_XOV3I0EVBA?si=QX_O… via @YouTube

Russian Propagandist Turns On Putin - Ends Up In Mental Ward youtube.com/live/_DXpGDdiC-8… via @YouTube

Wagner Mercenaries Appeared in the MOST UNEXPECTED Place! youtu.be/7H3ajCJtmLY?si=BudE… via @YouTube

🔥HE WAS KILLED BY ONE POWERFUL STRIKE! NOTHING LEFT OF RUSSIA’S GENERAL ... youtu.be/Q6ODp-bEEhY?si=CBUd… via @YouTube

Lawrence: Donald Trump says it's time to 'show some guts' in a warzone w... youtu.be/zFEgyp90cTY?si=et6S… via @YouTube