César Salinas (@csalinas) header
César Salinas (@csalinas) profile picture

Ad astra

Monterrey, Nuevo León
Joined April 2010
Photo by César Salinas (@csalinas)Photo by César Salinas (@csalinas)Photo by César Salinas (@csalinas)Photo by César Salinas (@csalinas)Photo by César Salinas (@csalinas)Photo by César Salinas (@csalinas)Photo by César Salinas (@csalinas)
César Salinas retweeted
“SpaceX is becoming a monopoly.” Competitors:
175
498
40
10,082
470,358
After a few weeks of not being able to work on Ornith 1.5 122B on both Intel Arc Pro B65 running on llama.cpp... we got new numbers 🦆 26k bench prompt - DFlash On PP 5,572 tok/s TG 81 tok/s (Draft 954/1784) ~57.5GB VRAM Usage between both cards 26k bench prompt - DFlash Off PP ~6,170 tok/s (almost double vs v12) TG ~73.5 tok/s ~55.4 VRAM Usage between both cards Still working on the custom XE2MX quants (2,3 and 4 bit) which specifically targets the XE2 arch (B60, B65 and B70) Hope some of this can be carried to Qwen4 (part of this work with 3.8 dense, giving about ~2,800 pp and ~42 tg, 28GB VRAM) GitHub and Hugging Face after I move it onto current llama.cpp
6 versions later... less VRAM usage, 24% prefill and 22% decode improvement on Ornith 1.5 122B with two Intel Arc Pro B65 in llama.cpp 🦆 Same custom 3bit quant (XE2MX3), 26k prompt and 262k Q8 context with Dflash2 Prefill: 3355 tok/s Tg: 60 tok/s Still a WIP
1
324
6 versions later... less VRAM usage, 24% prefill and 22% decode improvement on Ornith 1.5 122B with two Intel Arc Pro B65 in llama.cpp 🦆 Same custom 3bit quant (XE2MX3), 26k prompt and 262k Q8 context with Dflash2 Prefill: 3355 tok/s Tg: 60 tok/s Still a WIP
Last one for tonight 🦆 Ornith 1.5 122B on two Intel Arc Pro B65 V5, no drafter: 3128 prefill, 36.8 decode V6, DFlash2 n=3: 2705 prefill, 49.3 decode VRAM went down because the two card scheduler was keeping 4 copies of the same scratch, now we only keep 1
3
1
4
765
Last one for tonight 🦆 Ornith 1.5 122B on two Intel Arc Pro B65 V5, no drafter: 3128 prefill, 36.8 decode V6, DFlash2 n=3: 2705 prefill, 49.3 decode VRAM went down because the two card scheduler was keeping 4 copies of the same scratch, now we only keep 1
Ornith 1.5 122B, same 2x Intel Arc Pro B65s, 262k contex loaded, Q8 KV, 26k benchmark prompt, running on a custom format (XE2MX3) V2 results: 2781 prefill and 26.8 tg V5: 3128 prefill, 36.8 decode Still no drafter/mtp/dflash 🦆
1
1
1
549
Ornith 1.5 122B, same 2x Intel Arc Pro B65s, 262k contex loaded, Q8 KV, 26k benchmark prompt, running on a custom format (XE2MX3) V2 results: 2781 prefill and 26.8 tg V5: 3128 prefill, 36.8 decode Still no drafter/mtp/dflash 🦆
Ornith 1.5 122B already running at 3000 prefill tok/s (same ~26k tokens prompt, 262k context loaded w/Q8 KV) on the pair of Intel B65s in llama.cpp 🦆
1
1
1
2
273
Ornith 1.5 122B already running at 3000 prefill tok/s (same ~26k tokens prompt, 262k context loaded w/Q8 KV) on the pair of Intel B65s in llama.cpp 🦆
Ornith 1.5 122B running on 2x Intel Arc Pro B65 in llama.cpp, one of them over Thunderbolt (~6-8% performance penalty) 262k context, Q8 KV cache, 26k token benchmark prompt: ~2780 prefill tok/s ~26.8 gen tok/s (no drafter and not even tested yet tbh) Custom 3-bit quant (3.06 bpw, ~49GB) + custom Xe2 kernels, ~61GB VRAM in use Still a WIP, Intel B70 and B60 will be compatible
1
1
208
Ornith 1.5 122B running on 2x Intel Arc Pro B65 in llama.cpp, one of them over Thunderbolt (~6-8% performance penalty) 262k context, Q8 KV cache, 26k token benchmark prompt: ~2780 prefill tok/s ~26.8 gen tok/s (no drafter and not even tested yet tbh) Custom 3-bit quant (3.06 bpw, ~49GB) + custom Xe2 kernels, ~61GB VRAM in use Still a WIP, Intel B70 and B60 will be compatible
1
1
2
364
NVIDIA Nemotron 3.5 Bolt running under llama.cpp on a single and a pair of Intel Arc Pro B65 with custom SYCL kernels, XE2MX4 and the Liliccorr drafter Two B65s (XE2MX4, 262k Q8 KV): ~2214 prefill / ~90 tg One B65 (IQ2_XXS + XE2MX4, 262k Q8 KV, 16k prompt): ~2040 prefill / ~34 tg without the drafter ~1490 prefill / ~52 tg with Lilicorr Concurrent on two B65s with drafter (8 slots with 16k Q8 KV): ~136 tg aggregate on an 8-request Card 1 on PCIe 4.0 x4, Card 2 in a Thunderbolt 3 USB-C dock (about 5~8% slower) B60 and B70 have not been tested but should be compatible Code and GGUFs: github.com/cesarsal1nas/llam…
1
1
276
Custom 4bit GGUF for Intel Xe2 under SYCL (Xe2MX4) for this llama.cpp fork Orca Cyber 27B on a B65, ~2034 prefill / 38 tg on a 26k prompt with cheap MTP, unrotated PPL 6.96 vs 6.81 for the IQ4_XS source (this specific GGUF was requantized from IQ4_XS, not clean BF16) 1–8 concurrent requests with 16k ctx each, best total throughput ~99 tok/s at 4 requests including prefill, scaling isn't linear Weights stay on GPU, an extra attention pack sits in RAM for prefill, fast path needs patched oneDNN, patches and instructions are in the repo Only B65 tested here but should also work on Arc Pro B60 and B70 as both are Xe2 EXL3 support for the 4090 and B65 is still there but no optimizations were delivered as Xe2MX4 was the focus of this update GGUF on HF, benchmarks in the README github.com/cesarsal1nas/llam… huggingface.co/cesarsal1nas/…
2
210
OmaLauncher on a Google TV (TCL) just 4 the lulz 😂 Took a bit to get the Arch image and pacman working, and a lot of stuff is still broken so I'm leaving Grok to fix the rest
2
226