Stephen Panaro (@flat) header
Stephen Panaro (@flat) profile picture

Stephen Panaro

@flat

making coffee and other things. @BrewTimerApp

 boston
stephenpanaro.com
Joined May 2013
  • Tweets905
  • Following25
  • Followers545
Replying to @anemll
@anemll
1
1
124
btw, you can quantize the “hard-to-quantize” Llama 3.1 8B now. (LDLQ is GPTQ)
Image from Stephen Panaro (@flat)
1
3
838
Why not just release the weights at this point?
Image from Stephen Panaro (@flat)
Image from Stephen Panaro (@flat)
1
2
7
580
Replying to @anemll
Might be something interesting in Instruments. “Inference details”
Image from Stephen Panaro (@flat)
1
2
310
Figured out 4-bit /per-tensor/ quantization for Qwen2.5-0.5B. It’s on par with per-group GPTQ which is kinda cool (tbh non-uniform helps a lot). 🖇️Weights, evals, more details below.
This markdown table: | Tasks | |f16 | this | gptq | |-------------|---|-----:|-------:|-------:| |arc_challenge|↑ |0.2918| 0.2705 |*0.2730*| |arc_easy |↑ |0.6465|*0.6393*| 0.6031 | |boolq |↑ |0.6208| 0.5862 |*0.6232*| |hellaswag |↑ |0.4061| 0.3888 |*0.3969*| |piqa |↑ |0.7051|*0.6861*| 0.6801 | |winogrande |↑ |0.5635|*0.5762*| 0.5659 | |average |↑ |0.5390|*0.5245*| 0.5237 |

ALT This markdown table: | Tasks | |f16 | this | gptq | |-------------|---|-----:|-------:|-------:| |arc_challenge|↑ |0.2918| 0.2705 |*0.2730*| |arc_easy |↑ |0.6465|*0.6393*| 0.6031 | |boolq |↑ |0.6208| 0.5862 |*0.6232*| |hellaswag |↑ |0.4061| 0.3888 |*0.3969*| |piqa |↑ |0.7051|*0.6861*| 0.6801 | |winogrande |↑ |0.5635|*0.5762*| 0.5659 | |average |↑ |0.5390|*0.5245*| 0.5237 |

1
1
2
1,007
Have further tuned my lil’ quantization kernel. 0.3s (original, PyTorch) 0.12s (+MLX) 0.09s (+tiny Metal kernel) 0.051s (now, big fused kernel go brr) Speedup is >2x for larger matrices. h/t MLX learned/stole a lot from the mv kernel
Image from Stephen Panaro (@flat)
2
1
18
1,174
Added some benchmarks for opt-125m. Seems good, but probably need to run larger models for a clearer picture.
Image from Stephen Panaro (@flat)
1
168
Might have a new favorite ModernBERT token:
Image from Stephen Panaro (@flat)
1
3
546
Claude Code seems promising for lil’ niche apps. Here’s a quick menu bar tokenizer: → Paste IDs and see tokens → …or vice versa. → Copy text → …or `torch.Tensor([…`. ✨
Image from Stephen Panaro (@flat)
3
281
Wanted to see if the randomness of the random rotation matrices was at play. Doesn’t seem like it. Even ones that did better struggled on some benchmarks. e.g. # 8 is good but not on BoolQ or Hella. (All HQQ because it quantizes fast.) (Sorry for the chart crime. Data in alt.)
|Model |Arc-C |Arc-E |BoolQ |PIQA |Wino |Hella |-- |-- |-- |-- |-- |-- |-- |Qwen2.5-0.5B float16 |0.2884|0.6452|0.6211|0.7029|0.5675|0.4052 |Qwen2.5-0.5B HQQ |0.2816|0.6330|0.5220|0.6828|0.5604|0.3959 |Qwen2.5-0.5B-Rotated HQQ H0|0.2722|0.6103|0.4829|0.6817|0.5485|0.3891 |Qwen2.5-0.5B-Rotated HQQ H1|0.2713|0.6124|0.6242|0.6931|0.5651|0.3876 |Qwen2.5-0.5B-Rotated HQQ H2|0.2739|0.6267|0.4783|0.6850|0.5556|0.3844 |Qwen2.5-0.5B-Rotated HQQ H3|0.2696|0.5926|0.4936|0.6823|0.5564|0.3821 |Qwen2.5-0.5B-Rotated HQQ H4|0.2867|0.6221|0.6315|0.6736|0.5556|0.3875 |Qwen2.5-0.5B-Rotated HQQ H5|0.2645|0.5812|0.6190|0.6616|0.5635|0.3872 |Qwen2.5-0.5B-Rotated HQQ H6|0.2756|0.6326|0.4930|0.6850|0.5730|0.3855 |Qwen2.5-0.5B-Rotated HQQ H7|0.2901|0.6490|0.5349|0.6834|0.5604|0.3859 |Qwen2.5-0.5B-Rotated HQQ H8|0.2782|0.6431|0.6456|0.6866|0.5533|0.3885 |Qwen2.5-0.5B-Rotated HQQ H9|0.2841|0.6406|0.5193|0.6915|0.5572|0.3887

ALT |Model |Arc-C |Arc-E |BoolQ |PIQA |Wino |Hella |-- |-- |-- |-- |-- |-- |-- |Qwen2.5-0.5B float16 |0.2884|0.6452|0.6211|0.7029|0.5675|0.4052 |Qwen2.5-0.5B HQQ |0.2816|0.6330|0.5220|0.6828|0.5604|0.3959 |Qwen2.5-0.5B-Rotated HQQ H0|0.2722|0.6103|0.4829|0.6817|0.5485|0.3891 |Qwen2.5-0.5B-Rotated HQQ H1|0.2713|0.6124|0.6242|0.6931|0.5651|0.3876 |Qwen2.5-0.5B-Rotated HQQ H2|0.2739|0.6267|0.4783|0.6850|0.5556|0.3844 |Qwen2.5-0.5B-Rotated HQQ H3|0.2696|0.5926|0.4936|0.6823|0.5564|0.3821 |Qwen2.5-0.5B-Rotated HQQ H4|0.2867|0.6221|0.6315|0.6736|0.5556|0.3875 |Qwen2.5-0.5B-Rotated HQQ H5|0.2645|0.5812|0.6190|0.6616|0.5635|0.3872 |Qwen2.5-0.5B-Rotated HQQ H6|0.2756|0.6326|0.4930|0.6850|0.5730|0.3855 |Qwen2.5-0.5B-Rotated HQQ H7|0.2901|0.6490|0.5349|0.6834|0.5604|0.3859 |Qwen2.5-0.5B-Rotated HQQ H8|0.2782|0.6431|0.6456|0.6866|0.5533|0.3885 |Qwen2.5-0.5B-Rotated HQQ H9|0.2841|0.6406|0.5193|0.6915|0.5572|0.3887

2
182
I think you could put a tiny attention head-sized rotation matrix just before RoPE. For SpinQuant you get a bunch more rotation matrices to learn. Cost is a reasonably small # of extra FLOP/weights. Last time posting some version of this image, I promise. (At least this week.)
Modified diagram from SpinQuant, showing a fused rotation matrix in the Q and K weights and a head-sized online inverse performed right before RoPE.

ALT Modified diagram from SpinQuant, showing a fused rotation matrix in the Q and K weights and a head-sized online inverse performed right before RoPE.

248
Liking the line of research where you multiply LLM weights by rotation matrices and the model still works. Most do it in between layers, but you can also sneak one between Q/K and RoPE. Extra parameters? None. Useful? …Maybe. Cool? I think so. (See R₅ below.)
Attention portion of the SpinQuant diagram, with R₅ added.

ALT Attention portion of the SpinQuant diagram, with R₅ added.

1
1
5
1,205
The RoFormer paper (the one that introduced RoPE) has a lovely presentation of their rotation matrix and later an efficient way to calculate it. TIL Llama (so ~everyone) uses a different matrix and calculation. Oof.
Image from Stephen Panaro (@flat)
4
336
D’oh. QuaRot already tried weight-only GPTQ in the appendix. Looks nice. Even more interested in AWQ et al now. (RTN beating GPTQ for 7B 🤨 though)
Image from Stephen Panaro (@flat)
Stephen Panaro (@flat) profile pictureStephen Panaro@flat17 Jan 2025
Might test this out. Seems low effort/fairly interesting. GPTQ, AWQ, llama.cpp, maybe HQQ. Anything others?
2
4
679
Load more