Massimo Bini (@mwbini) header
Massimo Bini (@mwbini) profile picture
Joined May 2021
Photo by Massimo Bini (@mwbini)Photo by Massimo Bini (@mwbini)
PEFT has reached 20k GitHub stars ⭐⭐⭐⭐⭐ Right on time for the new PEFT release, v0.18.0. And it's not a small one at that, with lots of new PEFT methods and other improvements. Details in the 🧵
1
5
1
14
5,936
Multiplicative ft lags behind additive ft (LoRA) in terms of speed We narrow this gap • block-diagonal ETHER/+ has no params decrease → modest performance change • different diag blocks affect different W blocks → split allows for lower compute and parallelization → ⏩ 4/6
1
1
657
Comparing to previous multiplicative methods: previous works put emphasis on hyperspherical energy retention (OFT, BOFT) → we bring partly contrasting insights, emphasizing the importance of having a boundary on the Euclidean distance instead 5/6
1
1
512
This was a joint work done with @confusezius @zeynepakata @anna_khoreva 🙌 For more details • longer overview eml-unitue.de/publication/et… • code github.com/mwbini/ether • paper arxiv.org/abs/2405.20271 6/6
1
7
458
This leads to • non-deteriorating nature of the transformations • robustness to learning rates / hyperparameters choices • strong parameter-efficiency 3/6
1
2
468
❔Is constraining the finetuning to low-rank matrices (s.a. in LoRA) the best approach? In our new ICML paper (ETHER) we propose to learn hyperplane reflections of pretrained weights as an alternative • fewer parameters (from 1 vector per layer) • interesting properties! 1/6
1
17
4
57
16,479
We propose: ETHER, uses Householder reflections • most efficient (one learnable vector per layer) • constant distance (wrt to identity matrix - neutral transformation) ETHER+, uses relaxed Householder w/ interaction of 2 hyperplanes • best performant • bounded distance 2/6
1
1
543