
PhD student @uni_tue @ExplainableML @HelmholtzMunich
Joined May 2021
- Tweets7
- Following202
- Followers61
- Likes90
Massimo Bini retweeted
PEFT has reached 20k GitHub stars ⭐⭐⭐⭐⭐
Right on time for the new PEFT release, v0.18.0. And it's not a small one at that, with lots of new PEFT methods and other improvements. Details in the 🧵
Multiplicative ft lags behind additive ft (LoRA) in terms of speed
We narrow this gap
• block-diagonal ETHER/+ has no params decrease → modest performance change
• different diag blocks affect different W blocks → split allows for lower compute and parallelization → ⏩
4/6
1
1
657
Comparing to previous multiplicative methods: previous works put emphasis on hyperspherical energy retention (OFT, BOFT)
→ we bring partly contrasting insights, emphasizing the importance of having a boundary on the Euclidean distance instead
5/6
1
1
512
This was a joint work done with @confusezius @zeynepakata @anna_khoreva 🙌
For more details
• longer overview eml-unitue.de/publication/et…
• code github.com/mwbini/ether
• paper arxiv.org/abs/2405.20271
6/6
1
7
458
This leads to
• non-deteriorating nature of the transformations
• robustness to learning rates / hyperparameters choices
• strong parameter-efficiency
3/6
1
2
468
❔Is constraining the finetuning to low-rank matrices (s.a. in LoRA) the best approach?
In our new ICML paper (ETHER) we propose to learn hyperplane reflections of pretrained weights as an alternative
• fewer parameters (from 1 vector per layer)
• interesting properties!
1/6



