A central repository for pre-trained adapter modules in transformers! Active maintainers: @clifapt @h_sterz @LeonEnglaender @timo_imhof @PfeiffJo
Joined May 2020
- Tweets169
- Following1.3K
- Followers1.2K
- Likes228
🚀Adapters v1.2 is out!🚀
We've made Adapters incredibly flexible: Add adapter support to ANY Transformer architecture with minimal code!
We used this to add 8 new models out-of-the-box, incl. ModernBERT, Gemma3 & Qwen3!
Explore this +2 new adapter methods in this thread👇(1/5)
📢 New preprint 🎉
We - the AdapterHub team - present the M2QA benchmark to evaluate joint domain and language transfer!
🔬 Key highlight: We show that adapter-based methods on small language models can reach the performance of Llama 3 on M2QA! 🚀
👇
📢 New preprint 🎉
We introduce "M2QA: Multi-domain Multilingual Question Answering", a benchmark for evaluating joint language and domain transfer.
We present 5 key findings - one of them: Current transfer methods are insufficient, even for LLMs!
📜arxiv.org/abs/2407.01091
🧵👇
📊 Adapter Performance on Point! Our evaluations showcase that all adapter methods are competitive with full model fine-tuning across task classes. 🚀📈
Read more about Adapters in our blog post: adapterhub.ml/blog/2023/11/i…
🔗 Combining adapters was never this easy. Want to combine LoRA and Bottleneck Adapters? Need to stack multiple adapters to transfer to new languages, domains or tasks? Now you can! Adapters allows combining different adapter methods & even complex nested setups.
✨Adapter methods allow modular and parameter-efficient fine-tuning: They can be easily scaled to a myriad of tasks and modularly combined for transfer learning. 🧩↔️
🤖Adapters provides a unified framework that allows adding and combining adapters in one line of code. 🛠️
🎉 Exciting news! The new Adapters library for modular and parameter-efficient transfer learning is out! 🤖
Now simplified & disentangled from @huggingface
pip install adapters
pip install transformers
📄arxiv.org/abs/2311.11077
👾 github.com/adapter-hub/adapt…
#EMNLP2023
🧵👇
For many our tweet seems to be "unavailable" (@TwitterSupport) so lets try this one more time:
We are excited to release Version 3 of the AdapterHub framework!
We now support more architectures
- Prefix Tuning
- Compacter
- Parallel Adapter
which you can Mix-and-Match!
We are excited to release Version 3 of the AdapterHub.ml framework!
We now support more architectures
- Prefix Tuning
- Compacter
- Parallel Adapter
and make it easy to UNION i.e. Mix-and-Match adapters
Our blog covers results and changes: adapterhub.ml/blog/2022/03/a…
He et al. (2021) 👆 further propose Mix-and-Match Adapters as a combination of Prefix Tuning and parallel adapters. This configuration is now supported by our new `ConfigUnion` class which allows you to group multiple configuration instances together.
Parallel adapters (@violet_zct @junxian_he @MaxMa1987 @BergKirkpatrick @gneubig) pass representations in parallel to the feed-forward or attention layer, as opposed to the established, sequential order of computations.
arxiv.org/abs/2110.04366
xitter.cf/violet_zct/status/1448…
Check out our recent work on parameter-efficient fine-tuning. We present a unified framework that establishes connections between state-of-the-art methods (e.g. Prefix-Tuning, Adapters, LoRA).
Great collaboration with @junxian_he and others @MaxMa1987, @BergKirkpatrick @gneubig!
The Compacter (@KarimiRabeeh @JamieBHenderson @seb_ruder) replaces the down- and up-projection layer with a phm layer. Here, the weights matrix is constructed from two smaller matrices by computing their Kronecker product.
arxiv.org/abs/2106.04647
xitter.cf/KarimiRabeeh/status/14…
Excited to share our new work with DeepMind: arxiv.org/abs/2106.04647
Our main contribution:
We propose Compacter (Compact Adapter) layers, a method to adapt large-scale language models, which only trains around 0.05% of a model's parameters and performs on par with fine-tuning.
