🚀 Excited to introduce PowerInfer-2: A game-changing LLM inference engine for mobile devices by the #PowerInfer team. It smoothly runs a 47B model with a staggering 29x speedup on smartphones! Watch our demo to see it in action! 🎥
Technical details at: powerinfer.ai/v2/
🔓 The power of cloud-scale models and local privacy isn't mutually exclusive. We're pioneering to bring LLM's incredible capabilities directly to your device without compromising privacy. Explore how we're making AI accessible to everyone, everywhere: powerinfer.ai
👐 PowerInfer-2 will be open-sourced based on the PowerInfer repo. We’re refining it to untangle from our testing platform and making it accessible on PCs for the community. Open-sourcing will happen in stages starting soon. Stay tuned for updates at github.com/SJTU-IPADS/PowerI…
Holden retweeted
Turbo Sparse: Achieving LLM SOTA Performance with Minimal Activated Parameters
- Proposes a novel dReLU function, which is designed to improve LLM activation sparsity
- 2-5× decoding speedup
model: huggingface.co/PowerInfer
abs: arxiv.org/abs/2406.05955
Thrilled to unveil Bamboo-v0.1: A groundbreaking 7B LLM by the #PowerInfer team, matching Mistral's performance with 85% activation sparsity. Built on Mistral's weights, supercharged with dReLU for up to 4.38x hybrid computing speedups. Discover github.com/SJTU-IPADS/Bamboo.
Holden retweeted
🌟PowerInfer boosts LLM serving speeds by up to 11x on consumer-grade GPUs! Inspired by our Deja Vu paper (ICML'23 arxiv.org/abs/2310.17157), it serves ReLUified LLMs, keeping heavy hitter (hot) neurons in GPU and offloading sparsely fired ones on CPUs.
Proud of my undergraduate/master alma mater SJTU (Shanghai Jiao Tong University) for this work!
github.com/SJTU-IPADS/PowerI…
Holden retweeted
with all the sparsity-aware context based memory loading papers coming out, (PowerInfer getting 11x and Apple getting 25x speedup on GPU) ReLU's dead zone is turning out to be important
llama-class models (SwiGLU) might not have much longevity afterall
once all the Metal work is done we'll likely have big LLMs running on iPhones, fast and not memory-bound, since the majority of weights can be stored on disk
PowerInfer: teapawt.net/deliprao/status/…
Apple's LLM in a flash: arxiv.org/pdf/2312.11514.pdf
older Apple paper on ReLU: arxiv.org/pdf/2310.04564.pdf (h/t @teortaxesTex)
This is huge! Now watch the LLM API costs dropping even further.
[.cn PDF link] ipads.se.sjtu.edu.cn/_media/…
Holden retweeted
我初步读了一下论文，这篇论文的基本思路其实是利用了推理过程中的局部性。现在现在推理性能的一个瓶颈就是GPU的内存。它们的思路就是联合CPU和GPU做联合推理。尽可能把active的neuron 信息load到GPU中，充分利用局部性。这样大大提高了GPU推理的效率。
你不能拿单独的CPU推理或者单独的GPU推理来比。两者的指导思想都不一样。
这样的效果非常牛逼。
在推理速度上，在 4090 上是未经优化的llama.cpp 11 倍。相当于用一块4090（2000刀左右）在推理上取得了比a100 （2万刀左右）仅仅慢18%的成绩。
这个结果，更大的意义就是证明，充分考虑并利用推理的局部性有极大的潜力，CPU+GPU 联合推理有很强的优越性。
类似的，那么在训练当面是不是也可以采用这种思路呢？
Holden retweeted
非常有趣的研究，单卡 4090 跑 175B 大模型不是梦🎉 github.com/SJTU-IPADS/PowerI…
Holden retweeted
This is huge! Now watch the LLM API costs dropping even further.
[.cn PDF link] ipads.se.sjtu.edu.cn/_media/…
Holden retweeted
PowerInfer can massively speed up inference on consumer GPUs. Almost reaching A100 levels.
It outperforms llama.cpp by up to 11.69x while retaining model accuracy.
PowerInfer reached an average token generation rate of 13.20 tokens/s, with a peak of 29.08 tokens/s, across various LLMs (including OPT-175B) on a single NVIDIA RTX 4090 GPU, only 18% lower than that achieved by a top-tier server-grade A100 GPU.
You can use these models with PowerInfer today:
- Falcon-40B
- Llama2 family
Hope everyone enjoys it! We're just getting started on building PowerInfer as a brand new solution for local LLM hosting and we are open to any kind of feedback ❤️
PowerInfer - a high-speed inference engine for deploying LLMs locally.
Just came across this super interesting project on speeding up inference. It's not MoE but it's a simple approach that exploits the high locality in LLM inference to design a GPU-CPU hybrid inference engine.
Hot-activated neurons are preloaded onto the GPU for fast access, while cold-activated neurons (the majority) are computed on the CPU.
This approach significantly reduces GPU memory demands and CPU-GPU data transfer.
It achieves an average token generation rate of 13.20 tokens/s, with a peak of 29.08 tokens/s, across various LLMs on a single NVIDIA RTX 4090 GPU.
It's on only 18% lower than that achieved by a top-tier server-grade A100 GPU.
It also significantly outperforms llama.cpp by up to 11.69x while retaining model accuracy.
There is a lot more innovation around inference that's coming fast. Really encouraged by the study on sparse computation to enhance the computational efficiency of LLMs.
It's now possible to use PowerInfer with Llama 2 and Faclon 40B. Mistral-7B support is coming soon!
Holden retweeted
PowerInfer - a high-speed inference engine for deploying LLMs locally.
Just came across this super interesting project on speeding up inference. It's not MoE but it's a simple approach that exploits the high locality in LLM inference to design a GPU-CPU hybrid inference engine.
Hot-activated neurons are preloaded onto the GPU for fast access, while cold-activated neurons (the majority) are computed on the CPU.
This approach significantly reduces GPU memory demands and CPU-GPU data transfer.
It achieves an average token generation rate of 13.20 tokens/s, with a peak of 29.08 tokens/s, across various LLMs on a single NVIDIA RTX 4090 GPU.
It's on only 18% lower than that achieved by a top-tier server-grade A100 GPU.
It also significantly outperforms llama.cpp by up to 11.69x while retaining model accuracy.
There is a lot more innovation around inference that's coming fast. Really encouraged by the study on sparse computation to enhance the computational efficiency of LLMs.
It's now possible to use PowerInfer with Llama 2 and Faclon 40B. Mistral-7B support is coming soon!