Local AI 路 Stretching a Single RTX 4090 24GB Form Over Function
Joined May 2024
- Tweets469
- Following458
- Followers174
- Likes4.1K
You measured the vram temps? Or its the main chip temp? Thats the data i need to know before try it on my 4090.
Any trade off in temps? 7% performance on prefill in exchange of the lifespan health of my graphics card seems not worth it to me.
hmm... thats not bueno. Not a nice trade shorten the lifespan of my card for some performance in some areas.
Pros and cons of doing this? Risks? Where the performance gains lands? I can see an improvement on gaming? Or only in prefill on inference?
Ghost In The Stack retweeted
Dear @nvidia, a friendly ask from a guy running 4 of your 3090s and 1 of 3080 Ti.
The 30 series is 6 years old. You dont make these cards anymore. Maybe its time to hand them over to the community completely.
My post about unlocking the 3090 got 270K views in a day, and the replies are the real story. People asking about the 4090, 3080, 3060 and 2080 Ti. People asking how to do it on Windows. Others listing what the community already patches on these cards: the NVENC session limit, P2P between GPUs.
Thousands of people are digging through your old cards with kernel modules, 1 setting at a time, because they love this hardware and want everything it can give.
Publish what is limited on consumer Ampere and Turing. Ship an official driver switch for older generations. Or just open it up.
I understand the business side. Some people might keep their old cards longer instead of buying new ones. But it would be a huge gesture. You talk about local AI on stage, you ship open models, you sell DGX Spark for home. This would show the care goes all the way down to a used 3090 under someone鈥檚 desk.
Give them to us. We will do the rest: benchmarks, guides, models tuned for these cards.
@NVIDIAAI
NVIDIA sold millions of 3090s with 50% of the tensor speed switched off. Not missing. Switched off.
1 kernel module flips it to full, and my BF16 tensor pipe went from 82 to 155 TFLOPS. Nothing else changed on the cards, and the outputs match the stock run byte for byte.
All so the gaming card wouldnt eat into workstation sales. Every 3090 owner ran at 50% for 6 years, so a pro card could look worth the price. Fine, its their business model.
And LLMs are the least interesting place for this. Decode waits on memory, and my engine runs its experts in int8. Still +10% on prefill.
The models that live on BF16 tensor math are everything else. Qwen Image 2.1 and FLUX 3 Klein, where every diffusion step is big matmuls. MiniMax H3, which takes 4 to 7 minutes on my cards. Music and speech models. And training, every LoRA anyone ever trained on a 3090 went through exactly the path that was cut.
Thats what I measure next. This is only the start.
OpenAi being the 3rd lab right now after Meta release, its something i didn't see coming. Wild Monday.
OpenAi being the 3rd lab right now after Meta release, its something i didn't see coming. Wild Monday.
Im rocking 3.8 Flash-Next on a 4090 + 64 ddr5, IQ3_S.
Its worth it try to switch to this quant of glm 5.3 flash, or better stay with the flash next?
Tbf, its the 5h limit what killed the resets for me. Idc if they anounce a reset tomorrow if its virtually imposible for me to burn my usage working on the firts place.
That's how it's always worked. You shut your mouth, and let your work do the noise. I see OpenAI doing just the opposite these past few weeks, lots of talk, but their work leaves much to be desired.
I loved 27b more than anyone in the world, but sadly, 3.8FN its just a superior model in all realms, im sorry ):
If you could run IQ3_S, you should never consider 27b again. Will see what the Qwen 4 family comes up to, maybe it change my mind.
KV 4bit? Or i don't understand how you get these numbers.
Its very important put the KV bit size on the numbers when you share something 馃槶
Don't lose hope, myself 2 or 3 months before could not believe running 180b MoE at these speeds and inteligence capabilities on my machine. Things are improving fast. Eventually youre going to be capable to do amazing things with your machine too. 32+16 its not a bad computer by any means.
Im not trying to be mean in any way, just a humble and honest opinion, but i dont think this is usefull for the vast majority of people if you need 256gb ram to make it work. Like, lets be serious, who the hell its running that? 0.00001% of people out there? With that being said, its always valuable for the community make the best of any configuration, and its always apreciated any advancement.
im runing Ista IQ3_S, which its like 3.5bit on the practice, with a 256k context, kv8, 70-80 tok/s pure decode, like 3.000 toks prefill on a 64gb machine. Extensive tests by Opus 5.5 to catch any degradation on the quant, nothing found yet. Can run for hours, its excelent in all ways.
The quants from ISTA are absolutely SOTA. Paired with Strata its mindblowing intelligence/performance on a gaming pc. I did many tests with Opus 5.5 to the IQ3_S, and the conclusion is, its perfect.
Really? I can't believe it. Many on twitter know about it and have screenshots about it tho, so they can't deny it. Shameles company omg