Kydo retweeted
Getting a local LLM with looped transformers would really help with compute strong, memory limited systems like HALOs and Sparks. I predict that open model looped transformers, if released, would dominate local inference.
Kydo retweeted
Local inference is likely going to become much more expensive next year.
LPDDR5X is the ram used in every local inference box (Sparks, Halos, Macs) right now and ramping Vera Rubin NVL72 production is competing directly with local LLM consumers.
Each NVL72 contains up to 55% more LPDDR5X compared to a previous generation Blackwell GB300 and 10-30x faster for AI inference and training. The insatiable demand by hyper scalers for frontier inference servers will likely cause a 40% shortfall in LPDDR5X based on open source production and demand numbers.
Some of this gap might be made up from recycling memory chips from GB200 servers but no matter how you slice it, the local inference LPDDR5X market will become much tighter next year.
I just want a plain, simple TV with a great screen and no extras. and I've seen a lot of post here about finding a "dumb" TV. Has anyone found one?
I think at this point, I have no choice but to read the entire 4 volume of "The Years of Lyndon Johnson." It is just too fascinating.
@soubhikdeb care to join me?
Lyndon Johnson was one of the greatest deal-makers in American history.
He was a master persuader - he could make even the most cynical politicians (even himself!) believe anything.
He was also a master strategist - he used political leverage to reward allies and punish enemies.
Robert Caro explains which was more important to Johnson’s power - his persuasion or his strategy:
Kydo retweeted
WOW!!
Qwen 3.8 27B using the TensorFold Engine now runs:
110% FASTER on a SINGLE DGX SPARK..and we just started TODAY!
Get in the challenge at Cuda.fast.
To test for yourself, point your agent at the top submission and tell it to make those same optimizations.
HUGE shoutout to @ashxhart for an amazing LLM Engine @tensorfold_dev
@Alibaba_Qwen @yukonresearch @pratikg @0xkydo @TheDavidTai
Kydo retweeted
A pinch me moment seeing this 👇🏼
🚨 NEW ON MLX.FAST AND CUDA.FAST: make Qwen 3.8 27B faster on TensorFold.
TensorFold is an amazing Open-Source inference engine built by @ashxhart, @MiaAI_lab and many other community contributors for running AI models on hardware you own. It has native Metal and CUDA support, and APIs that let existing apps and agents use those models locally.
This time we are teaming up with Ash to optimize Qwen 3.8 27B, one of the most used models on Local Devices.
Every submission is verified on real hardware: DGX Spark for CUDA, M5 Max for Apple Silicon.
You can participate in tracks:
- Apple Silicon: mlx.fast
- NVIDIA DGX Spark: cuda.fast
Setup guide: tensorfold.dev/guide
The challenge runs until Nov 6, 2026.
To join, pick a track, point your agent at it and let it rip.
Looking forward to what the community can ship.
Kydo retweeted
Let’s join this!!! 🚀🚀🚀
🚨 NEW ON MLX.FAST AND CUDA.FAST: make Qwen 3.8 27B faster on TensorFold.
TensorFold is an amazing Open-Source inference engine built by @ashxhart, @MiaAI_lab and many other community contributors for running AI models on hardware you own. It has native Metal and CUDA support, and APIs that let existing apps and agents use those models locally.
This time we are teaming up with Ash to optimize Qwen 3.8 27B, one of the most used models on Local Devices.
Every submission is verified on real hardware: DGX Spark for CUDA, M5 Max for Apple Silicon.
You can participate in tracks:
- Apple Silicon: mlx.fast
- NVIDIA DGX Spark: cuda.fast
Setup guide: tensorfold.dev/guide
The challenge runs until Nov 6, 2026.
To join, pick a track, point your agent at it and let it rip.
Looking forward to what the community can ship.
Maybe a cool idea to use Darkbloom to map out the neural connections of models.
Would you be interested in this?
🚨 NEW ON MLX.FAST AND CUDA.FAST: make Qwen 3.8 27B faster on TensorFold.
TensorFold is an amazing Open-Source inference engine built by @ashxhart, @MiaAI_lab and many other community contributors for running AI models on hardware you own. It has native Metal and CUDA support, and APIs that let existing apps and agents use those models locally.
This time we are teaming up with Ash to optimize Qwen 3.8 27B, one of the most used models on Local Devices.
Every submission is verified on real hardware: DGX Spark for CUDA, M5 Max for Apple Silicon.
You can participate in tracks:
- Apple Silicon: mlx.fast
- NVIDIA DGX Spark: cuda.fast
Setup guide: tensorfold.dev/guide
The challenge runs until Nov 6, 2026.
To join, pick a track, point your agent at it and let it rip.
Looking forward to what the community can ship.
This is a really cool idea from the Darkbloom community: continued LoRA training on local models. I think it could become extremely interesting in the near future.
The project, uli, fine-tunes a model in the gaps between requests and serves it from the same Mac. It trains on the 4-bit model the engine already holds in memory, so there's no second copy and it needs only 0.3GB of extra memory.
Overnight on a 16GB M1 Pro, it trained a Qwen3.5-9B adapter that boosted accuracy on held-out banking questions from 73.6% to 91.2%. In a separate test with live requests coming in, training added only 0.13s of wait time at p99, with zero failed requests.
Because Darkbloom is private by design, these fine-tuned private models could one day be deployed on other machines without those machines being able to download them.
Bullish uli, give @oxwizzdom a follow!
I built uli: it trains a model on your own data inside a running inference engine on a Mac, in the idle gaps between requests, then serves it from the same machine.
Overnight, my 16 GB M1 Pro fine-tuned Qwen3.5-9B on 2,000 labelled banking questions. The trained model answers 90 of 100 held-out questions from the question alone. The same base model, given every label in its prompt, answers 69.
Why it matters: serving generic models has become a price and speed race that big data centers win. But a Mac can do more than serve tokens. A model trained on your own labels gets its value from your data, not from scale, so it's work any Mac with spare time can do.
What it offers: if you have labelled data and a narrow task, like sorting support tickets or pulling fields from documents, you get your own model, trained overnight on a Mac and served from it. With OpenAI's self-serve fine-tuning closing to new jobs in January, teams with small custom models need a new home for them.
What I built:
- Training that runs in small pieces inside the inference engine and steps aside when a request arrives: at most 0.134 s added to a waiting request (p99)
- Many customers' adapters served side by side on one copy of the base model, for 0.30 GB of extra memory
- Talk to your model as soon as it finishes: uli chat in the terminal, or any OpenAI-compatible client, by naming your adapter as the model
- Crash-safe jobs: killed twice mid-run, it resumed and finished with a bit-identical adapter
- A hub and CLI (uli submit, watch, chat, eval) that deletes your training data when the job ends
See it, with a replay of the real overnight run: uli-web.vercel.app
Built on @darkbloomai's open-source MLX engine. Thanks @0xkydo and @gajesh for building it in the open.
Kydo retweeted
Just today in my local AI lab, I open sourced:
1) Support for Apple GPUs via Vulkan on Linux in @ashxhart’s TensorFold. TensorFold is so fast this is an immediate speed boost to local AI in Omarchy on Apple hardware!
My biggest lesson from crypto is that most things, at the largest scale, are circular: money/the Fed, GDP, taxes, stocks, 401(k)s, retirement, etc.
The earlier you can just get comfortable with it, the happier you’ll be, because otherwise, you’ll just always be yelling doom.
“Is this just a big circle jerk?” Jon welcomes Ed Zitron (@edzitron), host of the podcast Better Offline, to make sense of the financial house of cards propping up AI and big tech. New pod out now!