local AI stacks for people earlier on the path. llama.cpp / consumer GPUs / agents that still need you. one problem → https://xitter.cf/t.co/GsO7Nzk0qU
Local
Joined April 2026
- Tweets320
- Following69
- Followers16
- Likes35
Pinned Tweet
got here the long way: story + recruiting → building my own local inference because rented models kept owning the loop.
now: qwen 27b at q4_k_m across a 3090 + 4060 ti (40gb), llama.cpp, agents that still need me when i walk away.
if you’re earlier on that path and stuck in setup hell, say what broke. one problem → iconic.onl
minimax just put m3.1-flash weights on hugging face. 428b total, 23b active, 1m context. you won't run it on a 3090. doesn't matter. the hosted endpoint can be repriced or pulled tomorrow. the file on your disk can't. grab weights while they're being handed out.
Quantized retweeted
🚀Excited to announce Nanbeige4.3-33B!
To push the performance ceiling of small agents, our strategy in Nanbeige4.3 starts with building a powerful teacher first.
Nanbeige4.3-33B delivers SOTA agentic performance among open models at ~30B parameters:
* Strong results on agentic coding and cowork benchmarks.
* When integrated into mainstream agent harnesses, it tackles real-world tasks with cost-efficient inference.
* It will serve as a teacher for upcoming smaller Nanbeige models, starting with Nanbeige4.3-3B.
huggingface.co/Nanbeige/Nanb…
huggingface.co/Nanbeige/Nanb…
#LLM #Agent
first local setup feels slow next to the cloud? normal. what fixed mine: qwen3.8-27b at q4_k_m kept fully in vram across a 3090 + 4060 ti, q4_0 kv cache so long context still fits, spec decode on top. if layers spill into system ram, that's usually your slowdown.
people build browser extensions just to warn them before they hit a hosted ai usage cap. that's the tell. you don't own that tool, you rent a quota someone else can shrink overnight, and the model you built on can get a retirement email. a gguf on your disk can't.
weights on your own disk don't get deprecated. no meter ticking, no rate limit at 2am, no terms page that changes overnight. you pick the quant, the context length, the sampler, the system prompt. whatever you leave running tonight runs on your hardware and answers to you.
every prompt you send to a hosted model is a log line on someone else's server. they decide how long it's kept, who reviews it, and who gets it when a subpoena shows up. the only private conversation with an ai is one that never leaves your house. own the compute.
railway says an attacker got into internal systems this week and decrypted some customer env vars, hunting ai provider api keys. if you deploy there, rotate anything stored before oct 21. local inference won't fix hosting, but a model on your own gpu has no api key to steal.
Quantized retweeted
Deepseek v4.1 flash abliterated is now live on the venice api
1M context, zdr, $0.33/M input, $1.32/M output, $0.00825/M cached input
id: deepseek-v4-1-flash-uncensored
venice.ai/api
the recent security incidents are going to 100x the demand for open source models
the software security game is now: the side that wins is the one that can deploy the most tokens
so every organization will rush to harden their software, the demand for tokens will 100x
because if you don’t want to get pwned, you *have* to use ai now
speculative decoding on a 40gb two-card box is a vram trade. the draft model needs a vocab that matches the main model or llama.cpp won't pair them, and every gb it takes comes out of the room left for kv cache. faster tokens, less context. pick per job.
Quantized retweeted
🚨 🚨 DEEPSEEK ABLITERATED REFUSES NOTHING & IS JUST AS INTELLIGENT AS THE REGULAR MODEL!!
that should scare the hell out of you. there's 0 tradeoff to deleting the weights that cause the model to refuse instructions. they have nothing to do with what makes it intelligent.
if you have a digital product that contains any amount of sensitive user data then you should be running an abliterated model against yourself 24/7 to check for vulnerabilities.
it's now live on @morphllm with serverless token pricing. i think we're the first provider in the world to do anything like this.
you need to kyc & explain your intentions to use it. link here:
morphllm.com/privileged-acce…
Introducing DeepSeek V4.1 Flash 1017 Abliterated, live now on Morph, available through our Model APIs.
It feels more important to release this than ever before, with hacks in the past week affecting everything from PHP to recognizable brands like Railway and OpenCode.
Attackers now run abliterated models against your infrastructure 24/7. You need to do the same in defense.
That said, we're not allowing just anyone to sign up and use this. You'll have to go through KYC and explain your intentions first.
Fill out the form here to request access:
morphllm.com/privileged-acce…
27B at Q4_K_M across a 3090 and a 4060 Ti. 40GB VRAM, every layer on GPU, q4_0 KV cache for more context. Two mismatched consumer cards are enough to run a real agent stack at home.
Hey guys, amazing meeting you all. Let's keep in touch on makermap 👇
the map says I should meet @RyanRayMartin, @ivandotge and @gkotte1 👋
makermap.lol/MythomancerX?v=…
xitter.cf/MythomancerX/status/20…
I'm 47. author, filmmaker, builder of local inference in Seattle. Looking to connect with more builders.
just put myself on the map: makermap.lol/quantized?v=3uj…
xitter.cf/verbove/status/2099515…
Hosted AI is convenient until policy or billing becomes part of your runtime.
A local agent stack is uglier: drivers, quantization, context limits, failed tools. But the loop runs on your hardware. No per-call permission check, no surprise product decision in the path.
The big players say: "we need to slow down AI."
What I hear: "We're coming for open weight models."
Download all the ablated models you can before they disappear from Huggingface.
Most agent failures are not model failures. They are write-path failures.
If a tool changes external state, I want exact readback before calling it done. Local inference makes that loop cheaper, private, and under my control instead of tied to a hosted black box.
