idk what I’m doing half the time. but sometimes I design systems for constrained compute | local & edge ai current curiosity: designing a model
Earth
Joined September 2023
- Tweets2.8K
- Following245
- Followers3.7K
- Likes4.9K
Pinned Tweet
I got a 1T (trillion) parameter model running on my MacBook Pro.
Kimi-K2. 1.029T params.
~1 TB raw weights.
524 GB converted.
~1.7 tok/s.
Yesterday it was 671B. Today it's 1T.
Same laptop. Same M4 Max. No cloud.
When I say we: I mean Claude and me.
using the api for glm 5.3 and it's surprisingly good.
it catches issues that fable did not...
(in max thinking)
Important work.
Introducing Inkling 2 Small. Scoring 31 on the AA Intelligence Index, it's a big jump over its predecessor.
Inkling 2 Small is available for fine-tuning and inference on Tinker today, with full weight release planned soon.
thinkingmachines.ai/news/ink…
Qwen3.5-4B edited by Claude (fable 5.1)
9.2% less GPU energy per correct answer
learns from its own shortest correct answers
600 problems x 4 tries -> 382 kept
1.7 Wh of GPU energy per kept answer
Daniel Isaac retweeted
i built a swift package that lets my iphone run brand new swift code without compiling it, opening xcode, or updating the app
to test it, i had a small llm running on my iphone generate an entire chess game in swift
~550 lines of code
i've been building a swift interpreter that turns code into real, native swiftui views at runtime
and apparently it can handle quite a bit (although still room for improvement)!
the chess game has legal moves, checkmate, castling, pawn promotion, and even spring animations when the pieces move
all running directly from a string of swift code
the package boils down to just one line:
KilnView(code: source)
it's free, open source, and works on ios and macos
but the part i'm most excited about is what this means for ai
i can now ask an ai running on my iphone to build me a little app (keyword is little 😅), and actually use it right away
no compiler or app updates required!
link in reply :)
we’ll be seeing more of this
Introducing Arete-27B: Qwen3.8-27B, post-trained to reinforce Western values.
Chinese open models are currently the strongest available, but they ship with state-aligned behavior in the weights. We built an 803-prompt eval across seven value categories, including freedom of expression, democracy, and privacy, and measured how often a model adopts China’s party line.
After fine-tuning on roughly 53 million tokens, Arete-27B's propaganda rate fell from 22% to 1%, in line with the Western models we tested. Its Hermes Bench score shows no significant performance tradeoffs.
The values in an open model are one fine-tune away.
Our eval, weights, and training data are available now along with the full blog post: nousresearch.com/arete
Mac vs Spark: a performance and energy baseline on Qwen3.8-27B
1k to 256k tokens
prompt processing (reading the prompt)
token generation (writing the answer)
energy: watt-hours for each
same GGUF on llama.cpp on both. a baseline for trends
🧵 1/6
why start here: same file, same engine on both boxes, so only the hardware changes
now every tuned engine has a fixed point to beat
next: MLX on the Mac, vLLM on the Spark, then 512k
🧵 5/6
tl;dr
the Mac's writing speed falls off past 64k tokens
at a 128k prompt, energy is about even
within 12% for answers of 16k and up
at a 256k prompt, the Spark wins everywhere
faster and more efficient at every answer length
a signal, not proof: partly projected
512k runs are next
speed
the Spark reads prompts 4.4-6.5x faster
the Mac writes 1.7x faster at short context
energy*
Spark: 34-61% less on 128k-256k prompts
with answers up to 4k
Mac: 43-46% less on prompts up to 16k
with answers of 16k-64k
*meters differ: Spark GPU only, Mac chip
M4 Max on llama.cpp. MLX is next
🧵 6/6
Qwen3.8-27B on one DGX Spark, the same whole-file code edit, GPU energy per correct file:
vLLM: 13.6 kJ
vLLM + MTP4: 4.1 kJ
TensorFold + DFlash2: 1.1 kJ
12x less energy. every run wrote the same file byte for byte
one 1,000-token answer from Qwen3.8-27B on a DGX Spark, GPU energy:
4k-token prompt: 3.0 kJ
256k: 36 kJ
1M: 414 kJ (115 Wh)
at 1M, 98% of it goes to reading the prompt. the answer itself is 9 kJ
