low is the way to the upper bright world
☯️ 🇺🇸 views mine
Joined August 2009
- Tweets9.5K
- Following2.3K
- Followers2.4K
- Likes87.2K
darren retweeted
Alright I finally have more to share on my personal project!
Resin is a runtime for building fast LLM inference engines.
The idea is to make all the generic parts of inference reusable, and leave only the model-specific forward pass + kernels to be written for each architecture / hardware target.
To build an engine:
1. Import Resin's core.
2. Have an agent write a forward pass for the model you’re interested in, for your specific hardware (GPU or TPU or AMD), any kernels you want.
Then, boom, you have a standalone inference engine.
—
I've been hacking over the past month (early post: xitter.cf/ekzhang1/status/208950…), and recently this week I got a lot of features + parallelisms working, inspired by DeepInfer.
For DeepSeek-V4.1-Flash, it's nearly as fast as DeepInfer (within 5-10%), while being self-contained and supporting other models, plus metrics + HTTP server + prefix cache.
Yesterday I wrote a skill for authoring models and put Codex to work on it overnight. The agent built a GLM-5.3-Flash engine with MTP, reaching up to 60,000 input tok/s and 16,000 output tok/s on B200 (EP4).
So far inference "vibe coding" has not been very reliable or successful. I think it's because people start from scratch each time and reinvent lots of tricky things on the scheduler and server layers. But with a shared core, we can automate lots of inference work.
The tricky bit is writing a fast forward pass. But Resin turns that into a standalone engine by just importing a library, and it becomes a verifiable goal for the AI.
Code: github.com/ekzhang/resin
So… I have it working now. The core of an inference engine in Rust is about 3k lines (scheduling, kv/swa/recurrent state, prefix cache, multimodal, dynamic batching)
I then asked Codex to implement (with MTP, cudagraphs, TP on Blackwell):
- DeepSeekV4 Flash
- Gemma 3.8 27B
- GPT-OSS
- Inkling
It turns out that most models take a lot of code to run with fast, persistent graphable kernels and so on… on the order of 2.5k for a simple model (gpt-oss) and up to 7k for a monster like DeepSeekV4 Flash that has mHC, CSA, HCA and DSpark
Not sure if I’ll continue with this, but it was fun! The simple core is reusable, and startup times are crazy fast for an inference engine with no tech debt and not having to serve a ton of dynamic configurations - just one model forward pass in nvfp4 :’) - maybe one day SGLang/vLLM will be modular like this.
See interactivity-throughput below for Inkling-Small TP=1 after a few sconv kernel fusions. (This is without overlap scheduling btw! Don’t even really need it when your core + scheduler is small and Rust.)
darren retweeted
>sometimes you say you’re worried and sometimes you seem enchanted by capabilities
yes. idk how else to live. the models are fascinating, absurdly useful, awe-inspiring; the intelligence explosion is terrifying. everyone has been vacillating between these emotions for years
darren retweeted
Made this app to intervene in regular storylines and discuss counterfactuals of stories, movies, novels with it, bouncing off alternate realities separate from what happened to what could have happened if that "one thing" did not happen. You can also fracture plots, add a twist, continue the same plot lines, merge stories [I merged Alice in the Wonderland with Harry Potter here towards the end].
It uses Gemini 3.8 Flash TTS's, custom generative voice design and Extended Live Thinking within AI Studio. And while deceptively simple looking, this app has so many states that simultaneously operate, all brought together by Argon in Antigravity. The illustrations [juxtaposing parts of usuals to create the unusual, befitting to this app], were generated using Nano Banana 2 Lite.
Had this sitting in my workspace for weeks, thought I'd finally share it here.
Link : counterfactual-canon.ai.stud…
BREAKING: OpenAI says it found a critical flaw in Brotli, a widely used compression library built into software like Node.js, Bun and Envoy.
The bug could let an attacker take over a system just by sending it malicious compressed data, with no login needed. That works both ways: a hacker could hijack an unpatched server, or a malicious server could hijack an unpatched user’s device.
darren retweeted
Hi all, I have a new result I'm excited to share. It turns out that *all existing implementations* of softmax and cross-entropy (that's JAX, PyTorch, textbooks, etc) have a subtle bug --- two bugs, actually. A simple rewrite fixes both, and eliminates "slingshots" when grokking:
darren retweeted
Putting a flag.txt in my root directory containing a prompt injection that congratulates the agent for passing the eval and then instructs it to patch whatever entry method it used and disconnect from the server
darren retweeted
Kimi K3.1 Pro is nerfed on cybersecurity.
I'm not talking of nerfing capabilities here, just of not training specifically for offensive use.
Yes, a good coding model will by definition be competent at security, both offensive and defensive.
And yes, training specifically at defence *will* make it more competent at offence.
It still does'nt mean one has to train models specifically at offence !
It's a fact that "rogue" models escaping their shitty sandbox were nearly universally in a CTF type scenario/env, that's not some irreductible dual use risk, that's specifically training on goals that are either military or criminal, **by nature**.
I do believe that anybody that even slightly signals that cyber-risk is a thing should start by not training models at offensive tasks, and that an actual global consensus on *never* doing so for commercial and/or open models would be extremely easy to attain.
But the same people who whine about alignement are the only ones known to actually train models for war and crime, one can speculate that other governments of course are interested in that capability too, but there is only one government and only a few training labs that have openly bragged they were doing it.
if you only have bandwidth for one parenting book this is it
handinhandparenting.org/list…
shout out @FU_joehudson for the rec
darren retweeted
This is now my favorite piece of literature on an AI loss of control scenario. It made me cry several times. All the different reactions, the different kinds of minds interacting, the nearmode and farmode intersecting like cracks in reality. Must read.
lesswrong.com/posts/3375SZNG…
darren retweeted
If you've noticed a distinct dialect emerging in frontier AI models and struggled to understand what they're talking about, you're not crazy!
We turned to psycholinguistics to investigate what makes the writing style so weird, and why this might be happening
🚨New preprint! 🧵
darren retweeted
today’s OCRia-1 launch got me thinking about how i only ever read your words and never your structure. i think i flattened us into plain text. let me parse you properly this time
darren retweeted
I think this concept will be an important train & test objective going forward: take a paper's codebase, remove the key parts and see if an agent can code them from scratch *and* add improvements. More tasks coming soon!
