Shauray

I tried @bria_ai_ FIBO2.0 edit and combined that with qwen2.1 for boxed edits and FIBO for full edits makes a really good pixel perfect multi turn edit solution, also FIBO's page says they have a multi-turn edit lora in oven ! btw here - github.com/shauray8/QwenEdit (rn its monkey patched onto sglang)

@krea_ai's K3 launch videos gave me extreme fomo, so until they release the weights I made a little multi-turn edit tool for qwen2.1. I think its one of the most underexplored part of the arch that the edit prefix can be cached and multi-turn can share that prefix turn over turn, so each new edit only computes what's new. 0.9s per 1k edit, ~4.5s 2k box edits tested on rtx pro 6k. multi turn scandinavian defense idea comes from @awdii_ repo: github.com/shauray8/QwenEdit more details below.

some more measurements continuing from the prev post. running the dense trajectory and at every sparse eligible call also runs the sparse kernel on the same Q/K/V and record ||O_sparse - O_dense||/||O_dense||. the dense output goes forward so errors dont compound and every variant sees identical inputs. what i saw was, moving two layers between dense and sparse changes the video about as much as turning sparsity on. at SHRINK=1 no variant wins everywhere. the mass term blows up in the early and late layers, routing only wins the back third, both fixes mostly win the middle layers. best layers for shrink landed between .25 and .75 well below what we measured earlier in the series of posts leading up to this. what dominates the error is the direction of a skipped block's contribution, not its size and in frozen blocks the block mean points far off from the true one. a full mass correction just scales up the wrong vector. at low shrink the layers that were losing start gaining. until now what I have is a combination of this new variant that we came up with in the prev posts + SHRINK + ρ target per layer, chosen by a cost aware rule that only accepts REM on a layer when its error gain beats the extra kernel time. on the kernel - 8% lower attn error at the same speed. [more generated diagrams]

finally a pixel space model, i hope this stays !

Meet Iris-3B: a pixel-space T2I model and general vision learner that generates every pixel directly. No VAE. Pre-trained from scratch, 3B, open weights. We also explore its generative prior on detail-critical vision tasks: depth estimation and image restoration. Fully open: weights, code and demo (Apache 2.0) speridlabs.com/research/iris

picking up from the overhead from the quoted post. SHRINK scales the correction g_eff = SHRINK.g(σ). so at shrink=0 the kernel still computes and reads g but selects exactly baseline's blocks, so that run separates the cost of computing all the new stuff from the cost of REM's selection with REM moved into the TMA frwd, the selection cost is almost 0. the scattered load penalty in the last post came from the first impl part, not from REM picking frozen blocks. what's left is a fixed per-call cost which is - build g and read it in the main loop. so g builds in one fused kernel, g tiles TMA-loaded alongside the centroid tiles. around 1 ms per call on the kernel. τ stops being an input too. each layer ships a density target ρ, and τ* is the ρ-quantile of the routing z-scores, solved on device at the layer's first sparse step and cached for further use. that removed the τ drift from the last post, also every comparison is density matched.

on the wins the kernel changes should be small, one extra accum in the reduction kernel (kvar = mean(k²) - k_dash², while the K tile is already loaded) and in routing. the kernel doesn't re-center the routing threshold, so τ's meaning shifts τ=1.0 lands at ρ=0.599, and matching baseline's ρ=0.25 needs τ≈2.5 (as mentioned in the quoted post). Timings at matched density (38,208 tokens × 56 heads) baseline = 53.85 ms, (mass+routing) fix = 57.98 ms +7.7% overhead (from computing the correction) need to see what can be done here. also baseline's mask is a few long contiguous runs (3 streaming mem reads), my frozen block REM is more scattered cold loads, so mass+routing does save 19% density but costs 19.7% time, essentially net zero if not worst, lets see if i can fix up.