Software developer
Australia
Joined January 2008
- Tweets654
- Following585
- Followers431
- Likes34K
David A Roberts retweeted
for(float i,z,d,f;i++<1e2;o+=vec4(4,6,8.+z,0)/f-min(dFdx(z)*r.y+z,0.)/exp(d*d/.1)){vec3 p=z*(FC.rgb*2.-r.xyy)/r.y,c=p;p.z+=8.;c.z*=3.;for(f=1.;f++<9.;c+=sin(c.yzx*f+z+t*.5)/f);z+=min(f=.1+abs(.2*c.y+abs(p.y+.8)),d=max(length(p)-3.,.9-length(p-vec3(-1,1,3))))/7.;}o=tanh(o/2e3);
David A Roberts retweeted
Compute.toys finally got the Slang compiler integrated, which means its finally time to ditch WGSL 🎉🎉
I've ported my FFT to it, coding in it is much nicer. compute.toys/view/1922
Should try something with its autodiff next.
David A Roberts retweeted
Made a mixed radix FFT implementation in compute.toys compute.toys/view/1915
It is neat, but I still wonder how do you properly handle large prime factor size FFT's?
React Native compute.toys viewer for iOS/Android
David A Roberts retweeted
Optical illusions with diffusion models. There are so many good gifs on this page but honestly I would like several million more. dangeng.github.io/visual_ana…
David A Roberts retweeted
Made a really weird fractal flame based on a mandelbulb, looks like a moving stardust cloud. (oh the codec aint gonna like this video...)
compute.toys/view/728
Keeping the rng seed constant and varying the params makes the points move in interesting ways
Likewise when people ask whether LLMs *have* a world model, they're fundamentally getting it inside out. An LLM *is* a world model. RLHF/SFT tries to collapse it down to a single character, but it's still a model of the world focused on that character, rather than vice versa.
The disconcerting thing about LLMs is that the strange loop doesn't quite close in the same way as ours do, which makes identifying the world model with one of the characters *inside* of it rather more fraught.
This seems promising, porting code between languages is one of GPT-4's stronger skills (it's a kind of translation task after all) and they're able to formally verify its correctness to keep iterating until it gets it right.
In our latest blog, Galois Research Engineer Adam Karvonen writes about his experiment applying GPT-4 to the task of refolding macros into the Rust program.
galois.com/blog/2023/09/usin…
TinyStories-33M is an interesting model, 3x smaller than GPT2-Small, with only 4 layers. It doesn't appear to have any induction heads, but it can solve the IOI task. The IOI circuit seems to be simpler than the one in GPT2, with 4 groups of heads over the 4 layers. 🧵
This Post is from an account that no longer exists. Learn more
The lack of induction heads is obvious from the fact that the model is unable to predict repeated sequences of random tokens. No heads cleanly display the characteristic induction attention pattern.
However, L2H4 very weakly attends to the token following the previous duplicate.
