“there are no new ideas” 😅
To channel my inner Schmidhuber, we found this softmax "bug" ~1.5 years ago and published in this paper arxiv.org/pdf/2603.11682
In reality, it's pretty hard to come up with a training scenario where this matters because gradient w.r.t. highly likely tokens is tiny anyway. Nonetheless I recommend fixing it -- one less thing to worry about! Check out @jon_barron 's work for an elegant fix!
i wish i had read a spectral condition for feature learning before i bashed my head in trying to read tensor programs. this paper is beautiful
the automatic switching that airpods do between devices is one of the worst features. how do i get it to stop just switching which device my airpods connect to??
something i really appreciate about the humanoid animation subfield is the ability to make beautiful visualizations (after all, you can just render the simulation) 😆
$1.5T valuation and you still need to manually tell claude to use latex instead of unicode characters for math
a cool way to train idm’s online from experience! they make an important observation that naively generating random play data is usually not sufficient for covering the observation space, so you need to do some more intelligent exploration, and it turns out a good video planner with imperfect idm is a good candidate!
ali retweeted
Very excited about this one: a completely new representation learning objective, with a proper theory of what it converges to: the representation of an image converges to features such that their dot product with another image's representation is their pointwise mutual information! @BaadeAlan knocked it out of the park :) (1/n)
1/ The tokenizer is a new scaling axis!
We introduce an approach for diffusable representation learning that's principled, scalable, and works when DINO/LeJEPA collapse.
We get joint tokenizer+diffusion scaling laws and it's compute-optimal vs raw diffusion
w/ @drfeifei @GordonWetzstein @jcjohnss @ericryanchan @vincesitzmann @KyleSargentAI
Thread, Paper, and Blogpost below
alan really is a visionary in representation learning! i was really surprised to learn that contrastive learning implicitly factorizes pointwise mutual information, and more-so that we can estimate this scalably using generative models! this is a milestone for scalable tokenizers
1/ The tokenizer is a new scaling axis!
We introduce an approach for diffusable representation learning that's principled, scalable, and works when DINO/LeJEPA collapse.
We get joint tokenizer+diffusion scaling laws and it's compute-optimal vs raw diffusion
w/ @drfeifei @GordonWetzstein @jcjohnss @ericryanchan @vincesitzmann @KyleSargentAI
Thread, Paper, and Blogpost below
ali retweeted
1/ The tokenizer is a new scaling axis!
We introduce an approach for diffusable representation learning that's principled, scalable, and works when DINO/LeJEPA collapse.
We get joint tokenizer+diffusion scaling laws and it's compute-optimal vs raw diffusion
w/ @drfeifei @GordonWetzstein @jcjohnss @ericryanchan @vincesitzmann @KyleSargentAI
Thread, Paper, and Blogpost below