Pinned Tweet
Releasing ViTok-v2: open-source ViT auto-encoder codebase + pretrained weights
Train your own ViT auto-encoder on any streamed (hf://) or local webdataset. NaFlex pipeline handles any resolution and aspect ratio
Includes reproduced 350M and 4.5B models weights competitive at 256p, SOTA at high-res (512p+)
Philippe Hansen-Estruch retweeted
What makes reinforcement learning efficient for training generalist robot policies? A key ingredient is the action representation: what choices do we give the learner to explore? A suitable action representation can make the learning problem substantially easier. 🧵(1/6)
Philippe Hansen-Estruch retweeted
“We’re friends”
“We’re just friends”
One tiny word can change how a conversation continues. LLMs pick up on this 👀
We show this through human-LLM comparisons, generation experiments, and mech interp methods.
1/n
Philippe Hansen-Estruch retweeted
how to balance MoE load with quantile balancing in ~2 minutes.
now it just needs 3blue1brown’s voice.
“in QB we trust”
Philippe Hansen-Estruch retweeted
💥 The multiplication saga continues! It turns out, some frontier LLMs do struggle with simple multiplications.
Unlike GPT-6-Astra, Fable 5.1 with low reasoning has a blind spot where the accuracy of, say, 5x6 multiplications drops to ~0%!
The failure mode is very interesting: the 0% zone corresponds to a failure of adaptive thinking. The model decides that it shouldn't produce any reasoning and outputs the (wrong) answer directly. For larger multiplications, however, the model realizes it should produce a CoT first and then it gets the answer correctly. For very large multiplications (close to 20x20), the accuracy degrades again.
How many such simple blind spots do frontier LLMs still have? This is why slowing down and carefully studying generalization and safety is the top priority...
1/2
We’re sharing a solution to the Navier-Stokes Millennium Prize Problem, one of the deepest problems at the frontier of mathematics.
The proof was produced by a group of agents, using an OpenAI next-generation model significantly more capable than GPT-6 Astra.
The problem concerns whether the description of smooth three-dimensional fluid motion modeled by the Navier-Stokes equations can break down. It has remained unresolved for roughly 90 years.
Philippe Hansen-Estruch retweeted
Open-source video generation is now faster than playback without compromising quality.
Introducing Video Delta Net (VDN): hybrid attention for live text-to-video with near-lossless quality.
VDN accelerates Minimax-H3 by 75 - 90 x, generating 14 seconds of 768p video in 11 seconds on 8× NVIDIA B200 GPUs.
Checkpoints + training/inference code + Technical Blog ⬇️
(1/6)
We're releasing LAION-BVD: a 10-million-hour open video dataset for multimodal pre-training.
- 1.3B video URLs from CommonCrawl
- 80M downloaded videos
- 10M video hours
- 55M captioned clips
- 300M frame-caption pairs
🌐: projects.laion.ai/bvd/
Philippe Hansen-Estruch retweeted
🚀 What if video generators could build on representations that already understand the visual world?
We are excited to introduce V-RAE: Rethinking Video Latent Spaces for Generation.
Recent progress in image generation has begun to move beyond conventional VAE latents, exploring both direct pixel-space and representation-based approaches. Video generation, however, still depends heavily on latent compression, as the scale and redundancy of spatiotemporal data make direct modeling prohibitively expensive.
However, most video VAEs are optimized for pixel reconstruction, and a latent space that reconstructs well is not necessarily easy to generate.
V-RAE takes a different approach: it directly uses representations from frozen vision foundation models as the generative latent space, rather than as auxiliary supervision.
We study DINOv3, SigLIP2, EUPE, and V-JEPA 2.1. A lightweight temporal attention pooling module compresses their dense features by 4×, followed by a spatiotemporal Transformer decoder.
Under matched generation backbones, latent budgets, and training settings, V-RAE achieves:
🏆 2.13 rFVD on Kinetics-600
🎬 117.86 gFVD on UCF101 and 19.16 gFVD on Kinetics-600
⚡ Up to 6× faster convergence than VAE-based latent spaces
🧠 90.92% semantic probing accuracy on UCF101
🌍 Better future prediction on Cityscapes, reducing gFVD from 144.47 to 111.36
Our experiments also reveal a broader finding:
Good Reconstruction ≠ Good Generation.
During generation, predicted latents inevitably deviate from real encoding trajectories. If the latent space is not sufficiently smooth, small errors can be amplified into visible artifacts.
We therefore introduce tFVD to evaluate temporal smoothness and robustness to latent prediction errors. It correlates much more strongly with downstream generation quality, reaching 0.919 on Kinetics-600.
The takeaway:
A latent space is not merely where videos are compressed—it determines what the generator must learn.
When semantics and temporal structure are already organized in the representation, generation becomes easier to learn.
Representation first. Generation follows.
Many thanks to my mentors, @ScottNLP and @SQWu_Tori, for their continuous guidance and support.
I am also deeply grateful to @sainingxie for his valuable guidance and invaluable feedback, which greatly helped shape V-RAE. 🙏
Hi @_akhaliq, we would truly appreciate your help in sharing V-RAE with the broader AI research community. Thank you! 🙏
📄 Paper: arxiv.org/abs/2608.13556
💻 Code: github.com/V-RAE/V-RAE
🤗 Models: huggingface.co/Guomh0707/V-R…
🌐 Project: v-rae.github.io
#VideoGeneration #GenerativeAI #ComputerVision #WorldModels #RepresentationLearning #RAE
Philippe Hansen-Estruch retweeted
Replying to @HuggingPapers
Small clarification: TIPS has been out since 2024, we just uploaded it to HF so it's easier for people to use
TIPSv2 is also out, see below :)
huggingface.co/collections/g…
Philippe Hansen-Estruch retweeted
Replying to @sriramk
I am a famous AI researcher who lives in DC. I used to have a security clearance and do work for the government, but my clearance got taken away because I’m transgender.
Philippe Hansen-Estruch retweeted
Yesterday, my OpenAI collaborator and I gave a detailed talk on the Huggingface incident, our models creating "the message board", model misalignment, and more.
youtube.com/watch?v=87DyyMV0…
I hope it can answer a lot of the questions folks have, and we will release a full detailed postmortem at a later time!
Philippe Hansen-Estruch retweeted
still unclear to me why the Mag7 need to cut cap-ex if revenues can grow double digits for the next few years
Philippe Hansen-Estruch retweeted
My increasingly strong opinion is that compute is the biggest moat in AI.
Scientific breakthroughs don’t stay secret for long. Papers get published, researchers move, ideas leak, open source catches up. Give it enough time and almost everyone has access to roughly the same algorithms.
What doesn’t diffuse nearly as fast is the ability to run 10x bigger training runs, collect 10x more RL experience, serve billions of inference tokens, and iterate faster because you can afford thousands of experiments in parallel.
Scaling compute is brutally hard. It’s GPUs, power, substations, networking, cooling, land, permits, supply chains… and hundreds of billions of dollars. You can’t just decide to build that overnight.
The more I look at the industry, the more it feels like the frontier is less about having one genius idea and more about who can compound compute the fastest. Scaling laws reward whoever can keep scaling. That’s a very hard moat to replicate imo!
BREAKING: Nvidia is in talks with OpenAI to guarantee $250 billion in financing for a data center in Ohio.
Details include:
1. The guarantees from Nvidia would help OpenAI lease a 10GW project that SoftBank is developing in Ohio
2. In total, the project could cost more than $500 billion, the largest data center project ever announced
3. The power for the project is controlled by the US government and funded separately by Japan under a recent trade deal
4. Commerce Secretary Lutnick is reportedly involved in deciding who will get the power
The data center buildout just hit a whole new level.
Philippe Hansen-Estruch retweeted
Releasing the model weights and technical report of Kimi K3.
Kimi K3 is our most capable model: a 2.8T MoE model with native visual understanding and a 1M-token context window.
New model architecture: 2.5x the intelligence per unit of compute, not just more params.
Alongside Kimi K3, we're opening up more of the stack behind it — high-performance attention kernels, MoE communication library, and infrastructure for running agent environments at scale.
Model weights: huggingface.co/moonshotai/Ki…
Tech report: github.com/MoonshotAI/Kimi-K…
Tech blog: kimi.com/blog/kimi-k3
Philippe Hansen-Estruch retweeted
We've open-sourced FlashKDA, our high-performance CUTLASS-based implementation of Kimi Delta Attention kernels.
It delivers 1.72×–2.22× prefill speedup over the flash-linear-attention baseline on H20, and works as a drop-in backend for flash-linear-attention.
Explore on GitHub:
github.com/MoonshotAI/FlashK…
Philippe Hansen-Estruch retweeted
Replying to @mooncat_is
I don't think it's a lack of imagination. I also used to work at Anthropic, think trends will continue, and used to agree with this. But I've changed my mind and now disagree with this take.
Open models are already capable enough to do what you described. For example, I used Opus 4.6 to gain access to other folks medical records, hijack bank accounts, etc. back in February. GLM 5.1 is more capable than Opus 4.6 in most pentesting environments, and it came out in April.
Despite capable open-weight models existing, the sketchier folks I know are still using a Claude Code or Codex subscription for hacking. (Even well-resourced groups in other countries! They use the grey/black market of discounted Ant/OAI subscription tokens sold through resellers.) So I see most of the materialized risk here as still coming from Anthropic and OpenAI; safeguards aren't sufficient to stop a moderately dedicated actor.
The groups I know who are using open-weight models are legitimate offensive security companies. They won't break the rules to use subscription-based pricing, the open-weight models are more reliable in that they don't require specific jailbreaks nor hit classifiers, and the labs use massive partnerships or spend as a prereq for lowering classifiers/safeguards. I know of three legitimate groups running GLM 5.2 as their primary model.
That last part applies for Anthropic, too: I know of two instances where two different Anthropic GTM people used large comitted spend contracts as a prereq for lowering safeguards, and I directly witnessed one.
On the inside, I know the narrative and intent is genuinely about safety. But from the outside, Anthropic-the-system seems to be optimizing for revenue and control/power, isn't diffusing capabilities to defenders, and also doesn't have adequate safeguards to prevent misuse from dedicated bad actors.
As a result, I now lean towards a future where capable open models are freely available (at least for cyber, bio is harder); I don't trust Anthropic or other frontier labs to handle this sufficiently well without diffused capabilties given what I've seen so far.
Philippe Hansen-Estruch retweeted
For my first post, I’m sharing a letter @NVIDIA signed on why open models matter.
AI will transform every industry, power every company, and be built by every country.
Open models strengthen safety and cybersecurity, accelerate innovation and diffusion, and enable sovereignty.
The world needs both frontier closed models and frontier open models.
images.nvidia.com/pdf/Open-W…
































