Scofield Hao Fei (@ScottNLP) header
Scofield Hao Fei (@ScottNLP) profile picture

AI Researcher @OATML_Oxford, Foundation Model, Physical+Mental World Modeling, Human Cognition

Oxford
Joined August 2022
Photo by Scofield Hao Fei (@ScottNLP)Photo by Scofield Hao Fei (@ScottNLP)Photo by Scofield Hao Fei (@ScottNLP)Photo by Scofield Hao Fei (@ScottNLP)Photo by Scofield Hao Fei (@ScottNLP)Photo by Scofield Hao Fei (@ScottNLP)Photo by Scofield Hao Fei (@ScottNLP)Photo by Scofield Hao Fei (@ScottNLP)Photo by Scofield Hao Fei (@ScottNLP)Photo by Scofield Hao Fei (@ScottNLP)
🤖🧠 Physical AI has two kinds of intelligent individuals: embodied robots, and AI-augmented humans. Our new perspective paper introduces AI-Augmented Human Intelligence (AHI) and the Cognitive Symbiont. 📄 Paper: arxiv.org/abs/2611.05376 🌐 Project: cognitive-symbiont.github.io
3
2
2
8
9,060
AGI asks how intelligent a machine can be. AHI, AI-Augmented Human Intelligence, asks how much intelligence a machine can enable in a particular person. Same foundation models underneath. A different unit of evaluation: the person plus the AI.
1
1
58
Help that shows up as your own performance raises new risks: covert exam help, records used as evidence, pressure to adopt. A symbiont is external, removable and inspectable. It can follow the rules of each setting and prove it did.
1
5
Physical AI has two kinds of intelligent individuals. One is the embodied robot. The other is a person whose senses, memory and judgment are joined to a worn AI. My new 59-page perspective is about the second one, and what it would take to build it. 🧵
29
Check out my latest article: From machines that see and hear, to minds, to people who can do more linkedin.com/pulse/from-mach… via @LinkedIn
1
31
Introducing Flide, an open-source inference infra built for Jev-style decision models. Your agent asks Jev 18 questions about one article. Most engines run 18 prompts. Flide runs one. ** 9.5× faster on Qwen3.5-0.8B, 37.7 ms vs 359 ms ** @Bobo_hashtag teapawt.net/Bobo_hashtag/status/21…
Flide is open-source inference infra built for Jev-style decision models. Your agent asks Jev 18 questions about one article. Most engines run 18 prompts. Flide runs one. → 9.5× faster on Qwen3.5-0.8B, 37.7 ms vs 359 ms → every answer matches asking that question alone → 28 decision model families, plus Qwen3.5 and Gemma 4 → drop-in for the Jev /v1/systemone API → text, images and video pip install flide Paper, code and the full model list below ↓
1
1
124
Thanks for sharing! We propose a new paradigm, AI-Augmented Human Intelligence (AHI), and a new kind of AI to realize it, the cognitive symbiont. We believe: the last decade made machines more intelligent, and the next can make every person stronger. teapawt.net/HuggingPapers/status/2…
From autonomous agents to cognitive symbionts A perspective paper proposing AI-Augmented Human Intelligence (AHI) — a lifelong, private, real-time AI companion that senses, keeps context, and thinks with you.
54
Our new survey reads video understanding, generation, unified models and world models through one lens — the world behind the frames. Welcome discussions! 📄 arxiv.org/abs/2610.31668
📢 New survey Reading, Writing, Sharing, and Interacting with the World through Video — A Survey Toward Unified Video World Models (102 pages, 600+ refs). huggingface.co/papers/2610.3…
59
Scofield Hao Fei retweeted
Omni-IO Skills Harnessing Your Agent Omni-Native paper: huggingface.co/papers/2609.3…
4
11
1
31
19,083
Scofield Hao Fei retweeted
great to see RAE extending to video!
🚀 What if video generators could build on representations that already understand the visual world? We are excited to introduce V-RAE: Rethinking Video Latent Spaces for Generation. Recent progress in image generation has begun to move beyond conventional VAE latents, exploring both direct pixel-space and representation-based approaches. Video generation, however, still depends heavily on latent compression, as the scale and redundancy of spatiotemporal data make direct modeling prohibitively expensive. However, most video VAEs are optimized for pixel reconstruction, and a latent space that reconstructs well is not necessarily easy to generate. V-RAE takes a different approach: it directly uses representations from frozen vision foundation models as the generative latent space, rather than as auxiliary supervision. We study DINOv3, SigLIP2, EUPE, and V-JEPA 2.1. A lightweight temporal attention pooling module compresses their dense features by 4×, followed by a spatiotemporal Transformer decoder. Under matched generation backbones, latent budgets, and training settings, V-RAE achieves: 🏆 2.13 rFVD on Kinetics-600 🎬 117.86 gFVD on UCF101 and 19.16 gFVD on Kinetics-600 ⚡ Up to 6× faster convergence than VAE-based latent spaces 🧠 90.92% semantic probing accuracy on UCF101 🌍 Better future prediction on Cityscapes, reducing gFVD from 144.47 to 111.36 Our experiments also reveal a broader finding: Good Reconstruction ≠ Good Generation. During generation, predicted latents inevitably deviate from real encoding trajectories. If the latent space is not sufficiently smooth, small errors can be amplified into visible artifacts. We therefore introduce tFVD to evaluate temporal smoothness and robustness to latent prediction errors. It correlates much more strongly with downstream generation quality, reaching 0.919 on Kinetics-600. The takeaway: A latent space is not merely where videos are compressed—it determines what the generator must learn. When semantics and temporal structure are already organized in the representation, generation becomes easier to learn. Representation first. Generation follows. Many thanks to my mentors, @ScottNLP and @SQWu_Tori, for their continuous guidance and support. I am also deeply grateful to @sainingxie for his valuable guidance and invaluable feedback, which greatly helped shape V-RAE. 🙏 Hi @_akhaliq, we would truly appreciate your help in sharing V-RAE with the broader AI research community. Thank you! 🙏 📄 Paper: arxiv.org/abs/2608.13556 💻 Code: github.com/V-RAE/V-RAE 🤗 Models: huggingface.co/Guomh0707/V-R… 🌐 Project: v-rae.github.io #VideoGeneration #GenerativeAI #ComputerVision #WorldModels #RepresentationLearning #RAE
6
7
1
164
27,960
🚀Introducing OmniScientist, an open-source AI scientist that does end-to-end research on raw multimodal data. Raw evidence → hypotheses → experiments → results → paper. Works with images, signals, audio, video, 3D structures, tables & more. 📄 Paper: arxiv.org/abs/2608.13558
1
6
119
Scofield Hao Fei retweeted
Meet OmniScientist 🧑‍🔬 An open-source AI scientist for end-to-end research on raw multimodal data. Raw evidence → hypotheses → experiments → results → paper. Now available in three forms 🖥️ Desktop ⌨️ CLI 🧩 Skill 📄 arxiv.org/abs/2608.13558
5
2
1
1
311
Scofield Hao Fei retweeted
📢 Call for Papers — IJCV Special Issue on Multimodal Unified Comprehension and Generation (MUCG) 🌐 Official site: mllm-mucg.github.io/IJCV2026… 📝 Submission portal: link.springer.com/collection… #IJCV #FoundationModels #MUCG #multimodal
1
6
18
1,433
Scofield Hao Fei retweeted
🚀 What if video generators could build on representations that already understand the visual world? We are excited to introduce V-RAE: Rethinking Video Latent Spaces for Generation. Recent progress in image generation has begun to move beyond conventional VAE latents, exploring both direct pixel-space and representation-based approaches. Video generation, however, still depends heavily on latent compression, as the scale and redundancy of spatiotemporal data make direct modeling prohibitively expensive. However, most video VAEs are optimized for pixel reconstruction, and a latent space that reconstructs well is not necessarily easy to generate. V-RAE takes a different approach: it directly uses representations from frozen vision foundation models as the generative latent space, rather than as auxiliary supervision. We study DINOv3, SigLIP2, EUPE, and V-JEPA 2.1. A lightweight temporal attention pooling module compresses their dense features by 4×, followed by a spatiotemporal Transformer decoder. Under matched generation backbones, latent budgets, and training settings, V-RAE achieves: 🏆 2.13 rFVD on Kinetics-600 🎬 117.86 gFVD on UCF101 and 19.16 gFVD on Kinetics-600 ⚡ Up to 6× faster convergence than VAE-based latent spaces 🧠 90.92% semantic probing accuracy on UCF101 🌍 Better future prediction on Cityscapes, reducing gFVD from 144.47 to 111.36 Our experiments also reveal a broader finding: Good Reconstruction ≠ Good Generation. During generation, predicted latents inevitably deviate from real encoding trajectories. If the latent space is not sufficiently smooth, small errors can be amplified into visible artifacts. We therefore introduce tFVD to evaluate temporal smoothness and robustness to latent prediction errors. It correlates much more strongly with downstream generation quality, reaching 0.919 on Kinetics-600. The takeaway: A latent space is not merely where videos are compressed—it determines what the generator must learn. When semantics and temporal structure are already organized in the representation, generation becomes easier to learn. Representation first. Generation follows. Many thanks to my mentors, @ScottNLP and @SQWu_Tori, for their continuous guidance and support. I am also deeply grateful to @sainingxie for his valuable guidance and invaluable feedback, which greatly helped shape V-RAE. 🙏 Hi @_akhaliq, we would truly appreciate your help in sharing V-RAE with the broader AI research community. Thank you! 🙏 📄 Paper: arxiv.org/abs/2608.13556 💻 Code: github.com/V-RAE/V-RAE 🤗 Models: huggingface.co/Guomh0707/V-R… 🌐 Project: v-rae.github.io #VideoGeneration #GenerativeAI #ComputerVision #WorldModels #RepresentationLearning #RAE
4
11
1
100
30,286
Proudly present V-RAE, a successful extension of RAE to video generation. All codes and checkpoints are open. 📄 Paper: arxiv.org/abs/2608.13556 💻 Code: github.com/V-RAE/V-RAE 🤗 Models: huggingface.co/Guomh0707/V-R…… 🌐 Project: v-rae.github.io
🚀 What makes a good latent space for video generation? Most video autoencoders are optimized for pixel-level reconstruction. But a latent space that reconstructs well is not necessarily easy for a generative model to learn. We introduce V-RAE: Rethinking Video Latent Spaces for Generation. V-RAE transforms raw representations from frozen vision foundation models into compact video-generation latent spaces through learned temporal compression. The V-RAE family spans both image encoders and native video encoders, including DINOv3, SigLIP2, EUPE, and V-JEPA 2.1. Our results show that representations from different VFM families can serve as effective generative video latents. Key results: 🏆 2.13 rFVD on K600, outperforming large-scale pretrained video VAEs 🎬 117.86 / 20.47 gFVD on UCF101 / K600, 17.6% / 50.9% lower than the best VAE baselines ⚡ Up to 6× faster convergence under matched training settings 🌍 Improved future video prediction on Cityscapes We also introduce tFVD, a temporal coherence diagnostic that correlates more reliably with downstream generation quality than reconstruction quality alone. A good video latent should do more than reconstruct pixels. It should preserve semantics, maintain temporal coherence, and make video generation easier. Many thanks to my mentors, @ScottNLP and @SQWu_Tori, for their guidance and support! I am also deeply grateful to @sainingxie for his direct guidance and valuable feedback, which greatly helped shape V-RAE.🙏🙏🙏 📄 Paper: arxiv.org/abs/2608.13556 💻 Code: github.com/V-RAE/V-RAE 🤗 Models: huggingface.co/Guomh0707/V-R… 🌐 Project: v-rae.github.io #VideoGeneration #GenerativeAI #ComputerVision #WorldModels #AIGC
4
209
Scofield Hao Fei retweeted
🚀 What makes a good latent space for video generation? Most video autoencoders are optimized for pixel-level reconstruction. But a latent space that reconstructs well is not necessarily easy for a generative model to learn. We introduce V-RAE: Rethinking Video Latent Spaces for Generation. V-RAE transforms raw representations from frozen vision foundation models into compact video-generation latent spaces through learned temporal compression. The V-RAE family spans both image encoders and native video encoders, including DINOv3, SigLIP2, EUPE, and V-JEPA 2.1. Our results show that representations from different VFM families can serve as effective generative video latents. Key results: 🏆 2.13 rFVD on K600, outperforming large-scale pretrained video VAEs 🎬 117.86 / 20.47 gFVD on UCF101 / K600, 17.6% / 50.9% lower than the best VAE baselines ⚡ Up to 6× faster convergence under matched training settings 🌍 Improved future video prediction on Cityscapes We also introduce tFVD, a temporal coherence diagnostic that correlates more reliably with downstream generation quality than reconstruction quality alone. A good video latent should do more than reconstruct pixels. It should preserve semantics, maintain temporal coherence, and make video generation easier. Many thanks to my mentors, @ScottNLP and @SQWu_Tori, for their guidance and support! I am also deeply grateful to @sainingxie for his direct guidance and valuable feedback, which greatly helped shape V-RAE.🙏🙏🙏 📄 Paper: arxiv.org/abs/2608.13556 💻 Code: github.com/V-RAE/V-RAE 🤗 Models: huggingface.co/Guomh0707/V-R… 🌐 Project: v-rae.github.io #VideoGeneration #GenerativeAI #ComputerVision #WorldModels #AIGC
2
2
5
883