PhD @UCLANLP | prev. PE-AV & SAM @FAIR | Perception & Agents | On the job market

Joined October 2021
🚀 Introducing OpenVLHarness: Turn any LLM into an evidence-based reasoner that thinks and codes in visual space. 📈 GPT-6 Astra gains +9.9 points on average over 22 datasets. ⚡ Beats Codex and other VL harnesses by 7.7–8.5 points, at ≤37% of their cost. 🧵 More details ↓
1
3
19
546
Cheng-Fu Yang retweeted
Apple and UCLA present OpenVLHarness A training-free harness that gives multimodal LLMs a capability layer and persistent multimodal memory, boosting average scores by 9.7-12.6 points across 22 benchmarks.
6
4
5
57
4,047
Cheng-Fu Yang retweeted
Fine-tuning can improve a target skill while weakening others. Can we improve a fine-tuned LLM by undoing part of its fine-tuning? Introducing 🚀SPACE🚀: a post-hoc method that improves average performance both within and beyond the training domain. No extra training. No data. Just the pre- and post-SFT checkpoints for calibration with one single step of code. 1/n
4
17
1
47
2,226
Thanks @_akhaliq for sharing our work! 👇 Try our interactive demo with your own LLM! 🎬 Demo: huggingface.co/spaces/uclanl… 🔗 Website: openvlharness.github.io 📄 Paper: arxiv.org/abs/2610.26860 💻 Code: github.com/OpenVLHarness/Ope…
OpenVLHarness Real World Visual Agentic Tool-Use with Multimodal Memory paper: huggingface.co/papers/2610.2…
1
7
401
Cheng-Fu Yang retweeted
Huge thanks to @rohanpaul_ai for sharing our work! We’re excited to share LEAP: a new agentic framework for formal theorem proving! LEAP mimics a human mathematician's workflow by constructing proof graph via informal blueprints, then searching and backtracking using compiler feedback. The results? 
🏆 Solves 12/12 Putnam 2025 problems
📈 Raises Gemini-3.1-pro's performance from 10% to 70% 
Read the paper: arxiv.org/abs/2606.03303
Another great paper from Google. Shows general LLMs can solve formal math by planning proofs and checking each step. Raised general LLM performance from under 10% to 70%. A general LLM failed badly when asked to write full formal proofs in 1 try, but became much stronger when it planned, split the work into smaller claims, reused past claims, and learned from Lean’s feedback. The paper shows the weakness was not just the model’s math ability, but the way it was being used - the absence of structured interaction with a verifier. The key idea is that the model does not try to write one giant perfect proof at once, because that usually fails on long and tricky problems. Instead, LEAP stores the proof as a graph of goals and subgoals, so useful lemmas can be reused instead of rediscovered every time. The authors tested LEAP on Putnam 2025 and a new Lean benchmark built from 60 IMO-style problems, where ordinary one-shot proof writing did very poorly. LEAP solved all 12 Putnam 2025 problems and raised general LLM performance on the Lean IMO benchmark from under 10% to 70%. ---- Link – arxiv. org/abs/2606.03303 Title: "LEAP: Supercharging LLMs for Formal Mathematics with Agentic Frameworks"
1
10
16
3,702
Cheng-Fu Yang retweeted
I integrated our recent work, Spectral Progressive Diffusion a.k.a. SPEED (arxiv.org/abs/2605.18736), with the open-source @ideogram_ai model released yesterday. Turns out our method works out of the box and speeds up inference by up to 1.6× while preserving the high image quality! [1/4]
8
20
2
134
11,723
Cheng-Fu Yang retweeted
🌟 Excited to share Ctrl-R, accepted as an ICML 2026 Spotlight! Specify a target reasoning structure, Ctrl-R can enforce it during rollout, and retain accurate importance-sampling weights for principled policy optimization. paper: arxiv.org/abs/2603.01641 🧵[1/5]
3
23
104
9,526
Excited to share what I worked on this summer! PE-AV — a new multimodal encoder trained on ~100M audio–video-text pairs! ✔️ Unified Audio–Video–Text embeddings ✔️ SOTA across audio, video & AV benchmarks 📄 Paper: go.meta.me/e541b6 💻 code: shorturl.at/X0Win
We’re open-sourcing Perception Encoder Audiovisual (PE-AV), the technical engine that helps drive SAM Audio’s state-of-the-art audio separation. Built on our Perception Encoder model from earlier this year, PE-AV integrates audio with visual perception, achieving state-of-the-art results across a wide range of audio and video benchmarks. Its native multimodal support can assist people in everyday tasks, including sound detection and richer audio-visual scene understanding. 🔗 Read the paper: go.meta.me/e541b6 🔗 Download the code: go.meta.me/7fbef0
5
275
Cheng-Fu Yang retweeted
We released SAM Audio 🎧 and PE-AV (Perception Encoder AudioVisual) 👀👂. PE-AV representation powers SAM Audio’s data ⚙️ and SAM Audio Judge 🧑‍⚖️. 📄Paper: go.meta.me/e541b6 💻 code: shorturl.at/X0Win 🔗 SAM Audio: shorturl.at/Oa9EO But it doesn’t stop there ⤵️
We’re open-sourcing Perception Encoder Audiovisual (PE-AV), the technical engine that helps drive SAM Audio’s state-of-the-art audio separation. Built on our Perception Encoder model from earlier this year, PE-AV integrates audio with visual perception, achieving state-of-the-art results across a wide range of audio and video benchmarks. Its native multimodal support can assist people in everyday tasks, including sound detection and richer audio-visual scene understanding. 🔗 Read the paper: go.meta.me/e541b6 🔗 Download the code: go.meta.me/7fbef0
2
1
1
114
Very excited to share what I've been working on over the past few months! Try out the zero-shot SAM3 Agent with your own choice of images, complex language prompts and MLLM backbone: github.com/facebookresearch/… Here is one of my favorite examples: Prompt: "The cows behind the car"
Meet SAM 3, a unified model that enables detection, segmentation, and tracking of objects across images and videos. SAM 3 introduces some of our most highly requested features like text and exemplar prompts to segment all objects of a target category. Learnings from SAM 3 will help power new features in Instagram Edits and Vibes, bringing advanced segmentation capabilities directly to creators. 🔗 Learn more: go.meta.me/591040
1
5
2
37
4,477
I'll be presenting our paper: Verbalized Representation Learning (VRL) on Tuesday 11-2 PM at #ICCV2025 Come see how models can automatically generalize compositionally to new knowledge — without labeled data or human effort!
📣 New Paper: Verbalized Representation Learning (VRL) VRL bridges prompt engineering and representation learning to enable automatic interpretable feature extraction — all without gradient descent! 🔥 +29% over SOTA 📊 95% less data arxiv.org/abs/2411.18651 @uclanlp (1/n)
7
1
21
12,166
I'll be presenting on Thursday 11-1 PM EST at #COLM2025. Come see how we built self-evolving guardrails that address shifting norms and user feedback!
📣 Our paper “Customize Multi-modal RAI Guardrails with Precedent-based Predictions” has been accepted to #COLM2025! 🎉 TL;DR: - No hand-crafting detailed rubrics - Model auto-discovers and evolves its own rubrics - Plug and play - work with OSS and commercial model (1/N)
2
6
576
📣 Our paper “Customize Multi-modal RAI Guardrails with Precedent-based Predictions” has been accepted to #COLM2025! 🎉 TL;DR: - No hand-crafting detailed rubrics - Model auto-discovers and evolves its own rubrics - Plug and play - work with OSS and commercial model (1/N)
1
1
2
12
1,329
Results show that our method • Outperforms standard SFT and ICL across RAI categories • Scales well as the number of data grow • More robust to policy expansion (4/N)
1
1
103
Check out the full details in our paper: 📄 Paper: arxiv.org/abs/2507.20503 💻 Code: github.com/joeyy5588/Customi… Enjoyed collaborating with @c_christodoulop @ThanhDTran88 @rahul1987iit @kaiwei_chang !
1
68