Yue Fan

Excited to be at #CVPR2026 in Denver! This week: 🏆 Best Paper Award Runner-Up (WMAS) 🎤 Two keynote talks 📌 Two conference posters Across these projects, we’re pursuing a common vision for Agentic AI: agents that can reason in latent space, develop situated awareness of the real world, and continuously evolve from experience. Looking forward to great conversations and catching up with friends old and new!

DeepSeek V4 Multimodal paper: k-a.in/Thinking_with_Visual_… Our GRIT paper: arxiv.org/pdf/2505.15879

Huge congratulations to my PhD student #4 @YFan_UCSC on successfully defending his dissertation, “Building Human-Centered Multimodal AI Agents”! 🎉 Yue’s work tackles a deeply important question: how can we design AI systems that truly support and amplify human capabilities? His research spans accessibility-driven multimodal agents—enabling natural, assistive interaction across physical and digital environments—as well as advancing agent reliability through grounded reasoning and autonomous learning. Beyond his impactful research (with papers publishing in top venues like ACL, EMNLP, NeurIPS, and ICLR), Yue led the UCSC Athena team to win Second Place in the Alexa Prize SocialBot Grand Challenge and contributed to the success of the JARVIS team in the Alexa Prize SimBot Challenge ($100K cash prize in total). I’m incredibly proud of everything Yue has accomplished and excited to see what he does next. The future of human-centered agents is in great hands. 🚀

🚨 New paper alert 🚨  📌 How can we make GUI grounding models reliable in real-world interactions?  We introduce 🚀 SafeGround: Know When to Trust GUI Grounding Models via Uncertainty Calibration In GUI agents, a single wrong click isn’t just an error — it can trigger costly or irreversible actions (e.g., unintended payments 💸 or deleting important files 🗑️).  The real danger is silent failure: most GUI grounding models always output a coordinate, even when they’re unsure.  Instead of trusting a single predicted point, SafeGround:  • estimates spatial uncertainty from prediction variability  • calibrates a decision threshold with statistical guarantees  • enables risk-controlled GUI actions, even with black-box models  💻 Code: github.com/Cece1031/SAFEGROU…  📄 Paper: arxiv.org/pdf/2602.02419 🧵1/6 #Agents #GUI

Before o3 impressed everyone with 🔥visual reasoning🔥, we already had faith in and were exploring models that can think with images. 🚀 Here’s our shot, GRIT: Grounded Reasoning with Images & Texts that trains MLLMs to think while performing visual grounding. It is done via RL on just 20 examples, no SFT and no reasoning labels needed. 🧠 As a result, the GRIT method enhances and unifies MLLM’s grounding and reasoning ability. 🌐 Project page: grounded-reasoning.github.io… 📝 Paper: arxiv.org/abs/2505.15879 💻 Code: github.com/eric-ai-lab/GRIT.… 🧵1/6

🚨 Building Reliable Open-Ended QA? Meet SAFER — Risk-Controlled Sample-then-Filter for LLMs! 🚨 SAFER introduces a two-stage framework that controls uncertainty and guarantees statistical reliability in open-ended question answering by dynamically calibrating sampling and filtering. It helps models know when to trust their answers — and when to abstain.  💡 Why SAFER matters: In real-world deployments, LLMs must respond responsibly — knowing when to act confidently and when to abstain or defer. SAFER ensures that, with provable miscoverage control and minimal sample waste, LLMs can provide trustworthy answers under risk constraints. #AI #LLM  📄 Paper: arxiv.org/2510.10193 💻 Code: github.com/Cece1031/SAFER 🧵1/5

🎉 Congrats to @YFan_UCSC for being featured @AdobeResearch! Yue is the dream PhD student every advisor hopes for. 🚀 Over the past 4 years, he led a team to win 🥈 in the Alexa Prize SocialBot Grand Challenge and published multiple high-impact first-author papers, including: - AVDN (the first aerial vision-and-dialog navigation paper) - GRIT (teaching MLLMs to think with images) - GUI-Bee (autonomous exploration for GUI grounding) - R2H, Tree-of-Lens Grounding, Multipanel VQA, etc. It’s been inspiring to watch Yue grow into an independent researcher and rising star in multimodal & agentic AI. As he enters his final PhD year (and the job market!), I can’t wait to see what he’ll cook up next. Seize the top talent while you can! 🌟