Multimodal AI, AI Agents, Computer Vision. VP, Research @Samsung @Samsung_RA, Adjunct Professor @Stanford @StanfordSVL.

Stanford University
Joined April 2012
In case you missed it, here are my slides for my talk today at CONTEXTUS @eccvconf niebles.net/blog/2026/eccv-tโ€ฆ Thanks to the organizer for the invitation and the audience for the engaging questions!
๐Ÿ“ขIโ€™ll be speaking at the CONTEXTUS workshop @eccvconf 2026 in just a few minutes! 1:20PM ๐Ÿ“Join us at Skybox 504-505 (5th floor, yes) at Malmo Arena. Iโ€™ll be taking about our recent work on perception, evidence and efficiency for AI in the physical world. #ECCV2026
1
3
21
2,496
Juan Carlos Niebles retweeted
Super excited to introduce TRACER! Iโ€™ve always been interested in long-form video understanding (hourvideo.stanford.edu ๐Ÿ™‚). At @StanfordSVL, we pushed it to multi-day patient care in the ICU: video + EHR โ†’ track activity + behavior โ†’ assist clinicians with decision-making. It was great to co-lead this work with @DuranteZane and Dr. Paul Tang! arxiv.org/abs/2610.21266
1/ Can AI help doctors make more informed decisions based on patient activity and behavior over the course of a hospital stay? Today we introduce TRACER: a training-free multimodal harness that enables LLMs and VLMs to track patient activity using bedside video and health records. Blogpost: ai.stanford.edu/blog/tracer/ Paper: arxiv.org/abs/2610.21266
4
20
941
Juan Carlos Niebles retweeted
Super excited about this line of work! LILIE is an invertible, lossless image encoder trained with language supervision. Swapping pixel loss for LILIE-space loss improves T2I quality; combining the losses improves it further. GPIC gFD drops by over 13 points ๐Ÿ™‚ Great work led by @agarwal_suchir!! suchiragarwal.com/lilie/
1/8 Introducing LILIE: a Language-Informed and Lossless Image Encoder. We train a language-informed encoder to supervise end-to-end visual generative models. Against a pixel-space JiT model, LILIE improves gFD on GPIC by 14 points (66.84 -> 53.01).
3
12
1,183
Juan Carlos Niebles retweeted
1/8 Introducing LILIE: a Language-Informed and Lossless Image Encoder. We train a language-informed encoder to supervise end-to-end visual generative models. Against a pixel-space JiT model, LILIE improves gFD on GPIC by 14 points (66.84 -> 53.01).
5
27
3
161
17,534
Juan Carlos Niebles retweeted
Excited to share Halcyon, our latest research on multimodal pretraining! Building strong text-to-image generative models is simpler than we think! Blog: radicalnumerics.ai/blog/halcโ€ฆ Models: huggingface.co/collections/rโ€ฆ Paper: radicalnumerics.ai/assets/haโ€ฆ
1/8 Does text-to-image generation need language pretraining? Today, we share Halcyon, a family of early-fusion, base models trained from scratch. Halcyon outperforms strong late-fusion baselines with up to 275ร— less pretraining compute.ย Models and recipes are open. Halcyon is part of a research program at Radical Numerics on the science of multimodal pretraining and scaling.
5
31
3
89
18,760
Juan Carlos Niebles retweeted
The goal of building any technology, AI included, should be bettering human lives and society.
โ€œAny threat to human society, including existential, is within ourselves,โ€ says World Labs Technologies CEO Fei-Fei Li as she discusses the risks surrounding AI, and the responsibility humans have in shaping how the technology is developed and used. Listen to our full interview here: bloom.bg/3V74HgP
93
170
31
1,144
162,163
Juan Carlos Niebles retweeted
Iโ€™m speaking next at the #ECCV2026 Workshop on โ€œUniversal Representations for Perception, Reasoning, and World Modelingโ€ ๐Ÿค“
1
4
1
60
6,303
๐Ÿ“ขIโ€™ll be speaking at the CONTEXTUS workshop @eccvconf 2026 in just a few minutes! 1:20PM ๐Ÿ“Join us at Skybox 504-505 (5th floor, yes) at Malmo Arena. Iโ€™ll be taking about our recent work on perception, evidence and efficiency for AI in the physical world. #ECCV2026
1
4
1
17
4,086
I'm hiring! Looking for a Director of Vision Intelligence to join my team at @Samsung_RA ๐Ÿš€ If youโ€™re passionate about multimodal AI, computer vision, and bringing advanced research directly onto everyday devices, letโ€™s connect! Details & apply: sra.samsung.com/careers/detaโ€ฆ
4
24
2,323
Juan Carlos Niebles retweeted
Replying to @CVPR
@CVPR Attendees in Denver ๐Ÿ‡บ๐Ÿ‡ธ Do you have a plan for today? Are you onto AI for content creation, GenAI for video & multimedia? @cveu_workshop got you cover while you finish preparing your trip's agenda. June 3 ยท 8:30 AM โ€“ 12:30 PM ยท Room 501 More => cveu.github.io
3
4
1
9
4,290
Processing long videos with VLMs shouldn't scale quadratically. Enter StateKV! ๐ŸŽฌ๐Ÿ’ก By framing streaming prefill as a fixed-capacity temporal state, we unlock linear-time prefill while keeping full per-frame detail. Paper by @CristbalEyzagu2 and team๐Ÿ‘‡ arxiv.org/abs/2605.31598
1/ The biggest problem in video understanding today isn't the models. It's that we can barely run them. Introducing StateKV: an inference-time method that makes pretrained video VLMs scale linearly with video length.๐Ÿงต ๐Ÿ”— ceyzaguirre4.github.io/Stateโ€ฆ
5
13
2,795
Juan Carlos Niebles retweeted
It's #CVPR week! ๐ŸŽ‰ If you're looking to dive deep into the future of intelligent reasoning systems, we've got your ๐—ง๐—ต๐˜‚๐—ฟ๐˜€๐—ฑ๐—ฎ๐˜† ๐—บ๐—ผ๐—ฟ๐—ป๐—ถ๐—ป๐—ด plans sorted in #Denver. Join us for the 6th ๐— ๐˜‚๐—น๐˜๐—ถ๐—บ๐—ผ๐—ฑ๐—ฎ๐—น ๐—”๐—น๐—ด๐—ผ๐—ฟ๐—ถ๐˜๐—ต๐—บ๐—ถ๐—ฐ ๐—ฅ๐—ฒ๐—ฎ๐˜€๐—ผ๐—ป๐—ถ๐—ป๐—ด Workshop! We've put together an incredible lineup of keynotes, talks, and deep-dive discussions exploring the cutting edge of AI reasoning. You won't want to miss this. ๐Ÿ“… When: Thursday, June 4, 2026 | 8:55 AM - 12:30 PM MDT ๐Ÿ“ Where: ๐—ฅ๐—ผ๐—ผ๐—บ ๐Ÿฒ๐Ÿฌ๐Ÿญ, Colorado Convention Center, Denver, CO We are incredibly honored to welcome this stellar lineup of keynote speakers: โฐ ๐Ÿฌ๐Ÿต:๐Ÿฌ๐Ÿฌ ๐—”๐—  | ๐ŸŽค ๐—๐˜‚๐—ฎ๐—ป ๐—–๐—ฎ๐—ฟ๐—น๐—ผ๐˜€ ๐—ก๐—ถ๐—ฒ๐—ฏ๐—น๐—ฒ๐˜€ @jcniebles ๐Ÿ’ก โ€” ๐—ฆ๐—ฎ๐—น๐—ฒ๐˜€๐—ณ๐—ผ๐—ฟ๐—ฐ๐—ฒ ๐—”๐—œ ๐—ฅ๐—ฒ๐˜€๐—ฒ๐—ฎ๐—ฟ๐—ฐ๐—ต @SFResearch โฐ ๐Ÿฌ๐Ÿต:๐Ÿฏ๐Ÿฌ ๐—”๐—  | ๐ŸŽค ๐—๐—ถ๐—ฎ๐˜†๐˜‚๐—ฎ๐—ป ๐— ๐—ฎ๐—ผ @maojiayuan ๐Ÿ’ก โ€” ๐—จ๐—ป๐—ถ๐˜ƒ๐—ฒ๐—ฟ๐˜€๐—ถ๐˜๐˜† ๐—ผ๐—ณ ๐—ฃ๐—ฒ๐—ป๐—ป๐˜€๐˜†๐—น๐˜ƒ๐—ฎ๐—ป๐—ถ๐—ฎ @Penn โฐ ๐Ÿญ๐Ÿฌ:๐Ÿฐ๐Ÿฑ ๐—”๐—  | ๐ŸŽค ๐— ๐—ฒ๐—น๐—ฎ๐—ป๐—ถ๐—ฒ ๐— ๐—ถ๐˜๐—ฐ๐—ต๐—ฒ๐—น๐—น @MelMitchell1 ๐Ÿ’ก โ€” ๐—ฆ๐—ฎ๐—ป๐˜๐—ฎ ๐—™๐—ฒ ๐—œ๐—ป๐˜€๐˜๐—ถ๐˜๐˜‚๐˜๐—ฒ @sfiscience โฐ ๐Ÿญ๐Ÿญ:๐Ÿญ๐Ÿฑ ๐—”๐—  | ๐ŸŽค ๐—๐—ถ๐—ฎ๐—น๐—ผ๐—ป๐—ด ๐—ช๐˜‚ ๐Ÿ’ก โ€” ๐—ง๐˜€๐—ถ๐—ป๐—ด๐—ต๐˜‚๐—ฎ ๐—จ๐—ป๐—ถ๐˜ƒ๐—ฒ๐—ฟ๐˜€๐—ถ๐˜๐˜† @Tsinghua_Uni A huge thank you to our generous sponsors for making this event a reality: ๐Ÿค ๐— ๐—ถ๐˜๐˜€๐˜‚๐—ฏ๐—ถ๐˜€๐—ต๐—ถ ๐—˜๐—น๐—ฒ๐—ฐ๐˜๐—ฟ๐—ถ๐—ฐ ๐—ฅ๐—ฒ๐˜€๐—ฒ๐—ฎ๐—ฟ๐—ฐ๐—ต ๐—Ÿ๐—ฎ๐—ฏ๐—ผ๐—ฟ๐—ฎ๐˜๐—ผ๐—ฟ๐—ถ๐—ฒ๐˜€ (๐— ๐—˜๐—ฅ๐—Ÿ) @merl_news : ๐ŸŒ merl.com/ ๐Ÿค ๐—˜๐—น๐—ผ๐—ฟ๐—ถ๐—ฎ๐—ป ๐—”๐—œ @ElorianAI : ๐ŸŒ elorian.ai/ Massive shoutout to our dedicated organizers: ๐Ÿ”น ๐—”๐—ป๐—ผ๐—ผ๐—ฝ ๐—–๐—ต๐—ฒ๐—ฟ๐—ถ๐—ฎ๐—ป @anoopcherian โ€” ๐— ๐—˜๐—ฅ๐—Ÿ ๐Ÿ”น ๐—ฆ๐˜‚๐—ต๐—ฎ๐˜€ ๐—Ÿ๐—ผ๐—ต๐—ถ๐˜ @Suhas_Lohit โ€” ๐— ๐—˜๐—ฅ๐—Ÿ ๐Ÿ”น ๐—ž๐˜‚๐—ฎ๐—ป-๐—–๐—ต๐˜‚๐—ฎ๐—ป ๐—ฃ๐—ฒ๐—ป๐—ด @KuanChuanPeng โ€” ๐— ๐—˜๐—ฅ๐—Ÿ ๐Ÿ”น ๐—›๐—ผ๐—ป๐—ด๐—น๐˜‚ ๐—ญ๐—ต๐—ผ๐˜‚ @zhou_honglu โ€” ๐—ฆ๐—ฎ๐—น๐—ฒ๐˜€๐—ณ๐—ผ๐—ฟ๐—ฐ๐—ฒ ๐—”๐—œ ๐—ฅ๐—ฒ๐˜€๐—ฒ๐—ฎ๐—ฟ๐—ฐ๐—ต ๐Ÿ”น ๐—ž๐—ฒ๐˜ƒ๐—ถ๐—ป ๐—ฆ๐—บ๐—ถ๐˜๐—ต @realkevinsmith โ€” ๐— ๐—œ๐—ง @MIT ๐Ÿ”น ๐—๐—ผ๐˜€๐—ต๐˜‚๐—ฎ ๐—•. ๐—ง๐—ฒ๐—ป๐—ฒ๐—ป๐—ฏ๐—ฎ๐˜‚๐—บ โ€” ๐— ๐—œ๐—ง As AI systems become increasingly capable across vision, language, action, and interaction, reasoning remains one of the field's most exciting open challenges. This workshop brings together researchers and practitioners across academia and industry to discuss the latest advances in multimodal reasoning, foundation models, physical & agentic reasoning, and humanโ€“AI comparisons. ๐Ÿ‘‰ ๐—š๐—ฒ๐˜ ๐˜๐—ต๐—ฒ ๐—ณ๐˜‚๐—น๐—น ๐˜€๐—ฐ๐—ต๐—ฒ๐—ฑ๐˜‚๐—น๐—ฒ ๐—ฎ๐—ป๐—ฑ ๐˜„๐—ผ๐—ฟ๐—ธ๐˜€๐—ต๐—ผ๐—ฝ ๐—ฑ๐—ฒ๐˜๐—ฎ๐—ถ๐—น๐˜€ ๐—ต๐—ฒ๐—ฟ๐—ฒ: ๐ŸŒ marworkshop.github.io/cvpr26โ€ฆ Will you be joining us on Thursday? Drop a comment or tag a colleague you're attending with! ๐Ÿ‘‡ #CVPR2026 #CVPRWorkshop #AI #MultimodalAI #Reasoning #FoundationModels #MachineLearning #MultimodalReasoning #IntelligentSystems
6
15
1,425
Massive drop for the AI community! ๐Ÿš€ 28 trillion pixels, 100M+ images, and entirely permissively licensed for commercial and research use. A major step forward for open, stable, and scalable generative modeling. Congrats @keshigeyan and team on the launch! ๐Ÿ‘‡
1/ Introducing GPIC: a Giant Permissive Image Corpus and benchmark for visual generation! ๐Ÿš€100M VLM-captioned image-text pairs for training ๐Ÿ“Š1M image-text pairs for benchmarking ๐Ÿ–ผ๏ธ~28 trillion pixels ๐Ÿค—Centrally Hosted โœ…Fully permissive for research + commercial use Dataset, benchmark and models๐Ÿงต๐Ÿ‘‡ Co-led with @KyleSargentAI
1
3
5
964
Juan Carlos Niebles retweeted
#KostasThoughts: While consensus among reviewers is appreciated, it is not strictly necessary. As an Area Chair, what matters most is understanding each reviewerโ€™s reasoning and decision-making process. That is why it is imperative that reviewers provide a clear final summary explaining why they recommend accepting or rejecting the paper, which concerns were addressed through the rebuttal and discussion, and which concerns, if any, remain unresolved. Simply putting โ€œmy concerns were not resolvedโ€ is not being a responsible reviewer. If you initially misunderstood parts of the paper or made an error in your assessment, thatโ€™s okay we all make mistakes. Please put personal ego aside and focus on making the correct decision based on the technical merits of the paper, the rebuttal, and the discussion.
The #ECCV2026 reviewer discussion period has started! Reviewers should carefully read the authorsโ€™ rebuttal, consider the other reviews, and actively participate in the discussion BEFORE finalizing their reviews.
1
4
27
3,723
For those who couldn't make the talk, Iโ€™ve just posted a summary and slides on my blog. ๐Ÿฆพ Check it out here: niebles.net/blog/2026/agentiโ€ฆ Work at @SFResearch by @zhou_honglu, @artemispng, @ZiyangW00, Kanchana Ranasinghe and others. #ICLR2026
Heading to the DATA-FM workshop at #ICLR2026? ๐Ÿฆพ Catch my invited talk at 1:30 PM today in Room 203A+B: "Agentic Ambient Intelligence: Perception, Reasoning & Action." Iโ€™ll be diving into 4 new research pillars for moving AI into the physical world. Come say hi! ๐Ÿ“
4
1
11
1,823
Heading to the DATA-FM workshop at #ICLR2026? ๐Ÿฆพ Catch my invited talk at 1:30 PM today in Room 203A+B: "Agentic Ambient Intelligence: Perception, Reasoning & Action." Iโ€™ll be diving into 4 new research pillars for moving AI into the physical world. Come say hi! ๐Ÿ“
2
1
6
3,010
Thank you @dimadamen for coming and for the shout out!
Workshops are always more interesting than the main conference! โฆ@iclr_confโฉ workshop on whatโ€™s next in next token predication w talk from โฆ@jcnieblesโฉ
4
389
Juan Carlos Niebles retweeted
Workshops are always more interesting than the main conference! โฆ@iclr_confโฉ workshop on whatโ€™s next in next token predication w talk from โฆ@jcnieblesโฉ
3
1
27
2,559
Juan Carlos Niebles retweeted
Predicting future optical flow turns out to be surprisingly powerful; it boosts both robot control ๐Ÿค– and video generation ๐ŸŽฅ. ๐Ÿš€ New Research Alert! ๐Ÿ“ฐ Paper: arxiv.org/abs/2601.10781 I'm thrilled to share the release of ๐—™๐—ผ๐—™๐—ฃ๐—ฟ๐—ฒ๐—ฑ (paper/code/checkpoint/interactive demo) - our model forecasts optical flow conditioned on language instructions. ๐Ÿ‘‰ ๐—ง๐—ฟ๐˜† ๐—ถ๐˜ ๐˜†๐—ผ๐˜‚๐—ฟ๐˜€๐—ฒ๐—น๐—ณ ๐˜ƒ๐—ถ๐—ฎ ๐˜๐—ต๐—ฒ ๐—ฑ๐—ฒ๐—บ๐—ผ link on our project website! โ€ข Website: fofpred.github.io/ โ€ข HuggingFace checkpoint: huggingface.co/Salesforce/FOโ€ฆ โ€ข Demo: fofpred.salesforceresearch.aโ€ฆ โ€ข GitHub: github.com/SalesforceAIReseaโ€ฆ In this work, we argue that what has to be provided to the models for many (robot) control and (video) generation tasks is how things should move in the future. This inherently is a challenging task, as it requires: 1โƒฃ - a clear formulation of what type of โ€˜future motionโ€™ we will be computing 2โƒฃ - curating large enough training data with strong supervisory signals 3โƒฃ - an elegant model capable of fully digesting such (potentially cross-domain) training data while generalizing across domains. We focus on ๐—™๐˜‚๐˜๐˜‚๐—ฟ๐—ฒ ๐—ข๐—ฝ๐˜๐—ถ๐—ฐ๐—ฎ๐—น ๐—™๐—น๐—ผ๐˜„ ๐—ฃ๐—ฟ๐—ฒ๐—ฑ๐—ถ๐—ฐ๐˜๐—ถ๐—ผ๐—ป - a language-conditioned, spatially dense motion prediction task. We discuss a ๐˜€๐˜‚๐—ถ๐˜๐—ฎ๐—ฏ๐—น๐—ฒ ๐—บ๐—ผ๐—ฑ๐—ฒ๐—น ๐—ฎ๐—ฟ๐—ฐ๐—ต๐—ถ๐˜๐—ฒ๐—ฐ๐˜๐˜‚๐—ฟ๐—ฒ (a unified VLM-diffusion model) and show ๐—ต๐—ผ๐˜„ ๐˜„๐—ฒ ๐—ฐ๐—ฎ๐—ป ๐—ต๐—ถ๐—ด๐—ต๐—น๐˜†-๐˜€๐—ฐ๐—ฎ๐—น๐—ฎ๐—ฏ๐—น๐—ฒ ๐˜๐—ฟ๐—ฎ๐—ถ๐—ป our model with videos from various sources (web-scale human activity videos with noisy camera motion + robot action videos in controlled environments). FoFPred adapts to both ๐—น๐—ฎ๐—ป๐—ด๐˜‚๐—ฎ๐—ด๐—ฒ-๐—ฑ๐—ฟ๐—ถ๐˜ƒ๐—ฒ๐—ป ๐—ฟ๐—ผ๐—ฏ๐—ผ๐˜๐—ถ๐—ฐ ๐—บ๐—ฎ๐—ป๐—ถ๐—ฝ๐˜‚๐—น๐—ฎ๐˜๐—ถ๐—ผ๐—ป and ๐—น๐—ฎ๐—ป๐—ด๐˜‚๐—ฎ๐—ด๐—ฒ-๐—ด๐˜‚๐—ถ๐—ฑ๐—ฒ๐—ฑ ๐—บ๐—ผ๐˜๐—ถ๐—ผ๐—ป ๐˜ƒ๐—ถ๐—ฑ๐—ฒ๐—ผ ๐—ด๐—ฒ๐—ป๐—ฒ๐—ฟ๐—ฎ๐˜๐—ถ๐—ผ๐—ป - to the best of our knowledge, the first to support such cross-domain versatility. We hope this work paves the way toward generalizable, ๐—บ๐—ผ๐˜๐—ถ๐—ผ๐—ป-๐—ฎ๐˜„๐—ฎ๐—ฟ๐—ฒ ๐˜„๐—ผ๐—ฟ๐—น๐—ฑ ๐—บ๐—ผ๐—ฑ๐—ฒ๐—น๐˜€ that understand and act through dynamic visual grounding. Be sure to check out the project website and read our paper, which covers lots of useful information, including qualitative results, ablations (on human web videos, architecture choices, camera motion separation / motion target disentanglement), and our insights on open challenges and future directions. ๐Ÿ‘ Huge shoutout to Kanchana Ranasinghe @kahnchana, our intern at Salesforce AI Research @SFResearch, for the outstanding work on this project and for making an excellent open source! ๐Ÿซถ๐Ÿ‘๐Ÿ˜ #WorldModels #DiffusionModels #VideoGeneration #Robotics #EmbodiedAI #GenerativeAI
Introducing FOFPred โ€” a language-driven future optical flow prediction framework that enables improved robot control and video generation. Instead of reacting to motion, FOFPred predicts how motion will evolve โ€” conditioned on natural language. ๐ŸŒ Project: fofpred.github.io ๐Ÿ“„ Paper: arxiv.org/abs/2601.10781 ๐Ÿ’ป Code: github.com/SalesforceAIReseaโ€ฆ ๐Ÿค— Model: huggingface.co/Salesforce/FOโ€ฆ ๐Ÿ•น๏ธ Demo: fofpred.salesforceresearch.aโ€ฆ ๐Ÿงต[1/3]
3
6
837