
Multimodal AI, AI Agents, Computer Vision. VP, Research @Samsung @Samsung_RA, Adjunct Professor @Stanford @StanfordSVL.
Stanford University
Joined April 2012
- Tweets962
- Following429
- Followers5.6K
- Likes3.9K
Pinned Tweet
In case you missed it, here are my slides for my talk today at CONTEXTUS @eccvconf
niebles.net/blog/2026/eccv-tโฆ
Thanks to the organizer for the invitation and the audience for the engaging questions!
Juan Carlos Niebles retweeted
Super excited to introduce TRACER!
Iโve always been interested in long-form video understanding (hourvideo.stanford.edu ๐). At @StanfordSVL, we pushed it to multi-day patient care in the ICU: video + EHR โ track activity + behavior โ assist clinicians with decision-making.
It was great to co-lead this work with @DuranteZane and Dr. Paul Tang!
arxiv.org/abs/2610.21266
1/ Can AI help doctors make more informed decisions based on patient activity and behavior over the course of a hospital stay?
Today we introduce TRACER: a training-free multimodal harness that enables LLMs and VLMs to track patient activity using bedside video and health records.
Blogpost: ai.stanford.edu/blog/tracer/
Paper: arxiv.org/abs/2610.21266
Juan Carlos Niebles retweeted
Super excited about this line of work!
LILIE is an invertible, lossless image encoder trained with language supervision. Swapping pixel loss for LILIE-space loss improves T2I quality; combining the losses improves it further. GPIC gFD drops by over 13 points ๐
Great work led by @agarwal_suchir!!
suchiragarwal.com/lilie/
Juan Carlos Niebles retweeted
1/8 Introducing LILIE: a Language-Informed and Lossless Image Encoder.
We train a language-informed encoder to supervise end-to-end visual generative models. Against a pixel-space JiT model, LILIE improves gFD on GPIC by 14 points (66.84 -> 53.01).
Juan Carlos Niebles retweeted
Excited to share Halcyon, our latest research on multimodal pretraining!
Building strong text-to-image generative models is simpler than we think!
Blog: radicalnumerics.ai/blog/halcโฆ
Models: huggingface.co/collections/rโฆ
Paper: radicalnumerics.ai/assets/haโฆ
1/8 Does text-to-image generation need language pretraining?
Today, we share Halcyon, a family of early-fusion, base models trained from scratch. Halcyon outperforms strong late-fusion baselines with up to 275ร less pretraining compute.ย Models and recipes are open.
Halcyon is part of a research program at Radical Numerics on the science of multimodal pretraining and scaling.
Juan Carlos Niebles retweeted
The goal of building any technology, AI included, should be bettering human lives and society.
โAny threat to human society, including existential, is within ourselves,โ says World Labs Technologies CEO Fei-Fei Li as she discusses the risks surrounding AI, and the responsibility humans have in shaping how the technology is developed and used.
Listen to our full interview here: bloom.bg/3V74HgP
Congrats @jcjohnss @AlexAlahi @drfeifei ! ๐๐
#ECCV2026 @StanfordAILab @StanfordHAI @StanfordSVL
Juan Carlos Niebles retweeted
Iโm speaking next at the #ECCV2026 Workshop on โUniversal Representations for Perception, Reasoning, and World Modelingโ ๐ค
I'm hiring! Looking for a Director of Vision Intelligence to join my team at @Samsung_RA ๐
If youโre passionate about multimodal AI, computer vision, and bringing advanced research directly onto everyday devices, letโs connect!
Details & apply:
sra.samsung.com/careers/detaโฆ
Juan Carlos Niebles retweeted
Replying to @CVPR
@CVPR Attendees in Denver ๐บ๐ธ
Do you have a plan for today?
Are you onto AI for content creation, GenAI for video & multimedia?
@cveu_workshop got you cover while you finish preparing your trip's agenda.
June 3 ยท 8:30 AM โ 12:30 PM ยท Room 501
More => cveu.github.io
Processing long videos with VLMs shouldn't scale quadratically. Enter StateKV! ๐ฌ๐ก
By framing streaming prefill as a fixed-capacity temporal state, we unlock linear-time prefill while keeping full per-frame detail.
Paper by @CristbalEyzagu2 and team๐
arxiv.org/abs/2605.31598
1/ The biggest problem in video understanding today isn't the models. It's that we can barely run them.
Introducing StateKV: an inference-time method that makes pretrained video VLMs scale linearly with video length.๐งต
๐ ceyzaguirre4.github.io/Stateโฆ
Juan Carlos Niebles retweeted
It's #CVPR week! ๐ If you're looking to dive deep into the future of intelligent reasoning systems, we've got your ๐ง๐ต๐๐ฟ๐๐ฑ๐ฎ๐ ๐บ๐ผ๐ฟ๐ป๐ถ๐ป๐ด plans sorted in #Denver.
Join us for the 6th ๐ ๐๐น๐๐ถ๐บ๐ผ๐ฑ๐ฎ๐น ๐๐น๐ด๐ผ๐ฟ๐ถ๐๐ต๐บ๐ถ๐ฐ ๐ฅ๐ฒ๐ฎ๐๐ผ๐ป๐ถ๐ป๐ด Workshop! We've put together an incredible lineup of keynotes, talks, and deep-dive discussions exploring the cutting edge of AI reasoning. You won't want to miss this.
๐
When: Thursday, June 4, 2026 | 8:55 AM - 12:30 PM MDT
๐ Where: ๐ฅ๐ผ๐ผ๐บ ๐ฒ๐ฌ๐ญ, Colorado Convention Center, Denver, CO
We are incredibly honored to welcome this stellar lineup of keynote speakers:
โฐ ๐ฌ๐ต:๐ฌ๐ฌ ๐๐ | ๐ค ๐๐๐ฎ๐ป ๐๐ฎ๐ฟ๐น๐ผ๐ ๐ก๐ถ๐ฒ๐ฏ๐น๐ฒ๐ @jcniebles ๐ก โ ๐ฆ๐ฎ๐น๐ฒ๐๐ณ๐ผ๐ฟ๐ฐ๐ฒ ๐๐ ๐ฅ๐ฒ๐๐ฒ๐ฎ๐ฟ๐ฐ๐ต @SFResearch
โฐ ๐ฌ๐ต:๐ฏ๐ฌ ๐๐ | ๐ค ๐๐ถ๐ฎ๐๐๐ฎ๐ป ๐ ๐ฎ๐ผ @maojiayuan ๐ก โ ๐จ๐ป๐ถ๐๐ฒ๐ฟ๐๐ถ๐๐ ๐ผ๐ณ ๐ฃ๐ฒ๐ป๐ป๐๐๐น๐๐ฎ๐ป๐ถ๐ฎ @Penn
โฐ ๐ญ๐ฌ:๐ฐ๐ฑ ๐๐ | ๐ค ๐ ๐ฒ๐น๐ฎ๐ป๐ถ๐ฒ ๐ ๐ถ๐๐ฐ๐ต๐ฒ๐น๐น @MelMitchell1 ๐ก โ ๐ฆ๐ฎ๐ป๐๐ฎ ๐๐ฒ ๐๐ป๐๐๐ถ๐๐๐๐ฒ @sfiscience
โฐ ๐ญ๐ญ:๐ญ๐ฑ ๐๐ | ๐ค ๐๐ถ๐ฎ๐น๐ผ๐ป๐ด ๐ช๐ ๐ก โ ๐ง๐๐ถ๐ป๐ด๐ต๐๐ฎ ๐จ๐ป๐ถ๐๐ฒ๐ฟ๐๐ถ๐๐ @Tsinghua_Uni
A huge thank you to our generous sponsors for making this event a reality:
๐ค ๐ ๐ถ๐๐๐๐ฏ๐ถ๐๐ต๐ถ ๐๐น๐ฒ๐ฐ๐๐ฟ๐ถ๐ฐ ๐ฅ๐ฒ๐๐ฒ๐ฎ๐ฟ๐ฐ๐ต ๐๐ฎ๐ฏ๐ผ๐ฟ๐ฎ๐๐ผ๐ฟ๐ถ๐ฒ๐ (๐ ๐๐ฅ๐) @merl_news :
๐ merl.com/
๐ค ๐๐น๐ผ๐ฟ๐ถ๐ฎ๐ป ๐๐ @ElorianAI :
๐ elorian.ai/
Massive shoutout to our dedicated organizers:
๐น ๐๐ป๐ผ๐ผ๐ฝ ๐๐ต๐ฒ๐ฟ๐ถ๐ฎ๐ป @anoopcherian โ ๐ ๐๐ฅ๐
๐น ๐ฆ๐๐ต๐ฎ๐ ๐๐ผ๐ต๐ถ๐ @Suhas_Lohit โ ๐ ๐๐ฅ๐
๐น ๐๐๐ฎ๐ป-๐๐ต๐๐ฎ๐ป ๐ฃ๐ฒ๐ป๐ด @KuanChuanPeng โ ๐ ๐๐ฅ๐
๐น ๐๐ผ๐ป๐ด๐น๐ ๐ญ๐ต๐ผ๐ @zhou_honglu โ ๐ฆ๐ฎ๐น๐ฒ๐๐ณ๐ผ๐ฟ๐ฐ๐ฒ ๐๐ ๐ฅ๐ฒ๐๐ฒ๐ฎ๐ฟ๐ฐ๐ต
๐น ๐๐ฒ๐๐ถ๐ป ๐ฆ๐บ๐ถ๐๐ต @realkevinsmith โ ๐ ๐๐ง @MIT
๐น ๐๐ผ๐๐ต๐๐ฎ ๐. ๐ง๐ฒ๐ป๐ฒ๐ป๐ฏ๐ฎ๐๐บ โ ๐ ๐๐ง
As AI systems become increasingly capable across vision, language, action, and interaction, reasoning remains one of the field's most exciting open challenges. This workshop brings together researchers and practitioners across academia and industry to discuss the latest advances in multimodal reasoning, foundation models, physical & agentic reasoning, and humanโAI comparisons.
๐ ๐๐ฒ๐ ๐๐ต๐ฒ ๐ณ๐๐น๐น ๐๐ฐ๐ต๐ฒ๐ฑ๐๐น๐ฒ ๐ฎ๐ป๐ฑ ๐๐ผ๐ฟ๐ธ๐๐ต๐ผ๐ฝ ๐ฑ๐ฒ๐๐ฎ๐ถ๐น๐ ๐ต๐ฒ๐ฟ๐ฒ:
๐ marworkshop.github.io/cvpr26โฆ
Will you be joining us on Thursday? Drop a comment or tag a colleague you're attending with! ๐
#CVPR2026
#CVPRWorkshop
#AI
#MultimodalAI
#Reasoning
#FoundationModels
#MachineLearning
#MultimodalReasoning
#IntelligentSystems
Massive drop for the AI community! ๐ 28 trillion pixels, 100M+ images, and entirely permissively licensed for commercial and research use.
A major step forward for open, stable, and scalable generative modeling. Congrats @keshigeyan and team on the launch! ๐
1/ Introducing GPIC: a Giant Permissive Image Corpus and benchmark for visual generation!
๐100M VLM-captioned image-text pairs for training
๐1M image-text pairs for benchmarking
๐ผ๏ธ~28 trillion pixels
๐คCentrally Hosted
โ
Fully permissive for research + commercial use
Dataset, benchmark and models๐งต๐
Co-led with @KyleSargentAI
Juan Carlos Niebles retweeted
#KostasThoughts: While consensus among reviewers is appreciated, it is not strictly necessary. As an Area Chair, what matters most is understanding each reviewerโs reasoning and decision-making process. That is why it is imperative that reviewers provide a clear final summary explaining why they recommend accepting or rejecting the paper, which concerns were addressed through the rebuttal and discussion, and which concerns, if any, remain unresolved. Simply putting โmy concerns were not resolvedโ is not being a responsible reviewer.
If you initially misunderstood parts of the paper or made an error in your assessment, thatโs okay we all make mistakes. Please put personal ego aside and focus on making the correct decision based on the technical merits of the paper, the rebuttal, and the discussion.
The #ECCV2026 reviewer discussion period has started! Reviewers should carefully read the authorsโ rebuttal, consider the other reviews, and actively participate in the discussion BEFORE finalizing their reviews.
For those who couldn't make the talk, Iโve just posted a summary and slides on my blog. ๐ฆพ
Check it out here: niebles.net/blog/2026/agentiโฆ
Work at @SFResearch by @zhou_honglu, @artemispng, @ZiyangW00, Kanchana Ranasinghe and others.
#ICLR2026
Heading to the DATA-FM workshop at #ICLR2026? ๐ฆพ
Catch my invited talk at 1:30 PM today in Room 203A+B: "Agentic Ambient Intelligence: Perception, Reasoning & Action."
Iโll be diving into 4 new research pillars for moving AI into the physical world.
Come say hi! ๐
Heading to the DATA-FM workshop at #ICLR2026? ๐ฆพ
Catch my invited talk at 1:30 PM today in Room 203A+B: "Agentic Ambient Intelligence: Perception, Reasoning & Action."
Iโll be diving into 4 new research pillars for moving AI into the physical world.
Come say hi! ๐
Thank you @dimadamen for coming and for the shout out!
Workshops are always more interesting than the main conference!
โฆ@iclr_confโฉ workshop on whatโs next in next token predication w talk from โฆ@jcnieblesโฉ
Juan Carlos Niebles retweeted
Workshops are always more interesting than the main conference!
โฆ@iclr_confโฉ workshop on whatโs next in next token predication w talk from โฆ@jcnieblesโฉ
Juan Carlos Niebles retweeted
Predicting future optical flow turns out to be surprisingly powerful; it boosts both robot control ๐ค and video generation ๐ฅ.
๐ New Research Alert! ๐ฐ Paper: arxiv.org/abs/2601.10781
I'm thrilled to share the release of ๐๐ผ๐๐ฃ๐ฟ๐ฒ๐ฑ (paper/code/checkpoint/interactive demo) - our model forecasts optical flow conditioned on language instructions. ๐ ๐ง๐ฟ๐ ๐ถ๐ ๐๐ผ๐๐ฟ๐๐ฒ๐น๐ณ ๐๐ถ๐ฎ ๐๐ต๐ฒ ๐ฑ๐ฒ๐บ๐ผ link on our project website!
โข Website: fofpred.github.io/
โข HuggingFace checkpoint: huggingface.co/Salesforce/FOโฆ
โข Demo: fofpred.salesforceresearch.aโฆ
โข GitHub: github.com/SalesforceAIReseaโฆ
In this work, we argue that what has to be provided to the models for many (robot) control and (video) generation tasks is how things should move in the future. This inherently is a challenging task, as it requires:
1โฃ - a clear formulation of what type of โfuture motionโ we will be computing
2โฃ - curating large enough training data with strong supervisory signals
3โฃ - an elegant model capable of fully digesting such (potentially cross-domain) training data while generalizing across domains.
We focus on ๐๐๐๐๐ฟ๐ฒ ๐ข๐ฝ๐๐ถ๐ฐ๐ฎ๐น ๐๐น๐ผ๐ ๐ฃ๐ฟ๐ฒ๐ฑ๐ถ๐ฐ๐๐ถ๐ผ๐ป - a language-conditioned, spatially dense motion prediction task. We discuss a ๐๐๐ถ๐๐ฎ๐ฏ๐น๐ฒ ๐บ๐ผ๐ฑ๐ฒ๐น ๐ฎ๐ฟ๐ฐ๐ต๐ถ๐๐ฒ๐ฐ๐๐๐ฟ๐ฒ (a unified VLM-diffusion model) and show ๐ต๐ผ๐ ๐๐ฒ ๐ฐ๐ฎ๐ป ๐ต๐ถ๐ด๐ต๐น๐-๐๐ฐ๐ฎ๐น๐ฎ๐ฏ๐น๐ฒ ๐๐ฟ๐ฎ๐ถ๐ป our model with videos from various sources (web-scale human activity videos with noisy camera motion + robot action videos in controlled environments).
FoFPred adapts to both ๐น๐ฎ๐ป๐ด๐๐ฎ๐ด๐ฒ-๐ฑ๐ฟ๐ถ๐๐ฒ๐ป ๐ฟ๐ผ๐ฏ๐ผ๐๐ถ๐ฐ ๐บ๐ฎ๐ป๐ถ๐ฝ๐๐น๐ฎ๐๐ถ๐ผ๐ป and ๐น๐ฎ๐ป๐ด๐๐ฎ๐ด๐ฒ-๐ด๐๐ถ๐ฑ๐ฒ๐ฑ ๐บ๐ผ๐๐ถ๐ผ๐ป ๐๐ถ๐ฑ๐ฒ๐ผ ๐ด๐ฒ๐ป๐ฒ๐ฟ๐ฎ๐๐ถ๐ผ๐ป - to the best of our knowledge, the first to support such cross-domain versatility. We hope this work paves the way toward generalizable, ๐บ๐ผ๐๐ถ๐ผ๐ป-๐ฎ๐๐ฎ๐ฟ๐ฒ ๐๐ผ๐ฟ๐น๐ฑ ๐บ๐ผ๐ฑ๐ฒ๐น๐ that understand and act through dynamic visual grounding.
Be sure to check out the project website and read our paper, which covers lots of useful information, including qualitative results, ablations (on human web videos, architecture choices, camera motion separation / motion target disentanglement), and our insights on open challenges and future directions.
๐ Huge shoutout to Kanchana Ranasinghe @kahnchana, our intern at Salesforce AI Research @SFResearch, for the outstanding work on this project and for making an excellent open source! ๐ซถ๐๐
#WorldModels
#DiffusionModels
#VideoGeneration
#Robotics
#EmbodiedAI
#GenerativeAI
Introducing FOFPred โ a language-driven future optical flow prediction framework that enables improved robot control and video generation.
Instead of reacting to motion, FOFPred predicts how motion will evolve โ conditioned on natural language.
๐ Project: fofpred.github.io
๐ Paper: arxiv.org/abs/2601.10781
๐ป Code: github.com/SalesforceAIReseaโฆ
๐ค Model: huggingface.co/Salesforce/FOโฆ
๐น๏ธ Demo: fofpred.salesforceresearch.aโฆ
๐งต[1/3]































