Excited to present our #ECCV2024 paper on “Understanding Physical Dynamics with Counterfactual World Modeling” with @honglin_c, Kevin Feigelis, @recursus and @dyamins. Come by poster 260 at the exhibition area from 430-630pm today @eccvconf TLDR: We introduce Counterfactual World Modeling (CWM) — a visual world model that can be prompted to extract zero-shot vision structures such as keypoints, optical flow and segmentation. We demonstrate that these structures are useful for physical dynamics understanding, achieving state-of-the-art performance on the Physion intuitive physics benchmark. In our paper, we also discuss how CWM builds up scene understanding capabilities analogous to Pearl’s ladder of causation. We hope this will help provide a path to move towards a foundation model of vision with a causal understanding of the world. @yudapearl We’re eager to see others build on top of our work. Code, models and data can be found in the links below Project page: neuroailab.github.io/cwm-phy… Paper: arxiv.org/pdf/2312.06721 Github code: github.com/neuroailab/cwm_dy… Hugging face demo: huggingface.co/spaces/rmvenk… 1/🧵