Remember Niantic? Yes, that @NianticLabs from Pokemon Go @PokemonGoApp. Well, they partnered with @FlexionAI, and their work is pretty awesome. They demonstrated an end-to-end real2sim2real pipeline for humanoid navigation. Here is the step-by-step breakdown: - an operator walks an office once with an off-the-shelf 360° camera - Niantic Spatial estimates camera poses, and trains a 3D Gaussian splat (overly simplified: they get a fast low compute 3D model) - the splat and mesh is normalized to match NVIDIA's NuRec volume spec - that file now seemlessly loads into Isaac Sim and Isaac Lab, and allows for parallel reinforcement learning! - the resulting RGB-only local navigation policy is dropped onto a physical humanoid - said humanoid walks the real office to three commanded target poses Worth mentioning: it is a demonstration, not a product launch. No humanoid platform is named, no camera model, no GPU count, and no training time. This is demo of feasability of a clever piece of engineering imho. Something interesting I learnt while looking into this: Niantic Spatial holds roughly 10 million scanned locations, built from ~30 billion images, captured over a decade of Pokémon GO and Ingress players. And: they are still taking in about a million fresh scans a week! Also the main learning here imoh: embedding only RGB camera beats stereo depth cameras by about four points in sim. This is insane! Depth cameras are close to being useless for daily humanoid applications; the future will be RGb and Lidar, since depth estimation and 3D reconstruction algorithms have become so good. Final fun fact: Nvidia mafia runs deeps -> Flexion's CEO and CTO are ex-Nvidia, NVIDIA's NVentures is an investor in Flexion, and this demo uses Nvidia Isaac.