When Apple said they were rolling out 'spatial media' this is what I actually expected - 3D video with six-degrees of freedom. Instead Vision Pro users got old wine (VR180 video) in a new bottle (Apple's MV-HEVC format). Conveniently, here's an open source NeRF-based volumetric video engine, with WebXR streaming to Vision Pro and Quest. Oh and it supports Mac too. All this from a little known company called Lifecast (YC '21). Let's dive into their new paper "Baking Neural Radiance Field Video to Immersive Layered Depth Images" - which aims to prove that volumetric video capture, editing and playback is very possible on today's hardware. 🔴Central Approach: Multi-View Video → NeRFs → LDIs You can also use a simple VR180 camera (e.g. Canon R5C) or a big ball of cameras (GoPros) to capture 3D video of a scene, make some NeRFs, then bake it down into a Layered-Depth Image (LDI) based video format. It's kind of like taking a detailed, hand-crafted sculpture (the NeRF) and then creating a simplified, 3D-printed version (the LDI) that captures the essence of the original, but is easier to distribute and display. According to the paper, it's a 3 layer LDI - which isn't super high but still a pretty substantial jump up from flat stereo. However, the reason for the authors sticking to just 3 layers is very practical. 🔴Why 3 Layers is a Sweet Spot for Volumetric Video Turns out you can can easily pack a 3x3 grid of 1920x1920 resolution layers into a 5760x5760 image - which is the max resolution video most VR headsets (with mobile chips) can decode. Plus, the paper uses an equiangular (vs. equirectangular) projection which keeps the pixel density in the center region comparable to "8K" resolution 180° VR video. That's quite the sweet spot IMO. 🔴Things Move Fast! Yet Old Ideas Often Find New Homes Low key stoked to see 'LDIs' and 'equiangular projections' being used to bake out these 4D radiance fields. Circa 2017-2020 my team at Google used exactly these approaches to enable light field video (@broxtronix et al) and high resolution video streaming on YouTubeVR. What's wild is 4 years later this paper brings a 238x speedup vs. the Deep View Video paper - and the pipeline can be run on a single GPU workstation vs. a Google datacenter. 🔴Closing Thoughts IMO, the visual fidelity is still not there with complex subject matter - like, 3 layers will only get you so far. But what this paper/pipeline proves to me is that high res volumetric video streaming is absolutely possible on mobile hardware. A simpler 3 layer format like this could also make editing of volumetric video in existing tools like premiere/resolve a lot easier than we might have expected.