Paresh Masade

Stunning new AI model from Microsoft research. Will share my quick thoughts below as I read the paper. 1. Love the fact that they used AI generated faces to create AI generated videos. 2. Talking face models have so far concentrated on lip synch. After all, thats the most important body part when you are speaking. But it gives robotic output. 3. They are mapping facial dynamics like " including lip motion, (non-lip) expression, eye gaze and blinking, among others " and mapping to single latent variable. This is the hard part. Once you map to a single variable, you can model its distribution. 4. So how is the latent space of holistic face movement built: VoxCeleb2 dataset with talking heads. Build a transformer encoder-decoder model with inputs: facial image->canonical 3D appearance volume Vapp an identity code zid a 3D head pose zpose a facial dynamics code zdyn. Vapp,zid,zpose and zdyn are extracted by different encoder. So we take the image and get these. Then a single decoder D takes these latent variables as input and reconstructs the face image. They have come up with different loss functions to guide the network to learn the correct pose based on the speech. Once the latent space is learnt, we can extract the facial dynamics and head movements from real-life talking face videos and train a diffusion based generative model.

The purpose of an IIT education and in fact any education is not to find a job. Job is a side effect because society has need for your knowledge. Make yourself useful to society. In fact students may be sad now, but they will realise later that this might have been good for them

My speaker notes for the keynote I gave at #AIdays2024 #LetsChangeTheNarrative.