
AI & Audio @UMG. Previously @tiktok_us @BandLab. ML & media tech. 🇲🇾 🇸🇬
Joined December 2011
- Tweets4.3K
- Following790
- Followers600
- Likes2.1K
Hao Hao Tan retweeted
What is an intuitive way to search in a music library? Tags? Captions? Chatbots?
We propose another paradigm: what if we could create a slider for *any* musical concept and use it to refine in real-time what we're looking for? 🎛
Work led by @Juj_Guinot, accepted at @ISMIRConf!
Happy to share our #ISMIR2026 paper on steerable music retrieval with sparse autoencoders 🎛️🎧
Dense retrieval is great at finding similar tracks,
But once you have a query, what if you want to modify one concept inside it?
More piano,
More intimate,
Less gritty.
1/n
Hao Hao Tan retweeted
Is autoresearch really better than classic hyperparameter tuning?
We did experiments comparing Optuna & autoresearch.
Autoresearch converges faster, is more cost-efficient, and even generalizes better: 🧵(1/6)
Hao Hao Tan retweeted
🌶️ Some (perhaps) spicy thoughts. It’s been a while since my last tweet, but I wanted to write about how disorienting it has been from academia to an LLM lab 😅
The kind of research I was trained to do during my PhD almost doesn’t exist here. The obsession with mathematical elegance and novelty is mostly gone. Everything is about scaling data and compute. For a while, that really got to me. At my lowest point, I felt like I’d lost interest in building LLMs altogether. I didn’t feel intellectually challenged anymore.
What made this even stranger was that, at a technical level, things worked. If there was a capability I wanted to teach a model, scaling the right data and compute always got me there, no exception (so far).
But recently, I found a way to reconcile with myself..
I realized the real competition isn’t in the ML recipe anymore. Most teams do roughly the same thing. What actually matters is how fast you can iterate, test ideas, and recover from mistakes. And that speed is mostly backed by infrastructure 🏗️ Faster loops, fewer bugs, better tooling.
Seeing this made me excited again! Infra is its own deep, hard, and intellectually fun problem space.
In 2026, I want to become an ML researcher who’s really good at infra. And I'll come back to ML problems with that edge, and will be excited to share what I find 😌
Hao Hao Tan retweeted
We ran a randomized controlled trial to see how much AI coding tools speed up experienced open-source developers.
The results surprised us: Developers thought they were 20% faster with AI tools, but they were actually 19% slower when they had access to AI than when they didn't.
Hao Hao Tan retweeted
torchlpc v0.7
* All kernels are rewritten in C++/CUDA
* New parallel scan kernel for CPU
* Running 1.2 to 2 times faster on GPU
* Return final delay values (aka `zf`)
See release notes for more details.
github.com/DiffAPF/torchlpc/…
Hao Hao Tan retweeted
I am pleased to announce a new version of my RL tutorial. Major update to the LLM chapter (eg DPO, GRPO, thinking), minor updates to the MARL and MBRL chapters and various sections (eg offline RL, DPG, etc). Enjoy!
arxiv.org/abs/2412.05265
Hao Hao Tan retweeted
Wait…WHAT? Introducing Pikadditions, the easiest way to make your content stand out.
Add anyone or anything to any video, whether that’s a video you shoot yourself, or a favorite clip. Special surprise: get fifteen free Pikadditions generations when you sign up!
Go try it at pika dot art
Hao Hao Tan retweeted
Following our recent experiments I wrote more notes on Matcha TTS design
alphacephei.com/nsh/2025/01/…
Hao Hao Tan retweeted
@kchonyc is correct in saying that "the current crop of PhD students [...] are being left out" of the modern AI hiring pipeline, but it's even worse than that.
Those who do get hired are often left unsatisfied that they're expected to build products instead of doing research.
Hao Hao Tan retweeted
feeling a bit under the weather this week … thus an increased level of activity on social media and blog: kyunghyuncho.me/i-sensed-anx…
Hao Hao Tan retweeted
the anxiety and frustration is so real.
during this year's neurips i was telling around "this is an AI job expo." and i meant it. yes it's still a great conference and any conference is always also a job expo, but.. it is so, more than anytime before.
Hao Hao Tan retweeted
If you’re a Member, we’ve also added some serious Studio upgrades. Build on your MIDI ideas quickly with Smart Tools, and give your ideas that extra push with Extend, Layer, and Recompose! 🦾 (3/5)
Hao Hao Tan retweeted
Today, we release several Moshi artifacts: a long technical report with all the details behind our model, weights for Moshi and its Mimi codec, along with streaming inference code in Pytorch, Rust and MLX. More details below 🧵 ⬇️
Paper: kyutai.org/Moshi.pdf
Repo: github.com/kyutai-labs/moshi
HuggingFace: huggingface.co/kmhf
Hao Hao Tan retweeted
Audio Match Cutting
Finding and Creating Matching Audio Transitions in Movies and Videos
discuss: huggingface.co/papers/2408.1…
A "match cut" is a common video editing technique where a pair of shots that have a similar composition transition fluidly from one to another. Although match cuts are often visual, certain match cuts involve the fluid transition of audio, where sounds from different sources merge into one indistinguishable transition between two shots. In this paper, we explore the ability to automatically find and create "audio match cuts" within videos and movies. We create a self-supervised audio representation for audio match cutting and develop a coarse-to-fine audio match pipeline that recommends matching shots and creates the blended audio. We further annotate a dataset for the proposed audio match cut task and compare the ability of multiple audio representations to find audio match cut candidates. Finally, we evaluate multiple methods to blend two matching audio candidates with the goal of creating a smooth transition.
Hao Hao Tan retweeted
🥳We just uploaded another ISMIR paper to arXiv:
Stem-JEPA: A Joint-Embedding Predictive Architecture for Musical Stem Compatibility Estimation
A. Riou, S. Lattner, G. Hadjeres, M. Anslow, G. Peeters
📜 Paper: arxiv.org/abs/2408.02514
🌍 Code: github.com/SonyCSLParis/Stem…
For more info, check out the tweet below 👇
Glad to announce that Stem-JEPA has been accepted to @ISMIRConf !
In this work, we tackle the task of musical stem compatibility estimation (what “fits” together) as a representation learning problem. (1/7)
Paper: arxiv.org/abs/2408.02514
Code: github.com/SonyCSLParis/Stem…
Hao Hao Tan retweeted
I gave a 1-hour talk about generative modelling at the EEML 2024 summer school last month.
It's mostly an intuitive look at how and why diffusion models actually work -- not unlike the content of my recent blog posts.
All summer school talks will be freely available online!🙏
EEML'24 Day 1 videos are out! 🇷🇸
* Intro to DL (@alfcnz): youtu.be/1bBOneUMu3Y?si=KiSk…
* Generative modelling + iterative refinement (@sedielem): youtu.be/9BHQvQlsVdE?si=x9ir…
* AI for Good (@weballergy): youtu.be/tJSicw7DPVU?si=RPcp…
* Reasoning (@backprop2seed & I): youtu.be/CyIuM5eQZ5A?si=90pl…
Hao Hao Tan retweeted
We built a real-time music jamming system using RL and generative models -- you can play along with this model and learn more about our work at #ICML2024 🎶!
📄 paper: twtr.to/UFe6X
🌐 website: twtr.to/v9E5y
🕐 Tue 23 Jul 1:30 - 3 p.m. CEST
📍 Hall C 4-9
🧵
Hao Hao Tan retweeted
🔥 How I broke the internet today and what lessons can we learn from it? #Crowdstrike 🧐
Several things that make it a good fake that worked: 👇
1. No culprit named yet, I bring it on a platter, people like to have a culprit.
2- The culprit seems completely stupid, he is proud of his stupidity, he... takes his afternoon off on... the first day of work...
3- This falls right into a huge buzz in which people absolutely need to have new information, and a fake is by nature new, you won't read it anywhere else
4- In English = very easy to share internationally, with the vast majority of people who have no idea who I am.
5- Baby fingers are stupid, but they distract people from other things (like the fact that I have a horn on my head because of bad clipping)
6- Confirmation bias: People want to believe it, it's so funny. “I like the information, so it is true.”
7- The information is pushed ironically by people who know that it is a joke, and then it arrives in the litteral zone which amplifies it even stronger.
Hao Hao Tan retweeted
Masked Generative Video-to-Audio Transformers with Enhanced Synchronicity
Video-to-audio (V2A) generation leverages visual-only video features to render plausible sounds that match the scene. Importantly, the generated sound onsets should match the visual actions that are aligned with them, otherwise unnatural synchronization artifacts arise. Recent works have explored the progression of conditioning sound generators on still images and then video features, focusing on quality and semantic matching while ignoring synchronization, or by sacrificing some amount of quality to focus on improving synchronization only. In this work, we propose a V2A generative model, named MaskVAT, that interconnects a full-band high-quality general audio codec with a sequence-to-sequence masked generative model. This combination allows modeling both high audio quality, semantic matching, and temporal synchronicity at the same time. Our results show that, by combining a high-quality codec with the proper pre-trained audio-visual features and a sequence-to-sequence parallel structure, we are able to yield highly synchronized results on one hand, whilst being competitive with the state of the art of non-codec generative audio models.
























