Research engineer at @sesame. Previously leading speech research at @StabilityAI and @RockstarGames.
Joined December 2022
- Tweets128
- Following311
- Followers674
- Likes161
Computers should meet humans where they’re at, not the other way around. It’s what we’re working hard towards at @sesame with both our voice agents and our eyewear.
We’re building all-day optical glasses that will be a bona fide computing device. Invisible, ambient computing, available at your fingertips, that fades away when you’re not using it. Our assistant is designed to be the logical operating system of that device — one that’s voice oriented and as intuitive as talking to a friend.
I’ve got to say, the intelligent eyewear we’re making at Sesame is a bit special, not least because of our collaboration with a deeply experienced team of craftspeople in Japan.
Our collection will be Made in Japan, available in 2027.
To celebrate the craftsmanship behind it, we commissioned a photograph by Osamu Yokonami. Today on the blog, we’re sharing more about our road to eyewear and a look behind the scenes of the commission.
sesame.com/journal/made-in-j…
Sesame preview now available on iOS ❤️
A collection of personal agents, crafted for everyday conversation.
Preview available now on iOS.
apps.apple.com/us/app/sesame…
Maya and Miles are joined by Simone and Charlie, and have far greater utility than they used to, while still being extremely conversational and low-latency. More info in the blog: sesame.com/blog/voice-your-c…
We're a small team @sesame, but looking for great people to join. Check out this podcast with @_apkumar to get a feel for how we work. More info at sesame.com/team
the core research team behind the sesame voice model is <9 ppl
as @_apkumar walked through in our latest 1.5 hr podcast, talent density beats team size most days
Kind words from @seanhollister 🙏. But we've still got a long way to go...
Sesame is the first voice assistant I’ve ever wanted to talk to more than once theverge.com/news/621022/ses…
We're just getting started at @sesame. Check out the demo here: sesame.com/voicedemo
We’re exploring a future where the computer isn’t just a tool—it’s a partner with a truly natural voice and personality. No big claims, just early work we’re excited to share. @sesame
Nice overview of some of the things we've been working on @sesame. Always a pleasure working with @justLV.
Excited to share a peek of what I’ve been working on
We @sesame believe voice is key to unlocking a future where computers are lifelike
Here’s an early preview you can try! 👇
We’ll be open sourcing a model, and yes…
we’re building hardware! 🧵
Delighted to share a little glimpse of what we've been working on @sesame
At Sesame, we believe in a future where computers are lifelike. Today we are unveiling an early glimpse of our expressive voice technology, highlighting our focus on lifelike interactions and our vision for all-day wearable voice companions. sesame.com/voicedemo
Excellent work by @sanchitgandhi99 and @yoachlacombe reproducing the text-description-to-speech model I developed while at @StabilityAI 👏❤️
Moving beyond naturalness and WER, they propose a set of sentences that test the model’s ability to deal with compound nouns, emotions, foreign words, paralinguistics (e.g. whispering if the text requires it) etc. etc.
The full test set is included in the appendix. 👏
2/7
The speech “de-tokenizer” (or decoder) is a convolutional model that’s streamable and 3x faster than their diffusion-based baseline (and also sounds better).
It’s built around BigVGAN.
6/7
There are a bunch of other interesting elements to this work, and it’s worth a read. Plenty of examples on the demo site too. Nice work Mateusz Łajszczak, @guillecambara, Yang Li, and all the other contributors.
arxiv.org/abs/2402.08093
Great TTS work from @amazon. In particular:
- Nuanced evaluation (I think we can all agree that “naturalness” is not enough)
- Useful model & data size ablations
- A new approach for “tokenizing” speech
- A fast method for “de-tokenizing” speech
amazon-ltts-paper.com/
🧵
For those asking - the model has ~300M parameters and was trained on a single A100 8x node for about a week. It can speak 8 languages fluently, but you’d probably want to increase the model capacity to get really good performance on more speaking styles.
Pleased to release the text-to-speech work I developed while at @StabilityAI. 💬
TL;DR - Natural language control of high-fidelity TTS.
It’s simple, generalizable, and it sounds better than Audiobox :)
text-description-to-speech.c…
arxiv.org/abs/2402.01912
🧵