Pinned Tweet
Introducing AudioPaLM: A Large Language Model That Can Speak and Listen
- LLM trained on speech and text
- Fused with magic of AudioLM
- Application to speech recognition and speech-to-speech translation
Demo website: google-research.github.io/se…
Paper: arxiv.org/abs/2306.12925
Google presents AudioPaLM: A Large Language Model That Can Speak and Listen
paper page: huggingface.co/papers/2306.1…
introduce AudioPaLM, a large language model for speech understanding and generation. AudioPaLM fuses text-based and speech-based language models, PaLM-2 [Anil et al., 2023] and AudioLM [Borsos et al., 2022], into a unified multimodal architecture that can process and generate text and speech with applications including speech recognition and speech-to-speech translation. AudioPaLM inherits the capability to preserve paralinguistic information such as speaker identity and intonation from AudioLM and the linguistic knowledge present only in text large language models such as PaLM-2. We demonstrate that initializing AudioPaLM with the weights of a text-only large language model improves speech processing, successfully leveraging the larger quantity of text training data used in pretraining to assist with the speech tasks. The resulting model significantly outperforms existing systems for speech translation tasks and has the ability to perform zero-shot speech-to-text translation for many languages for which input/target language combinations were not seen in training. AudioPaLM also demonstrates features of audio language models, such as transferring a voice across languages based on a short spoken prompt.
🚀
Today, we're releasing an updated Gemini 2.5 Flash Native Audio model. Now available via the Live API 🗣
blog.google/products/gemini/…
1
137
We are hiring! Come and join us in Google DeepMind in Zurich to work on next generation audio and speech technologies 😎
job-boards.greenhouse.io/dee…
Paul Rubenstein retweeted
Gemini 2.0 Flash comes with native audio output, and it’s actually wild 🤯 we are working hard to roll this out quickly to more folks!
Paul Rubenstein retweeted
Finally, native audio generation with Gemini! m.youtube.com/watch?v=qE673A…
Paul Rubenstein retweeted
I'm hiring!
I'm building 4 research groups under me at AISI (formerly the UK's Taskforce on Frontier AI) to work on foundational AI safety research.
[1/5]
gov.uk/government/publicatio…
The Gemini paper is on arxiv! It has been amazing working on Gemini's audio capabilities, building on earlier work with AudioPaLM. Humbling to have so many brilliant co-authors 🙏
arxiv.org/abs/2312.11805
18
1,181
Gemini is public! It has been awesome working on this with amazing colleagues to endow LLMs with audio capabilities 🙏
I’m very excited to share our work on Gemini today! Gemini is a family of multimodal models that demonstrate really strong capabilities across the image, audio, video, and text domains. Our most-capable model, Gemini Ultra, advances the state of the art in 30 of 32 benchmarks, including 10 of 12 popular text and reasoning benchmarks, 9 of 9 image understanding benchmarks, 6 of 6 video understanding benchmarks, and 5 of 5 speech recognition and speech translation benchmarks. Gemini Ultra is the first model to achieve human-expert performance on MMLU across 57 subjects with a score above 90%. It also achieves a new state-of-the-art score of 62.4% on the new MMMU multimodal reasoning benchmark, outperforming the previous best model by more than 5 percentage points.
Gemini was built by an awesome team of people from @GoogleDeepMind, @GoogleResearch, and elsewhere at @Google, and is one of the largest science and engineering efforts we’ve ever undertaken. As one of the two overall technical leads of the Gemini effort, along with my colleague @OriolVinyalsML, I am incredibly proud of the whole team, and we’re so excited to be sharing our work with you today!
There’s quite a lot of different material about Gemini available, starting with:
Main blog post: blog.google/technology/ai/go…
60-page technical report authored by th Gemini Team: deepmind.google/gemini/gemin…
In this thread, I’ll walk you through some of the highlights.
Paul Rubenstein retweeted
Introducing Gemini 1.0, our most capable and general AI model yet. Built natively to be multimodal, it’s the first step in our Gemini-era of models. Gemini is optimized in three sizes - Ultra, Pro, and Nano
Gemini Ultra’s performance exceeds current state-of-the-art results on 30 of the 32 widely-used academic benchmarks. With a score of 90.0%, Gemini Ultra is the first model to outperform human experts on MMLU.
blog.google/technology/ai/go…
Paul Rubenstein retweeted
PaLM + AudioLM = AudioPaLM !
We start from PaLM pretrained on text and extend its vocab w/ audio tokens. This model can then be finetuned on a mix of any (speech, text) task e.g. ASR, TTS, MT and speech2speech translation in one's voice! 🧵1/4
google-research.github.io/se…
My internships at Google were incredible learning experiences where I worked with some of the best people I've ever met, and they were pivotal to my post-PhD career. Applications for 2023 are now open until Oct 28th careers.google.com/jobs/resu…
Paul Rubenstein retweeted
Happy to announce DreamFusion, our new method for Text-to-3D!
dreamfusion3d.github.io
We optimize a NeRF from scratch using a pretrained text-to-image diffusion model. No 3D data needed!
Joint work w/ the incredible team of @BenMildenhall @ajayj_ @jon_barron
#dreamfusion
Paul Rubenstein retweeted
Given the soaring energy prices, Insulate Britain don't look so extreme now do they?
Paul Rubenstein retweeted
DALLE-2 has a secret language.
"Apoploe vesrreaitais" means birds.
"Contarra ccetnxniams luryca tanniounons" means bugs or pests.
The prompt: "Apoploe vesrreaitais eating Contarra ccetnxniams luryca tanniounons" gives images of birds eating bugs.
A thread (1/n)🧵
I asked DALLE-2: "What is tastier, a burger or sushi?"
Its answer: why choose? #dalle
1
"A mysterious watercolor painting of a figure meditating, a fine mist connects its brain to the universe". #dalle
"A watercolor painting of two figures meditating, their brains are connected by a fine mist" #dalle
1
"A watercolor painting of a group of people living in the moment, a cloud of mystical positive energy above them" #dalle
1


























