we can finally approximate how much compute it took all of evolution to reach our level of complexity
Can an LM, starting from random init (!!), learn to generate all of its pretraining data?
Introducing Self-Play Pretraining with Zero Data. Two models start from random initialization: a generator proposes programs for a universal Turing machine and a learner trains on their outputs. We never train on any real data, but see predictable scaling on natural datasets: zero-shot val loss on images, text, audio, and melodies decreases predictably with self-play compute. And the learner develops in-context learning capabilities.
A fun proof-of-concept, co-led with @AdityaCowsik and @KfirDolev and co-authors @gbruno_dl, @ANourya @noahdgoodman, and @YoavLevine.
Replying to @ipivto
shot on 35mm! big shoutout to twentyfivecent instagram.com/p/DYz0_OnjFrL/…
emi retweeted
Design, craft and quality are the key differentiators now — both for customers and for talent.
One of the biggest reasons I decided to join @base is because @jessepollak made it clear that craft and quality is genuinely something that himself and the team care about. I can’t wait to build that culture even further.
emi retweeted
People can work surprisingly hard without burnout as long as they feel like they’re winning.
emi retweeted
launching the training run by manually sshing into each node and starting the script because you cba to figure out how the cluster management software works
emi retweeted
tfw when you're doing a big training run and saving checkpoint kills off a GPU fan and it starts thermal throttling
emi retweeted
Confession: I regret vibe coding so much over the past year or so and am actively trying to depend less on LLMs for writing code. I can't help but feel like my focus, debugging skills, and syntax muscle memory is worse off. Trading low level abilities for speed and product sense is good in the short term, but is not sustainable over a long career in technology.
emi retweeted
i love how every time i have a hard problem i want to solve now, i'll spin up 8 tabs and ask the same question to:
- sonnet 3.7
- sonnet 3.7 thinking
- gpt 4.5
- o3-mini-high
- o1-pro
- o3-deep-research
- gemini 2.0 pro experimental
- grok 3 thinking
emi retweeted
crazy that python doesn't have async await. all data analysis should have been done in javascript
Readers added context they thought people might want to know
Python has coroutines and async await keywords.
Official documentation:
docs.python.org/3/library/asyn…
emi retweeted
The last thing your AI lab sees before getting one-shotted by a Chinese bootleg LLM team with a $5M cluster – at San Francisco, CA
emi retweeted
Audio is the shortest road to AGI.
Everyone is pushing for AI to be the ideal of rational intelligence. A fulcrum of sterile and accurate knowledge, devoid of bias and opinion, this version of AI will certainly change the world, but a lack of creativity on our part is stopping it from becoming a revolution.
What everyone truly wants is to be able to relate and be understood; everything else is a corollary. We are social animals.
AI can solve this, but it can't be gray, metallic and neutered. It needs to be emotional, natural, and real. We must re-formalize AI as something that can portray emotions and theory of mind, not just spit out facts.
This is why the current generation of LLMs are not classified as AGI. AGI is shakily defined on a technical level, what really matters is when we collectively feel AGI has arrived, and I argue that IQ wise current LLMs are already well past what’s needed for that, the bottleneck is largely just emotional quotient (EQ) and social intelligence. True AGI needs to maintain the illusion of having a Self, emotional understanding, and empathy.
Making an AI that has theory of mind, has high EQ, and feels as natural to talk to as a friend might sound like wishful thinking, and as long as we stay within the paradigm of text language models and TTS it will be! It turns out, however, there is an elegant solution to this problem, and the bottleneck is in our data medium of choice for language models: text.
Text represents a fraction of the human condition. It is a hyper compressed, low bit rate, quantized representation of language and human thought. This makes it very easy to train language models on it, but the compromise is losing out on all the components of human interaction that make us most human: prosody, emotion, and time. All of this information is encoded in audio through rhythm, stress, and intonation.
The solution is audio. Specifically, training a language model directly on speech, circumventing text. This type of model is called a SpeechLM.
There are four massive advantages to training a speechLM (an LM directly on audio) compared to an LLM + TTS based conversational AI like ChatGPT voice.
1. Prosody and emotion as input.
ChatGPT voice only takes the pure semantic content of your speech as input, having extracted it from reductive ASR models like whisper. The output is not dependent on your mood or delivery: if you yelled at it or cried to it, it would make no difference.
2. Interruptibility.
Audio, unlike text, has a notion of time. This opens the doors to interruptible conversations, as a speechLM could always be generating audio (silence sometimes), it must only be faster than real time. Current conversational AIs are turn based, no matter how low the latency, this is wholly unnatural. Two people can have a conversation talking over each other the entire time and understand each other perfectly.
3. Thinking ability.
Another hidden benefit of the notion of time is in the property of silence. Models can learn to use silence as a thinking capacity for the incoming tokens, kind of how when you respond to someone you naturally take a second to think. This solves the age-old problem of a dynamic computational budget per token. Text models don’t have this: they spend the same amount of FLOPs telling you what the meaning of love is as counting to one hundred. The amount spent between writing one word or another is not encoded in text.
4. Feedback loop between semantics and prosody.
This inches at what it really means to be human, the relationship (or conflict) between our rational and irrational self, our id and our ego. As you talk, the content of your speech is influenced by your mood, and vice versa, in ways you rarely realize. This property creates an emergent effect in behavior that is key to defeating uncanny valley and achieving theory of mind.
So why hasn’t this been done? One would think that training a language model directly on conversations will yield higher quality conversations and agents than training on text. Babies learn language through audio before ever encountering text!
Well, it has been tried.
arxiv.org/abs/2305.13009
arxiv.org/abs/2209.03143
The issue is that current autoregressive language model architectures do not scale well to audio. Results at best achieve local grammatical correctness and at worst a pseudo sims language. Audio is too high dimensional, and our architectures and audio representations are not hierarchical enough. Scaling up doesn’t seem to help either, the largest SpeechLM ever trained was 13b parameters and it still sucks. In addition, everyone that has tried this has mostly just trained on audiobooks, a laughably bland window of human social interaction and conversation. Thankfully, there is a near infinite amount of conversational audio data on the internet.
The architecture that will solve speechLMs is one that can simultaneously and efficiently process audio at different levels of resolution, mainly:
a. ~5hz (pseudo text)
b. ~20hz (prosody, emotion, style)
c. ~150hz (speaker timbre/identity)
TTS attempts to model b and c as a function of a. We need a model that attempts to model a, b, and c as a recursive function of themselves.
So what is the shortest possible path to building this? Well OpenAI can’t decide to pull a Sora, scaling current architectures doesn’t help. We need new architectures. One could start a research lab and hope for a breakthrough - I would argue, however, that the best way to solve a problem is to play with it, and one great way to play with it is by making a product. Thankfully, generative audio is not just a road to AGI, but also a fantastic product.
And you don’t need to do text-to-speech, in fact, that is probably a bad idea. TTS is highly supervised, which means there is much higher tolerance for error in the architecture and little to no motivation to push semantic abilities. A much better problem to work on is music and general audio generation. This problem requires deeply hierarchical and creative capabilities, and you can immediately get satisfactory results with it.
Demand for generative general audio is only set to exponentially grow from here on out. It will become a core ingredient of creator tools for game, music, and film and it will also power all AI generated video and interactive experiences (did you notice all the sora videos were missing sound?). All digital content will be generated, and sound is indispensable and unanimous.
In conclusion, we could keep pushing code and text based models, attempting to reach an impossible ideal of intellectual intelligence, or we could realize that all AGI needs is a high EQ and feel real. The only way to achieve that is through audio-based speechLMs. SpeechLMs need new architectures, and the best way to develop those is iteratively, by working on a music and general audio product. Audio is the shortest road to AGI. This is why I am building @audiogen_co.
If this excites you, reach out! I am always looking for people to share ideas with. If you want to help build this, join us at Audiogen ;)
We just raised a pre-seed from Entrepreneur First in London, are moving to San Francisco, and are applying to aigrant.com. Wish us luck!
emi retweeted
Take a look at the first ten startups from our latest batch pitching at Demo Day in San Francisco 🧵
emi retweeted
I was into music production long before getting into ML. Heres a brain dump of some predictions:
- Music generative AI is not going to replace music production, songwriters, etc
- two lanes are emerging: full song generation (suno, etc) AND ai tools for artists
- full ai generated songs won’t pollute the billboard charts significantly, but will become incredibly prominent, heard everywhere.
- a few “virtual artists” with good branding, backed by big record labels with lots of $$$, will become popular - a la vtubers. They’ll churn out fully ai generated music to their fans. Record labels don’t have to pay them! We know they’ll love that 🤑. Printing money.
- however, soon a very significant amount (>80%) of billboard charts will contain songs that have ai generated audio samples in the same way sampling became prominent in the past 20-30 years, via ai music tools for creators.
- just like chat gpt outputs are everywhere in homework, press releases, emails, etc. the winners of the ai music creation tool space will have their outputs in most of the music being played on the radio. These samples will be manipulated just like producers currently manipulate samples they find for their songs
- the “winners” here will be ones with best distribution to music producers
- barrier to entry for music production will continue to be lowered at a level comparable to how it was lowered with YT tutorials over the past 15y ➡️Young artists ++
- the gap between open and closed models for music will be larger than with LLMs. I hope I’m wrong here, but unfortunately the copyright issues are hairier which incentivizes folks to keep their model weights trained on unlicensed data closed so they avoid get pwnd by scary top record label lawyers



















