AI Researcher, engineer @OpenMachineAI @EntropyAICo
San Francisco, CA
Joined January 2025
- Tweets52
- Following41
- Followers189
- Likes98
Pinned Tweet
TTS data cleaning: beyond VAD, diarization, and ASR
Where your audio comes from shapes how you clean it. Background music, multilingual speech, and uneven recording quality call for different checks. Here’s where four pipelines focus their effort.
🤖 Made with AI
CosyVoice 3
For multilingual data, CosyVoice 3 cross-checks transcripts from multiple ASR models and adjusts punctuation to match actual pauses, reducing transcription errors and punctuation that doesn’t fit the audio.
🤖 Made with AI
You can transcribe every word correctly and still end up with bad TTS training data.
Knowing what was said isn’t enough. You also need to know who said it, even when speakers keep switching back and forth.
Here’s a quick look at speaker diarization in our data pipeline.
The pipeline behind the demo: raw audio → transcription → diarization → alignment.
Words, timestamps, speaker turns, and overlaps all come together on a single timeline for TTS training.
2/n
We use pyannote Community-1 for diarization and NVIDIA Parakeet v3 for transcription.
We built our own alignment layer and reuse intermediate features to avoid redundant processing on long recordings.
A step-by-step guide to building a TTS training dataset at the scale of tens of millions of samples.
Over the past few months, we've been working on ways to collect and build high-quality TTS training datasets at scale. Compared to collecting text data for LLMs, working with speech data is significantly more complicated, with plenty of engineering challenges along the way.
There's surprisingly little public discussion about this, especially when it comes to the practical challenges of collecting speech data at scale.
So I wanted to share what we've learned over the past few months: the problems we've run into, the lessons learned, and how we designed and built our data engine from scratch.
Hopefully, this will be useful to others working on TTS and large-scale speech datasets. We're also hoping to open-source the full codebase down the road so others can build on it.
I also put together a video to make the article easier to follow.
Why limit ChatGPT to just one computer?
With Open Machine, you can connect your laptop and remote servers to ChatGPT and tell it exactly what to do on each one.
Run code on one machine, check logs on another, and leave the rest alone.
All from the same chat.
Recently, a friend and I built a little tool called Open Machine.
Basically, it uses MCP to connect ChatGPT to your local computer and remote servers, giving regular ChatGPT Codex-like capabilities, and in some cases even more flexibility. Things like running and editing code, managing remote servers, and handling browser tasks.
We actually built it for ourselves at first.
In our day-to-day work and model training, we kept running into the same problems: token and context limits, growing amounts of data that were hard to organize, and the hassle of sharing information and handing off work between teammates.
Model training was especially frustrating. We often had to manage multiple servers and constantly switch between our local machines and different remote environments.
So we wanted to build something that could bring all of this together and let AI handle more of the actual work.
That's really how Open Machine started.
We've now open-sourced it, although there's still a lot we want to improve.
Next, we're planning to build Desktop and Cloud, making it easier to manage multiple remote servers from one computer, and to index, sync, and update local files and databases. The idea is to make it easier for individuals and teams to build and manage their own local knowledge bases.
I'm sure a lot of people working on model training or software development have run into similar problems.
If this sounds useful, feel free to check it out! We'd also love to hear your feedback and ideas.
GitHub: github.com/OpenMachineLabs/o…
The future of AI is not just about answering what humans say, but understanding what they truly mean. This article explores why games may be the most powerful environment for training that capability.
After a year of hard work, we’re finally ready to meet you all.
This is the speech system we believe will redefine gaming. It doesn’t just generate speech , it understands your words, your tone, and your emotion, then responds in real time as part of the conversation.
I've recently been reading some relevant research and experiments, and I've discovered that AI-driven games can keep players engaged for longer. The reason is simple: NPCs are no longer cold, pre-scripted characters; they can surprise players and offer them unique experiences. This process significantly enhances player immersion and replayability.
However, AI-driven NPCs are not without their controversies. The emergence of LLMs has indeed brought unprecedented playability, but the uncertainty and high API costs have deterred many. I think the key issue lies in how to effectively utilize them.
In my opinion, AI is just the exterior; the story is the soul. A truly enjoyable AI game must have a "director" behind it, mastering key story points and driving the player's emotional climaxes. The AI can freely play with details, but the overall story's rhythm, structure, and tension are controlled and designed by a human.
Beyond that, AI brings more than just fresh dialogue. It empowers NPCs with environmental awareness, allowing them to sense the player's presence and actions. It allows NPCs to express more than just dialogue, expressing genuine reactions complete with emotions, voices, breathing, and even laughter. It enables NPCs to build memories, remembering every detail of their relationship with the player. It truly aligns dialogue and actions, and even radically alters NPC logic through a new behavioral decision tree.
I believe this isn't just a simple functional upgrade; it will usher in a new wave of gaming, rekindling the passion of players who once felt esports wasted. AI narrative games aren't just about "chatting"; they offer players unprecedented and unique experiences.
In future games, storytelling will still be the core of the game, but AI will be the lifeblood that gives that soul warmth and transformation.
I believe SLMs and local models represent the future trend, as they are sufficiently affordable to meet basic daily needs while prioritizing user data privacy and security, aligning more closely with real-world usage requirements. Particularly in the gaming industry, while LLMs can grant NPCs more flexible behaviors, the high costs of API services and concerns about data leaks remain glaring issues. In contrast, SLMs not only offer low-cost and privacy-preserving advantages but also provide players with exclusive memory continuity, allowing NPCs to truly become long-term companions uniquely yours.
For AI products, people generally think the intelligence dominates everything, while privacy and cost are seen as secondary. The industry’s path shows this: we’ve spent huge money in leading labs to build the largest models with exceptional intelligence.
But I believe we’ve overlooked another path that’s just as important — the case for local models.
Where local models are slowly emerging:
- Cognitive Kernel of the SLM OS. This is the cognitive core of the OS. It doesn’t need to be very large or know everything. it only needs to understand the user’s intent and call the right apps or tools when needed. Ideally, a few billion parameters will be enough.
It’s built directly into the OS with native support for text/audio/vision. So users never need to download or configure anything, and it will automatically return the result in the right format, whether text/audio or vision.
- Super Personal Assistant App. This is the application layer of the SLM OS. It is built as an execution agent that works offline with access to the local device and application data, coordinating and interpreting your actions.
For example, most AI assistants can only reply to an email. This one can pull from unified application data, summarize your meeting notes, and draft a reply the way you want, while leaving the final decision to send up to you.
It can also learn from user feedback, continually improving how it handles tasks. The killer feature is cross-app automation + local brain search. For instance, when you ask “When did I say XXX?” or “Where is the photo of me and XXX?” it can return the correct result in less than 500 milliseconds.
- Game characters in AI-native games. Traditional games rely on scripts and behavior trees to control game characters. After a few tries, everything feels repetitive and predictable, and players end up quitting. With SLMs combined with natural TTS, that logic is completely changed.
Through deep integration of SLMs with the game engine, every NPC can become a unique companion (with their own personality/background/speaking style). More than that, the storyline can follow the choices made by the player and their companions. This is what we call a “never-ending game.”
And these models live on your device, built right into the game files so you hardly notice them. They can remember the adventures you share, the stories you tell, and the things you care about. Over time, they can feel like your best friend.
Local models win on these factors:
- Low interaction latency: local models can respond in < 500 ms, with some native OS operations in < 50 ms. Game character speaking in < 800 ms close to human conversation speed.
- Private data access: the cognitive kernel of the SLM OS can natively access local data, while LLMs never can. Data quality decides everything for an AI product, so it is reasonable to see local SLMs perform better than LLMs.
- On-device finetuning: we may see better fine-tuning techniques that enable test-time training directly on edge devices. This would allow SLMs to improve personalization by learning from user interactions.
- Everyday tasks: most of the things we do each day are relatively simple. So we’d rather get an 85/100 answer in < 500 ms than wait 10 minutes for an LLM to call multiple tools just to give a 95/100 answer.
- Cost: whether it’s an OS or a game NPC, local SLMs can be used infinitely at zero cost, with no need to worry about inference expenses.
- Ownership: not your weights, not your brain.
Yes, LLMs will continue to get smarter, but most of our daily needs remain simple and unchanged. In some key domains, local SLMs can even perform better than LLMs. I believe we’ll see more impressive SLM use cases in the next 3–6 months, and it shouldn’t be a surprise if some of the best products don’t come from the big labs.
It's encouraging to see that by 2025, LLM development will no longer be a race to increase parameters, but will instead focus on efficient Model of Evidence (MoE), extended context handling, and building flexible ecosystems.
