a tiny schemer. engineering @trydaily and @pipecat_ai core maintainer.
Greater Los Angeles Area
Joined November 2009
- Tweets1.8K
- Following67
- Followers517
- Likes1.9K
Aleix Conchillo Flaqué retweeted
New Pipecat UI component: duplex timeline. I've been using this in dev for a while to visualize turns, latency, interruptions and back channeling.
ui.pipecat.ai/docs/component…
Our shadcn registry is "pipecat", if you're a CLI person.
npx shadcn@latest registry add @/pipecat
Aleix Conchillo Flaqué retweeted
This is awesome. Multi-character interactions in Unreal Engine, completely LLM-driven and unscripted. You can spin your own version of this up pretty easily.
This is another Pipecat + Jev experiment from @JonPTaylor.
Watch Jon walk around the game world, talk to the characters, and ask them to do things. They interact with Jon, with each other, and with the world.
Tech stack:
- One Pipecat agent process syncing state with Unreal via the Pipecat RTVI protocol.
- Three non-player characters. Each character is a Pipecat worker with its own LLM prompt and tools. Each character also has its own TTS service so its voice is unique.
- When Jon (the human) talks, Jev assesses Jon's intent and which character(s) he is talking to.
- Unreal 5.8 with Spatial Audio. (Watch the video with headphones on!)
- Pipecat C++ client embedded in Unreal Engine.
Aleix Conchillo Flaqué retweeted
Audition mode is coming along in Jamcat. The generated midi guide is used in audio-to-audio to generate stems that match your voice prompt. Each instrument type needs tuning and training, but it's not far off!
Aleix Conchillo Flaqué retweeted
Here's a complete voice agent tutorial focusing on a super low latency configuration: Pipecat PhoneLLM Alpha 1 on @modal, Deepgram transcription, Cartesia voice.
You can grab the code and jus run /setup in Claude or Codex.
But Merve's video also dives into the details of configuring model endpoints on Modal, the Pipecat bot file, and the front-end UI code in the example project.
Aleix Conchillo Flaqué retweeted
Introducing PhoneLLM, an open model for voice agents.
GPT 5.6 Terra performance on typical voice agent tasks at 1/3 the latency and 1/18 the cost.
For voice agents, we need models that are both very low latency and very good at tool calling and instruction following.
There's a trade-off here, and we often have to compromise on either latency or capability when building voice agents. With PhoneLLM (and the training and data stack that made this model possible) we're fixing this problem.
For the last couple of years, most of the effort in frontier model development has gone towards leveraging test-time compute. Which is awesome! Models of all shapes and sizes are available that perform really, really well ... if you have "thinking" turned on for your model.
But if you need your agent to respond at voice conversation speed, you can't use thinking models.
PhoneLLM is a full-weights fine-tune of NVIDIA Nemotron Nano 30B. We trained on a wide range of real-world telephone and customer support use cases. The training focused on taking the excellent Nano 30B base capabilities and teaching the model to do typical voice agent tasks with thinking disabled.
The results are really good: accurate tool calling and concise, on-topic responses in long conversations.
And fast: TTFAT measured server-side is <100ms if you run PhoneLLM on a lightly loaded B200. :-)
But seriously, when we characterize model latency, we do it with full, end-to-end, batched request simulations using real Pipecat voice agent pipelines.
You can serve more than 80 concurrent agents on a single B200 with P95 end-to-end TTFAT <600ms. Including network overhead. That's an LLM cost-per-minute around $0.0025. (1/4 of a cent.) At a latency lower than any third-party API offers today.
More details about this model, including weights on @huggingface, how to spin it up with one click on Modal, and a starter project repo you can clone, are in the thread ...
Aleix Conchillo Flaqué retweeted
Congrats to @DeepgramAI on the GA launch of Flux TTS, now live in Pipecat 🎉
@JonPTaylor looks at how Flux TTS delivers a more consistent voice experience. Flux reads the whole conversation, not just the next line — adaptive tone, consistent pronunciation, clean interruption handling natively (no SSML markup, no style tags!) all at sub-200ms latency.
➡️ docs.pipecat.ai/api-referenc…
Aleix Conchillo Flaqué retweeted
World Cup Voice Commentary right in your MenuBar!
@pipecat_ai powered voice bot, not only reports on the scores but can answer any questions I have about the games!
Using only local models on my M4 Max. Under the hood:
+ Kokoro TTS for commentary
+ Gemma 4e4 (with and without thinking)
+ Nemotron 3.5 for ASR
Built with Pipecat subagents: when the menubar starts, it spins up a main loop of
[STT]
|
[LLM] <-> [subagent when a game starts]
|
[TTS]
There is one subagent per game. Schedules are pulled regularly, so I can ask stuff like “Who will the US play in the round of 32?” and have it reason over fixtures, for example.
A collector ingests live commentary (per match-id, home/away, and metadata) into a local DB and exposes it via a CLI so pipecat's subagents can query it.
Each game subagent gets an initial prompt about “its” match and tools including an /analyze-game-skill. The skill was built and tested with the early R1 games, especially the chaos of Canada-Qatar game (with injuries, goals, and two yellow cards becoming a red!)
The result: when something happens in the game, the TTS chimes in, excited or dejected, about missed chances, corners, cards, etc. Subagents maintain their own context, so with push-to-talk, which invokes the STT, I can ask about what is happening in a current, past, or future games, fixtures, etc.
The main LLM routes to the correct subagent, or can directly check via CLI, or spins up a new subagent to go hunt down answers (e.g., scores about R1 games, schedule changes, etc.)
Today’s rain delays accidentally stress-tested having multiple concurrent games in Round 3, and the subagent orchestration held up nicely.
Clips + screenshot below 👇
Halftime announcement, second goal, and the third goal.
These were voice memo recordings, aargh
Aleix Conchillo Flaqué retweeted
"Every piece of software is going to have a voice interface. Everyone is going to have a voice as a primary input. I just don't think that idea has gotten the traction it deserves yet."
This week on Skywatch, @getbluejay_ai's car podcast, I sat down with @kwindla, co-founder of @pipecat_ai and @trydaily.
Kwin has been betting on real time infrastructure before most people knew they needed it. Ten out of twelve investors passed on Daily in 2016 saying they were just happy making phone calls. That same stubbornness eventually built Pipecat, one of the most widely used open source framework for voice agents. NVIDIA, AWS, and some of the largest companies in the world have standardized on it.
Before all of that, he was building gestural interfaces that ended up in Minority Report. Steven Spielberg shot demo reels for his company. Then came room sized systems for Fortune 500 companies. Then the bet nobody else wanted to take. A few things from this conversation I have not stopped thinking about:
- Almost every voice agent deployed in production today runs on a foundation model from one of the big labs. Kwin thinks that will not be true in two or three years. The future is open, customizable, low latency models built specifically for real time use cases.
- During early COVID when video usage exploded overnight, Kwin did not sleep more than three hours at a time for three months. They kept adding servers and chasing the next bottleneck. A lot of that growth turned out not to be durable revenue. He kept serving everyone anyway. That is the kind of thing that builds real infrastructure companies.
- Continual improvement is still unsolved. Closing the loop so your agents get better automatically from production data without human intervention at every step is what Kwin is calling the defining engineering challenge of 2026.
If you are building in voice AI, real time infrastructure, or agentic systems, this one is worth your time.
Full episode on YouTube and Spotify in the comments 👇
Aleix Conchillo Flaqué retweeted
Voice AI Meetup, Thursday May 7th. This one's a special crossover event. T-Bot, who hosts the global Voice AI Spaces meetups, is visiting San Francisco and will MC!
- NVIDIA researchers will present some of their really cool recent work on speech models.
- We'll have demos and two fireside chats, featuring new developments in models and evals, with @GradiumAI, @ArtificialAnlys, @ServiceNow, and @pipecat_ai.
- And, of course, 🍕 and great conversation.
- Thanks to the @trychroma team for hosting in their wonderful office/event space.
Registration link below. Come hang out with 150 old and new friends!
🤖 Made with AI
Aleix Conchillo Flaqué retweeted
Sub-agents in (latent) space!
We’ve been working on a side project.
As far as I know, this is the first massively multiplayer, completely LLM-driven game. Come play Gradient Bang with us. See if you can catch me on the leaderboard.
This whole thing started because I wanted to explore a bunch of things I’m currently obsessed with, in an application of non-trivial size, that felt both new and old at the same time.
So … a retro-style space trading game built entirely around interacting with and managing multiple LLMs. Factorio, but instead of clicking, you cajole your ship AI into tasking other AIs to do things for you.
Some of the things we’ve been thinking about as we hack on Gradient Bang:
- Sub-agent orchestration
- Partial context sharing between multiple LLM inference loops
- Managing very long contexts, and episodic memory across user sessions
- World events and large volumes of structured data input as part of human/agent conversations
- Dynamic user interfaces, driven/created on the fly by LLMs
- And, of course, voice as primary input
If you’ve been building coding harnesses, or writing Open Claw agents, or doing pretty much anything that pushes the boundaries of AI-native development these days, you’re probably thinking about these things too!
This is all built with @pipecat_ai, the back end is @supabase, the React front end is deployed to @vercel, and all the code is open source.
Aleix Conchillo Flaqué retweeted
Join us on Thursday in SF for conversations about voice agents, speech models, and realtime AI infrastructure.
I'm on a panel with:
- @natrugrats from @DeepgramAI
- @farazmsiddiqi from @getbluejay_ai
- Aaron Lee from Parakeet Health
There will be food and lots of opportunities to ask questions and share your knowledge.
One thing I'm looking forward to is comparing notes about GTC last week.
Aleix Conchillo Flaqué retweeted
NVIDIA Nemotron 3 Super launches today! We've been building voice agents with Super's pre-release checkpoints and running all our various tests and benchmarks.
Nemotron 3 Super matches both GPT-5.4 and GPT-4.1 in tool calling and instruction following performance on our realtime conversation, long context, real-world benchmarks. GPT-4.1 is the most widely used LLM today for production voice agents. So an open model that performs as well as GPT-4.1 on hard, voice-specific benchmarks is a big deal.
(Side note: we don't think a benchmark "tells the story" about a model's voice agent performance unless it tests model correctness across at least 20 human/agent conversation turns.)
The Nemotron models are *fully* open: weights, data sets, training code, inference code.
Nemotron 3 Super is 120B params, with a hybrid Mamba-Transformer MoE architecture for efficient inference. You can run it on NVIDIA data center hardware or on a DGX Spark mini-desktop machine.
1M token context.
Blog post with full benchmarks, thinking budget notes, inference setup on @Modal, and where we think this goes next. 👇
Aleix Conchillo Flaqué retweeted
Voice-controlled UI.
This is an agent design pattern I'm calling EPIC, "explicit prompting for implicit coordination." Feel free to suggest a better name. :-)
In the video, I'm navigating around a map, conversationally, pulling in information dynamically from tool calls and realtime streamed events.
There are two separate agents (inference loops) here: a voice agent and a UI control agent. They know about each other (at the prompt level) but they work independently.
Aleix Conchillo Flaqué retweeted
Benchmarking LLMs for voice agent use cases. New open source repo, along with a deep dive into how we think about measuring LLM performance.
The headline results:
- The newest SOTA models are all *really* good, but too slow for production voice agents. GPT-4.1 and Gemini 2.5 Flash are still the most widely used models in production. The benchmark shows why.
- Ultravox 0.7 shows that it's possible to close the "intelligence gap" between speech-to-speech models and text-mode LLMs. This is a big deal!
- Open weights models are climbing up the capability curve. Nemotron 3 Nano is almost as capable as GPT-4o. (And achieves this with only 30B parameters.) GPT-4o was the most widely used model for voice agents until quite recently, so a small open weights model scoring this well is a strong indication that production use of open weights models will grow this year.
Voice agents are a moderately "out of distribution" use case for all of our SOTA LLMs today. Literally, in the sense that there's not enough long, multi-turn conversation data in the training sets.
Everyone who builds voice agents knows this intuitively, from doing lots of manual testing. (Vibes-based evals!) This benchmark scores LLMs quantitatively on instruction following, tool calling, and knowledge retrieval in long-context, multi-turn conversations.
Aleix Conchillo Flaqué retweeted
Voice-only programming with Claude Code ...
I've been playing with @aconchillo's MCP server that lets you talk to Claude Code from anywhere, today. I always have multiple Claudes running, and I often want to check in on them when I'm not in front of a computer.
Here's a video of Claude doing some front-end web testing, hitting an issue and getting input from me, and then reporting that the test passed.
In the video the Pipecat bot is using Deepgram for transcription and Cartesia for the voice. (Note: I sped up the web testing clickety-click sections of the video.)
The code for the MCP server and the Claude skill are in the repo and Aleix wrote a really good README.md.
You can use any of Pipecat's network transports: generally WebRTC, but you could set this up so you can call Claude on the phone if you wanted to.
There's screen capture support, too, so you can view the Claude code window remotely. That's still a little experimental.
Because this is an MCP server, it's not specific to Claude Code. Try it in other environments! It should work in Clawdbot, Codex, etc ...
Aleix Conchillo Flaqué retweeted
Pipecat Cloud is @trydaily's enterprise hosting platform for open source voice agents. Today, after a 9-month beta period, we're promoting Pipecat Cloud to General Availability!
With Pipecat Cloud, you build your voice agent on @pipecat_ai’s open source, vendor neutral core, add your custom code and agent logic, and then “docker push” to Pipecat Cloud.
As with everything we do, Pipecat Cloud is engineered to give you flexibility, to not lock you into any service, including Pipecat Cloud itself. Any code that you can host on Pipecat Cloud you can self-host with no changes at all.
We've focused on delivering:
- fast agent start times (P99 <1s)
- multi-region hosting
- optimized global network transport
- direct connectivity to Twilio, Telnyx, Plivo, Exotel and other telephony providers.
- built-in @krispHQ VIVA models for noise reduction and turn detection
- integrations with all the AI services, observability tools, and everything else supported by Pipecat
You can sign up and "pipecat cloud deploy" immediately. We also have enterprise support contracts and can work with you to deploy a single-tenant, enterprise version of Pipecat Cloud in your VPC. Feel free to contact us if you have questions.
Aleix Conchillo Flaqué retweeted
Pipecat Cloud is now generally available.
Pipecat Cloud is a managed, vendor-neutral platform for deploying and scaling open source voice agents, with ultra-low latency, multi-region support, and enterprise-grade realtime infrastructure. Thank you to the more than 1,000 teams that built and scaled with Pipecat Cloud during the platform beta.
Aleix Conchillo Flaqué retweeted
🎉 We are proud to support @nvidia's new Nemotron models, announced today at CES2026.
We've been building high-performance voice agents with the new NVIDIA Nemotron Speech ASR model and integrating this model into Pipecat.
Nemotron Speech ASR is completely open (weights, training data, inference tools), designed from the ground up for low-latency use cases like voice agents, and scores very well on our benchmarks. It also runs cost-effectively at large scale. Congratulations to the NVIDIA team on their open model breakthroughs and stay tuned for news all week from CES.
Learn More: blogs.nvidia.com/blog/open-m…
Aleix Conchillo Flaqué retweeted
This robot assistant from the NVIDIA CES Keynote on Monday is going viral.
@NaderLikeLadder explains all the hottest emerging AI trends in one demo: AI applications in 2026 will be multi-model, multi-modal, hybrid cloud/local, use open source models as well as proprietary models, control robots and embedded devices in the physical world, and have voice interfaces. (And the demo had a cute robot *and* a cute dog. Gold.)
The demo was built with @pipecat_ai. NVIDIA posted a really nice technical walk-through and complete code. The Reachy Mini robot from @huggingface is open source hardware. (You can order it now, I have one!). You can run the assistant locally on your own hardware, in the cloud, or both.
Aleix Conchillo Flaqué retweeted
NVIDIA just released a new open source transcription model, Nemotron Speech ASR, designed from the ground up for low-latency use cases like voice agents.
Here's a voice agent built with this new model. 24ms transcription finalization and total voice-to-voice inference time under 500ms.
This agent actually uses *three* NVIDIA open source models:
- Nemotron Speech ASR
- Nemotron 3 Nano 30GB in a 4-bit quant (released in December)
- A preview checkpoint of the upcoming Magpie text-to-speech model
These models are all truly open source: weights, training data, training code, and inference code. This is a big deal! Jensen said in the CES keynote yesterday that he expects open source models to catch up to proprietary models this year in a number of categories. NVIDIA is putting their weight behind making this happen. (As Alan Kay said, the best way to predict the future is to invent it.)
The code for this agent is open source too, of course. You can deploy it to production with @modal and @pipecat_ai cloud, or run locally on an @nvidia DGX Spark or RTX 5090.