Pinned Tweet
Introducing Firedrill, a simulation and testing platform for AI agents. Run thousands of simulations to find where your AI agents break before production.
Before an agent handles customer work, you need to test how it behaves with different data, permissions and tool failures. Doing that across apps and tools means managing test accounts, recreating scenarios and cleaning up after every run.
Firedrill spins up synthetic tools with data you control, so you can test your existing agent without using production accounts. Choose from the library, including Gmail, HubSpot, Stripe and more.
Run workflows under different conditions, inspect what happened and test your fixes. Turn those scenarios into repeatable checks in your PR workflow to catch regressions before they reach customers.
Firedrill retweeted
The judges are in!
We’ve got a jury from Meta, SpaceXSI, Plato, Intuit, Amplitude, Superlinked, Persona SingleStore and more.
including @ReeyahSDasoriya @lingxi, @svonava, @rahulparundekar, @samfrannn, @KovacVeljko, @newtonasareII, @llewrek, and @akwong898.
May the best agents win.
luma.com/personalagenthack
Firedrill retweeted
Your personal agent should know what you’re working towards, who matters to you, and how you make decisions.
The better it understands you, the more it can do for you.
This Friday, we’re getting together the best builders to build personal agents with @typesafeai @Meta @MongoDB @Amplitude_HQ @statsig and @superlinked
📍 SF
📅 Fri, Nov 20
🏆 $9,000+ in cash, gift cards, credits, and other prizes
More partners include @AirbyteHQ @arcee_ai @RunFiredrill @GuildAI @withpersona @SingleStoreDB @cognee_
Limited spots. Apply: luma.com/personalagenthack
We’re hyped for the Personal Agents Hackathon in SF on Nov 20th!
If you’re building a personal agent, come hack with us. We’ll have Firedrill there so you can test what you build in a synthetic world of apps and data before demo time.
Apply: luma.com/personalagenthack
Firedrill is excited to co-host Personal Agents Hackathon, Nov 20, SF, alongside @typesafeai, @Meta, @AWS, @MongoDB, @Amplitude_HQ, @AirbyteHQ & @tokensandai.
Our CTO @das__kiran will be speaking, and our CEO @newtonasareII will be on the judging panel.
Apply here: luma.com/personalagenthack
Firedrill has been selected for the @MetaforDevs Global AI Developer Hackathon.
We'll be exploring how Meta's Muse models can expand our simulation capabilities, helping us create more realistic environments and a wider range of scenarios for AI agents.
Firedrill is a training ground where AI agents can practice, learn from failures, and improve themselves.
We're excited to be working with Meta and all the amazing partners @baseten @cerebral_valley @modal and @FireworksAI_HQ.
When you score an AI-agent simulation, decide which requirements must be satisfied exactly.
Firedrill lets you simulate your AI agent in a synthetic world with controlled scenarios, using synthetic tools and test data when needed. Your existing AI application runs in your own runner.
If you add an assessment, keep quality scores and exact checks separate.
Consider a hypothetical invoice-summary agent. Grade how clearly it explains the charges. Separately, check the structured total and currency against the supplied invoice data.
A polished explanation should not compensate for the wrong total. Keep that failed check visible even when the quality score is high.
Firedrill’s reports retain these as separate results. You define the criteria and the exact requirements for the task.
Use graded criteria for meaning, and exact checks for values that must match.
Scoring guide: lnkd.in/g7buSkUT
We will be at AGNTCon + MCPCon in San Jose, Oct 22–23 hosted by @linuxfoundation. Firedrill helps you simulate AI agent workflows with synthetic tools and test data to catch failures before they reach users.
Building agents? DM us to meet.
The team is heading to AGNTCon + MCPCon by @linuxfoundation, Oct 22–23 in San Jose!
Firedrill helps AI agent builders simulate agent behavior across synthetic tools and test data to find failures before touching customer accounts.
Building agents? Let’s meet and talk simulations.
See what your AI agent changed.
When an agent updates records across tools, you need to know whether those changes match the task you gave it.
Firedrill is a simulation and testing platform for AI agents. Connect your existing agent to synthetic tools, run a task with test data, then inspect the records it changed.
For a task that updates a record, start with three checks:
Did it update the right record?
Did it change the intended fields?
Did it leave unrelated records alone?
Those checks turn a test run into something you can investigate, fix and test again.
Give your agents somewhere to practice before production.
docs.firedrill.run/concepts/…
2
63
We simulated 1,575 scenarios for an AI agent using Jev (@typesafeai )
We've been experimenting with this, and I'm excited about what it opens up for AI agent testing.
Manual testing gets you through a few workflows. The tools respond. The answers look reasonable. You start feeling ready to ship.
But a successful tool call can still end in a wrong answer.
That's exactly what we found.
We connected our agent to synthetic Gmail, HubSpot, Stripe, QuickBooks, Slack, and Google Calendar in Firedrill. Then tested it with different data, missing records, conflicting information, and prompt injections.
Firedrill captured what the agent actually did. Jev evaluated its answers against the tool records.
1,575 scenarios.
3,684 tool calls.
Two confirmed wrong answers after successful lookups.
Two failures we could investigate before putting customer accounts on the line.
Here's where this gets exciting: developers can give their existing agents a practice ground. Connect to Firedrill's synthetic tools and test data, configure the situations you want to test, and evaluate both the actions and the answers.
Imagine putting your agent through thousands of messy situations before asking a customer to trust it. Finding the failures. Testing the fixes. Running those checks again.
That's the future of agent development we're building toward.
Keep building your agent. Give it somewhere to practice.
Find the failures before your customers do.
4
114
Give your AI agent somewhere to practice before production.
Firedrill is a simulation and testing platform with synthetic tools and data, so you can test agent actions without using live business records.
firedrill.run
2
60
Introducing Firedrill. The simulation and testing platform for AI agents.
Your agents take actions across tools and apps. How do you catch a failure that only shows up five steps into a workflow? With thousands of paths your customers could take, how many have you actually tested?
Simulate thousands of scenarios in minutes. Catch failures, uncover mistakes, and improve your agents before they reach production. Turn those scenarios into repeatable checks on PRs to catch regressions before merge.
Dry runs for AI agents. Before real consequences.
Available now, hosted or open source.
Sign up at firedrill.run
Github: github.com/firedrill-tools/f…
1
7
27,923













