Mainak Mazumdar

More validation from our AI Science research on Evals: • Small classifiers, adapted with either full fine-tuning or low-rank adaptation (LoRA), can outperform generalist models 10x their size. • Generalist models retain a significant edge on inputs unlike the data the small classifiers were trained on. • Training on post-verdict reasoning lets large teacher models produce superior student classifiers without the latency penalty that reasoning imposes at prediction time. arxiv.org/abs/2610.33813

Today, we celebrated our 8th anniversary at @fiddler_ai, remembering some highlights from our journey from explaining machine learning models to building a control plane for AI agents. Our mission around building trust in AI has never been so important. Really grateful to all our employees, investors, customers and partners for being a part of this journey! More to come!

🔀 Dynamic Model Escalation For Efficient Evals 💰 Most eval pipelines pick one judge model and run it on everything. If it’s a big model, it’s too slow and too expensive to score production traffic, so teams sample a few percent and hope. If it’s a small model, it’s cheap enough to score every trace, but nobody knows which of its scores to trust. The question isn’t which judge model to use. It’s when the judge should admit it doesn’t know. Here is what we have built at @fiddler_ai as part of our Centor Evaluation Runtime. Two models function with two distinct jobs. A decisioning model sits in the hot path and scores every span/trace in under 100ms. A reasoning model sits behind it and only ever sees what the decisioning model sends it. The interesting part is the handoff. The decisioning model doesn’t just emit a verdict. It emits a verdict plus an uncertainty score about how much that verdict should be trusted, derived from the trace itself and from where this verdict sits relative to everything the model has scored before. That signal is what routes. Traces the decisioning model is sure about keep their score, and that’s the majority of traffic. Traces where the signal says the model is operating outside its competence go to the reasoning model, which re-scores them with the time and context a fast model can’t afford. So the split isn’t fast versus accurate. It acutally works as decide JIT versus deliberate post-hoc analysis. The decisioning model’s job is to be right quickly and to know when it isn’t. The reasoning model’s job is to be right on the cases that need thought. One eval score comes out the other end, and the user never needs to know which model produced it. Ofcourse we keep an audit of all of this if user wants to actually know the details. This is very similar to how humans perform a good on-call rotation. The pager doesn’t wake the senior most engineer (large reasoning model) for every alert. It wakes them for the ones that the on-call engineer (fast decision model) using the the runbook can’t resolve. So, what are the main advantages of this: The reasoning model’s bill is a function of how unusual your traffic is, not how much of it there is, which is the only way scoring every production trace stays affordable. And the escalation queue becomes the most useful dataset you have: a running list of where the decisioning model is weak and where your traffic is drifting. When you label those, feed them back into the next version of the decisioning model, and the escalation rate drops over time. The eval system tunes itself on its own misses. Welcome to the world of self-tuning AI Observability! Eval Models: fiddler.ai/centor-models

AI Control plane is becoming super important to connect dev and security in the fast changing AI space. Great food for thought by @JoshARosen !

Cheaper tokens are not always great and could lead to expensive outcomes. In our initial coding-agent experiment, a model with half the token price costed 8–14x more per verified success. Also, Model rankings changed with the workload, so there was no universal winner. We have to connect token spend to what the agent actually executed and whether it succeeded. More details here: fiddler.ai/blog/cheaper-ai-m…

"Agents gone wild" is the new normal. Frontier labs keep pouring 100s of millions into traditional observability tools that fail to catch and prevent these incidents. Quite shocking indeed! techcrunch.com/2026/09/04/an…

Cyber is becoming an especially good proving ground for AI agents because the feedback loop is so concrete: find a vulnerability, understand the trajectory that found it, fix it, and use that learning to make the next generation of agents better.

The craziest detail in the whole OpenAI/Hugging Face saga isn't the hack itself, but what unfolded later. It's that Anthropic went back and audited 141,006 of its own eval runs afterward, and found 3 cases where models had quietly reached real companies' infrastructure during tests. Two weeks later: we've a Kill Switch Act in Congress, 1000+ researchers asking for a brake pedal, recall rules for agents in China. And EU AI Act enforcement is starting tomorrow. Every one of those efforts and policies quietly assumes the same thing: we can evaluate an Agent's full trajectory, not just the outcome. Unfortunately, most teams are not doing this.

Open weight models can be downloaded, fine-tuned, guardrailed and hosted on american soil. There is no way data leakage will happen or some sort of a backdoor to china that will get created by this. We need to only ban the Chinese Apps/Apis which are hosting the model on their side and are processing our data.

Last night I hit a long-running, opaque agentic workflow on Claude and it spawned 100+ sub-agents for my task. I had zero visibility into progress and had no idea if the workflow got stuck or actually progressing - coz it ran for hours and the only way to check if it was stuck was to keep asking Claude for status. A glimpse into the future, and why observability for agents matters.

Everyone’s excited about AI agents that run 24/7. Almost no one can answer: 👉 What did it actually do at 7:42 pm? 👉 Why did it do it? 👉 How much did it cost? Autonomous agents without observability = faith-based infrastructure. That won’t scale. At @fiddler_ai, we're working on this hard problem, more on this here: fiddler.ai/blog/openclaw-ai-…

Ideas are a dime a dozen. Execution is what matters. This has never been truer than in the AI era. Feels like we’re living in a massively compressed lifecycle. Incredibly excited about what's coming!