Founding Engineer at @OvermindLab Ex-@LightningAI, @premai_io, @shopadvisor_ai, @Mazaal_AI, @EpisourceLLC
Delhi / BLR
Joined October 2021
- Tweets681
- Following544
- Followers282
- Likes1.4K
kimi k3.1 pro is a 2.8T param MoE, scored 55 on @ArtificialAnlys's index, and @Kimi_Moonshot says full weights land by nov 4. good news for open models. but you're not serving 2.8T params for one step in your agent loop. the boring move is a ~9b trained on your own traces. our @Alibaba_Qwen 9b fine-tune on BioRED biomedical extraction: 54.9 F1 vs 14.1 for the frontier flagship, 6.6% hallucinated vs 48.2%.
You SHOULD start learning/using two things:
- Hermes agent
- local AI
- ̶B̶u̶i̶l̶d̶i̶n̶g̶ ̶y̶o̶u̶r̶ ̶o̶w̶n̶ ̶m̶o̶d̶e̶l̶s̶ (or maybe not.... @OvermindLab takes care of that)
That is the future.
the cake thing is not really a joke. that's what it feels like when the agent has one job and you still call a frontier model to do it.
langChain will show you the bad run. langfuse will chart it. very helpful. neither of them trains on it.
a small model that has actually seen your traces will beat the genius you rent, then you get to keep the weights. wild concept.
repo is public. go poke at it!
Introducing Overmind. The training platform for agents.
Overmind trained models across biomedical, legal, and engineering tasks beating the frontier and achieving:
• 7x better accuracy
• 20x cheaper usage
• 28x less hallucinations
Train your own: github.com/overmind-core/ove…
15x cost difference for the same SQLite rebuild.
Most of the tokens in a long agent run don’t need the absolute best frontier model.
Once a strong planner has collapsed the ambiguity, cheaper or more specialized models can execute the volume without losing quality.
That’s the practical benefit of custom models for specific use cases.
We’re building toward the same idea at @OvermindLab. Agents that get sharper from real production data instead of just renting the most expensive general intelligence for every step.
Spent some time with the @LangChain State of Agent Engineering report over the weekend.
The numbers that stuck with me:
57% of organisations now have agents in production.
Quality is the #1 blocker (32%). Cost has basically fallen off the list.
89% of teams have some form of observability.
Only 52% run real offline evals.
That 37-point gap between “we can see the agent” and “we know if it’s actually getting better” is where most of the pain lives.
We’ve been building @OvermindLab around exactly this gap; turning the traces people already have into policy-aware evaluation and regression-safe optimisation. But the bigger point stands whether you use us or not.
If you’re sitting on production agent traces and still debugging the same classes of failure every week, the missing piece is probably not more observability.
It’s closing the loop.
Curious what others are seeing.
If you’ve read the report (or just lived the same problems), what’s the biggest gap you’re still hitting between “we can watch the agent” and “we can trust and improve it”?
Report here if you haven’t looked yet:
langchain.com/state-of-agent…
Give an agent a goal.
Let it build.
Run evals.
Inspect the failures.
Repeat until the result passes.
Around the same time, OpenAI published some interesting Codex usage data:
- Active users grew more than 5× in H1 2026
- 10%+ of users manage 3 or more concurrent agents
- Delegation of 8+ hour tasks grew almost 10×
ref: arxiv.org/abs/2606.26959
We are moving from prompting agents to supervising long-running work.
That makes the loop around the model as important as the model itself.
Did the agent improve between attempts?
Is it solving the task or gaming the evaluator?
Why did a run fail?
When should it retry or stop?
What did the successful outcome cost?
A loop gives the agent more time to work.
It also gives small mistakes more time to compound.
“Loop engineering” is a hot buzzphrase after mentions of it by Boris Cherny (Claude Code’s creator) and Peter Steinberger (OpenClaw's creator) went viral on social media. Loops are now a key part of how we get AI agents to iterate at length to build software. In this letter, I’d like to share my 3 key loops, shown in the image below, for building 0-to-1 products. These loops guide not just how I build software, but also how I decide what software to build.
Agentic coding loop: Given a product specification and optionally a set of evals (that is, a dataset against which to measure performance), we can have an AI agent write code, test its work, and keep iterating until the code is bug-free and meets its specification. This idea of closing the loop took off around the end of last year, and it has been a game changer in enabling coding agents to work longer productively without human intervention. For example, over the weekend, I was building an app for my daughter to practice typing, and my coding agent could easily work for around an hour, using a web browser to check what it had built multiple times before getting back to me, without needing my intervention.
The engineering loop executes quickly. Every few minutes, the coding agent might build and test a new version of the software. I hear frequently from developers who are finding new ways to engineer more effective engineering loops. This is an active area of invention!
Developer feedback loop: In this loop, a developer examines the current product and steers the coding agent to improve it. Last year, a lot of developers (including me) were acting as the QA (quality assurance) function for our coding agents, manually finding bugs and then asking the agent to fix them. But with coding agents much more able to test their own code, the amount of time we need to spend on this function has decreased significantly. This allows us to make higher-level product decisions, such as what key features to offer, where the UI needs improvement, and so on.
The developer-feedback loop operates over time intervals between tens of minutes and hours — that's how frequently a developer might review a product and give feedback. In the case of the typing app, I changed my mind a few times about the visual design, what cat costumes she can unlock as she learns (she loves cats), and the user flow for a grown-up to log in and steer the child's learning experience.
When a developer has a clear vision for what to build, it is still a lot of work to translate that vision into a specification for a coding agent to implement. Further, after the developer has seen an implementation, they might update (or perhaps clarify) the spec to steer it toward what they want. If you find that the system repeatedly runs into certain problems, building a set of evals for the agent becomes useful.
AI-native teams are increasingly using AI to help shape product direction, for example, automating the gathering and analysis of usage data, summarizing written and verbal customer feedback, or carrying out competitive analysis. However, for pretty much all the products I’m involved in, I see humans as having a significant context advantage over current AI systems — we know a lot more than the AI system about the users and the context the product has to operate in — and thus humans play a critical role. Many people describe this human contribution as “taste,” but I prefer to think of it as humans having a context advantage, since that gives us a clearer path to helping AI systems get better. This also speaks to why this step can’t be automated: So long as the human knows something the AI does not, human-in-the-loop is needed to to inject that knowledge into the system.
External feedback loop: This includes a wide range of tactics like asking a few friends for feedback, launching to alpha testers, or putting the code into production with A/B testing. These tactics are usually slow, rarely taking less than hours and sometimes taking days or even weeks. This data informs the developer vision, which in turn continues to drive the detailed product spec, which in turn drives the coding agent.
With coding agents speeding up software development, more engineers are starting to play a partial product management role. For many engineers who are growing into this role, the hardest part is shaping the product vision and striking a balance between building (bridging the gap between vision and spec) and getting user feedback to evolve the vision. It is important to do both!
I will write more about how to do this in future posts, but for now, I find it encouraging that engineers are playing an expanded role (just as product managers and designers now do more engineering).
[Original text: The Batch]
more thoughts on this...
The strongest part of this argument is not “frontier labs are bad” or “tokens are a scam.”
That is too simplistic.
The more interesting point is that enterprise AI pricing and enterprise AI value are starting to drift apart.
Vendors sell usage.
Customers want outcomes.
A company does not care that an agent consumed 40k tokens.
It cares whether the invoice was processed correctly.
Whether the support ticket was routed to the right team.
Whether the clinical note followed policy.
Whether the contract clause was extracted without leaking sensitive context.
Whether the workflow got cheaper, faster, or more reliable after deployment.
This is where a lot of AI products will struggle.
A demo can hide this.
Production cannot.
Once agents touch internal systems, the questions change:
Who owns the data?
Can the model learn from our prompts?
Can we move between models?
What happens when the agent is wrong?
Where is the trace?
Who approved the action?
What gets improved after every run?
That is why I think the enterprise AI stack will become less about “which model did you use?” and more about the infrastructure around the model.
The useful layer is the one that can watch agent behaviour, evaluate it against business rules, find failures, test improvements, and help teams move toward cheaper or more specialised models where possible.
Tokens are an input cost.
The buyer wants proof that the system created value.
A lot of enterprise AI frustration comes down to one thing:
teams are paying for tokens, but they actually want outcomes.
Not prettier demos.
Not bigger context windows.
Not another API bill.
They want to know:
Who owns the data?
Where do prompts get cached?
Can we switch models?
Did the agent actually do the job?
What improves after every run?
That is where AI adoption gets serious.
The enterprise buyer is not just buying intelligence.
They are buying control.
A lot of enterprise AI frustration comes down to one thing:
teams are paying for tokens, but they actually want outcomes.
Not prettier demos.
Not bigger context windows.
Not another API bill.
They want to know:
Who owns the data?
Where do prompts get cached?
Can we switch models?
Did the agent actually do the job?
What improves after every run?
That is where AI adoption gets serious.
The enterprise buyer is not just buying intelligence.
They are buying control.
Monitoring and evals are great for telling you why your agent broke, but they still leave you to manually fix it. We built @OvermindLab to actually close the loop and handle the recovery autonomously. Massive day for the team getting this out!
some people just hate “wrappers” and wants everyone to build everything from scratch.
yes, kimi k2.5 exists, but does it have cursor-level reach?
everyone screams open-source… until someone actually ships a real product with it. then suddenly it’s a problem.
composer 2 is cheap, works, and users don’t care if it’s a wrapper.
same crowd also forgets SLMs are trained on data from big closed LLMs anyway.
cursor just distilled that into something usable.
the hate makes no sense lol
I am hiring an AI research engineer for our @premai_io Studio team. If you have experience working on LLM fine-tuning, RL fine-tuning, DPO, and model alignment, as well as deployments, do apply.
Link in first comment...
I am hiring an AI research engineer for our @premai_io Studio team. If you are excited about LLM fine-tuning, RL fine-tuning, and deployments, please apply.
premai.notion.site/ai-resear…