Founder & CEO @ Plumloom | Building LLM eval & reliability infrastructure | https://xitter.cf/t.co/fOF7ThgdPq | OSS Autoeval
San Francisco
Joined December 2008
- Tweets3.6K
- Following157
- Followers172
- Likes7
Pinned Tweet
“But our evals passed” is a painful explanation when an AI failure reaches customers.
A passing score can hide three different problems: a judge that doesn't reflect your team's standards, inconsistent judge assessments, or application behavior that varies across trials.
Plumloom Autoeval addresses these separately, helping teams build more reliable evaluation evidence and make better release decisions.
We're offering free Autoeval pilots for teams building AI applications and agents.
Bring a workflow you're already testing. We'll help you set up evaluations, review the results, and configure release gates. Hands-on onboarding, evaluation support, and AI model usage are included.
Interested? Sign up using this link:
plumloom.ai/free-pilot/
Your AI agent could be losing 16% in benchmark performance because of its search provider.
Exa 3.0 shows how swapping search providers changes results, even with the same model and harness.
Before upgrading your AI model, I’d check what the agent is actually retrieving.
Anthropic is taking on OpenAI Dots with Claude Duet. The pitch hits home: what did I forget today?
Duet monitors work apps and flags things you missed. One tester says it caught a property showing scheduled for the wrong day.
The real test? Knowing when to interrupt you.
We've just launched OpenAI Evals import in Plumloom.
If you've already run evals on OpenAI, you can now import your evals, runs, and results using your own API key.
We also analyze pass-rate trends and potentially flaky graders.
Try it at app.plumloom.ai
1/3 Anthropic tripled its workforce in a year, yet junior roles fell from 27% to 9% of technical hires.
As Claude takes on more coding and debugging, hiring managers are placing greater emphasis on autonomy and judgment.
2/3 Another interesting finding: as agents get faster, human attention becomes a constraint.
Employees spend more time supervising agents, reviewing outputs, and building tools to manage their work.
3/3 Engineers develop judgment through years of coding, debugging, and learning from mistakes. Much of that early work is moving to agents.
How do we develop experienced engineers if fewer people get the opportunity to start as junior engineers?
anthropic.com/research/emplo…
MiniMax took an unusual approach with M3.1 Flash.
An early version ran anonymously as Space Bunny, processing over 200 trillion tokens before its identity was revealed.
The official release now claims nearly 3x faster inference and 70% lower costs.
minimax.io/blog/minimax-m31f
Your coding agent can keep improving your AI app. But how do you know those improvements are reliable?
Claude Code, Codex, and Cursor can drive the iteration. We built Autoeval to measure reliability across trials and judges, and gate releases based on that evidence.
Open source. Apache-2.0.
github.com/AlchemyInCode/plu…
We've added Claude Opus 5.5 and Sonnet 5.5 to Plumloom, both supporting a 1M-token context window.
You can now evaluate these models with Autoeval, compare results across multiple trials, and run release gates directly from your terminal or agent harness.
Give it a try:
github.com/AlchemyInCode/plu…
A little creative experiment at Plumloom.
While designing our free pilot page, we explored retro pixel art inspired by early computing, with chunky pixels, limited colors, and Bayer dithering.
We liked it enough to make it part of Autoeval’s visual identity.
Sharing a few examples. Curious what you think!
SF Tech Week has more than 1,600 events, and LA Tech Week has around 800. Finding a handful that match your interests and schedule can take hours.
We built Tech Week Picks in Lovable, partly to explore what we could build with the platform. The app asks five questions about your role, goals, and interests, then recommends a personalized lineup without scheduling conflicts.
We chose TypeSafe's Jev because we needed fast, structured judgments. Jev classifies events by topic, audience, and format, then helps rerank the strongest matches. The scheduling and ranking rules stay in code.
The next challenge was tuning the recommendations for quality.
We installed Plumloom Autoeval directly in the Lovable development environment with one command:
npm i -g @plumloom/cli
From there, the Lovable coding agent could run evaluations against the actual app outputs, across multiple visitor profiles and trials, and examine relevance and factuality scores.
Autoeval uncovered unsupported explanations, weak recommendations toward the bottom of lineups, and noise in the event data that was producing false matches. The coding agent used those findings to refine the recommendation logic, and we reran the evaluations to check the changes.
This evaluation and tuning took a couple of hours. Manually testing and refining recommendations across those visitor profiles would have taken several days.
The early response has been encouraging. For SF Tech Week, 79% of visitors who started the quiz completed it.
What stood out from this experiment was how quickly we could move from building the app to evaluating and improving its recommendations, using Autoeval directly in the Lovable workflow.
Try Autoeval (open source):
github.com/AlchemyInCode/plu…
Link to the SF tech week: techweek.plumloom.ai/sf
And LA tech week:
techweek.plumloom.ai/ls
Here's the link to LA tech week:
techweek.plumloom.ai/la
And SF Tech week:
techweek.plumloom.ai/sf
A passing score doesn't tell the whole story. The model can behave differently across trials, and AI judges can disagree.
Here's how Plumloom Autoeval measures eval reliability and uses that evidence to make release decisions.
LA is next.
After seeing a strong signal from the SF Tech Week site,
I’m expanding it to LA Tech Week.
800+ events across the week. Answer a few questions and get a personalized lineup based on what you care about.
Built and evaluated with open source Plumloom Autoeval.
techweek.plumloom.ai/la
ALT Your LA Tech Week Lineup
1,600+ SF Tech Week events is a lot.
Built a small app to help people find the ones that actually fit their interests.
Under the hood: Jev for event classification/reranking, Plumloom Autoeval with an OpenAI GPT-5.6 Terra judge, all built in Lovable.
Private by design. No user info collected.
Here’s mine.
Try yours: techweek.plumloom.ai
A code change can leave an AI agent running while quietly breaking behavior that worked before.
On Sep 30, I’m hosting a webinar showing an eval-first CI workflow for catching those regressions before they ship, using Autoeval with Claude Code, Codex, or DeepSeek Harness.
We’ll cover how to turn agent failures into evals, separate real regressions from eval noise, and use the results as a CI gate.
👉 Register free: plumloom.ai/webinar/catch-ai…
Been looking more closely at Jev. Its scope is quite specific: fast, bounded decisions inside software like classify, route, score, extract or branch.
That works well for things like support routing or workflow decisions. Writing code, talking to customers, research and open-ended agent work are a different class of problem.
For higher-risk AI agents, production shouldn’t be where foreseeable failures are discovered.
Offline evals move known risk earlier. Production reveals what the eval suite missed.
And because both models and judges vary, one eval score is still only one observation.
I wrote out the fuller reasoning here, including where offline evaluation fits and why a single eval result can still be misleading:
lnkd.in/p/gmMZ9x8S