@spatialthinker8

Founder & CEO @ Plumloom | Building LLM eval & reliability infrastructure | https://xitter.cf/t.co/fOF7ThgdPq | OSS Autoeval

 San Francisco
github.com/AlchemyInCode/plumloom-autoeval-oss
Joined December 2008
  • Tweets3.6K
  • Following157
  • Followers172
9Media
A passing score doesn't tell the whole story. The model can behave differently across trials, and AI judges can disagree. Here's how Plumloom Autoeval measures eval reliability and uses that evidence to make release decisions.

Your AI eval passed. Would it pass again?

Your coding agent runs the evals, the score comes back above your release bar, and you ship. But in a probabilistic system, one passing score offers little evidence about how dependable that result is…

Article · @spatialthinker8
7
11
13,569
Load more