@Born_TechK

Foundations now. LLM products soon. | ML • Python • Data stack | Building in public for SaaS founders

 India
Joined August 2024
  • Tweets444
  • Following68
  • Followers83
55Media
1/ A support bot can give a customer the wrong price and still sound completely confident. I wanted to see if RAG evaluation metrics could catch that, so I deliberately planted wrong facts in a bot’s answers and tested 28 Q&As. Here’s what happened🧵
2
1
22
more replies
5/ The whole 28-row eval cost ~$0.06. I’m turning it into a reusable template for RAG support bots. If you run one for your SaaS and aren’t sure what it’s telling customers, reply or DM me. I’ll run it on a sample for free. Code + findings: [GitHub link, check below]
1
7
github.com/codewith-krishh/r…

GitHub - codewith-krishh/rag-eval-audit

Contribute to codewith-krishh/rag-eval-audit development by creating an account on GitHub.

github.com
1
5
Day 5 of building a RAG evaluator: I planted 10 known failures in a test set to see if the judge would catch them. It caught 10/10 It also flagged 4 of 10 perfectly good answers. The false alarms were harmless things like "According to the help center..." and "contact support."
Krish@Born_TechKOct 22
1/ Day 4: "the chatbot seems fine" is how most RAG projects get shipped. I built a reusable evaluator so I can hand a founder a number instead. Scored 28 support answers. Total judge cost: $0.06. Follow the thread below so you know what to check before any build👇
1
15
The judge treated them as unsupported claims. I agreed with it on 6 of 9 rows I graded by hand. Same input scored twice: faithfulness moved on 3 of 10 rows, and one flag flipped. #buildinpublic
13
1/ Day 4: "the chatbot seems fine" is how most RAG projects get shipped. I built a reusable evaluator so I can hand a founder a number instead. Scored 28 support answers. Total judge cost: $0.06. Follow the thread below so you know what to check before any build👇
Krish@Born_TechKOct 15
An eval set with no known failures can't tell you if your evaluator works. Day 3: I built a 28-question test set for a RAG support bot, then broke it on purpose. → 5 rows with the wrong docs retrieved → 5 rows with a planted false claim → 3 questions the docs can't answer →
1
1
33
more replies
4/ Another trap: 25% of answers were "I don't have that info." They scored as hallucinations. Naive hallucination rate: 78.6%. Counting only answers that actually tried: 71.4%. "I don't know" isn't a hallucination, so I report three numbers: abstention, hallucination, retrieval
1
4
5/ Under the hood: rate-limit throttle, retries, one bad row never kills the run, per-row cost tracking, results saved as CSV + JSON. Next: auditing the judge itself. Who grades the grader?
3
SaaS founders: what’s the biggest blocker to adding AI to your product or support? I build AI chatbots and knowledge-base tools. Before I decide what to build next, I want to hear from the people who’d use them.
50%Cost
0%Accuracy / hallucinations
0%Data privacy
50%No time or no team
2 votes • Final results
2
2
50
If your blocker isn’t on the list, tell me here. I read every reply.
1
25
An eval set with no known failures can't tell you if your evaluator works. Day 3: I built a 28-question test set for a RAG support bot, then broke it on purpose. → 5 rows with the wrong docs retrieved → 5 rows with a planted false claim → 3 questions the docs can't answer →
Krish@Born_TechKOct 14
A RAG eval score of 1.0 doesn't mean your chatbot is right. Day 2 of testing RAGAS: I wrote 4 deliberately bad answers and predicted the scores before running. → Planted a false claim: faithfulness 0.75, relevancy 0.98 →👇
1
1
6
81
→ 5 mistakes the 3B model made on its own The bot got 24 of 25 retrievals right. The 1 miss, plus its confident wrong answer, was my favorite row. Next: see which metric catches which failure.
1
27
A RAG eval score of 1.0 doesn't mean your chatbot is right. Day 2 of testing RAGAS: I wrote 4 deliberately bad answers and predicted the scores before running. → Planted a false claim: faithfulness 0.75, relevancy 0.98 →👇
Krish@Born_TechKOct 9
Your RAG chatbot sounds confident. How do you know it isn’t making things up? I’m building an evaluator that checks every answer against its sources (RAGAS). Day 1: no OpenAI key, so judge = Groq, embeddings = local. 3 dependency fights later, it scores. Next: I try to fool it
3
1
2
97
→ True advice not in the docs: faithfulness 0.25, relevancy 1.0 → Off-topic answer: relevancy went negative (-0.125) Relevancy can't see lies. Faithfulness punishes true-but-ungrounded answers. 10/12 predictions held.
1
18
Your RAG chatbot sounds confident. How do you know it isn’t making things up? I’m building an evaluator that checks every answer against its sources (RAGAS). Day 1: no OpenAI key, so judge = Groq, embeddings = local. 3 dependency fights later, it scores. Next: I try to fool it
1
1
74
Load more