Official Twitter account for the TREC RAG Tracks (2024 & 2025 & 2026)!
Joined February 2024
- Tweets125
- Following35
- Followers378
- Likes115
Pinned Tweet
Does retrieval help RAG or did the LLM already memorize the answer? 🤔 Too often, the overlap between RAG corpora and what LLMs “know” is unclear
Better RAG evaluation needs tighter alignment between NLP and IR
📚 That's why for RAG 2026 we are using @nvidia's ClimbMix corpus
And with that, all TREC RAG 2026 submissions are in 🎉
We had an incredible turnout this year, with 30 teams submitting more than 80 RAG runs and 110 retrieval runs!
A huge thank you to everyone who participated! We can’t wait to start evaluating your submissions 🚀
TREC RAG @ 2026 retweeted
🪠 Belatedly plugging a paper that attempts to bridge information nuggets and side-by-side comparisons with @arena data for complex LLM responses (tl;dr - pointwise vs. pairwise evaluations) dl.acm.org/doi/10.1145/38057… Paper led by @Sahel_Sharify @Ushivani3 @beirmug!
🚀 The TREC RAG 2026 baselines for both Retrieval and RAG tracks are live!
Explore them here👇
github.com/TREC-RAG/trec-rag…
🚨Official test topics for TREC RAG 2026 have been released! 🚨
🔎 Find them here: github.com/TREC-RAG/trec-rag…
Time to see what your agents can do 👀
TREC RAG @ 2026 retweeted
Nuggets, rubrics, side-by-side comparisons, citation support… all useful ways to evaluate RAG, but these pieces are generally scattered across different tools and workflows
Introducing RAGDoll 🐈, an end-to-end toolkit for evaluating long-form RAG answers 🧵
Disentangling retrieval’s role in RAG requires tighter NLP ↔ IR alignment - that’s why we’re using @nvidia's ClimbMix 📚
But we don’t know which questions LLMs trained on ClimbMix "know"...and which they don’t
Now we have NanoKnow projections for NQ + SQuAD on ClimbMix 🔥
Start playing with the data at: github.com/castorini/NanoKno…!
NanoKnow paper (accepted at SIGIR 2026 🎉): arxiv.org/pdf/2602.20122
Interested in helping us build benchmarks to disentangle sources of LLM knowledge? Don't forget to:
1️⃣ Sign up for TREC: trec.nist.gov/cfp.html
2️⃣ Join our mailing list: groups.google.com/g/trec-rag…
3️⃣ Join us on SIGIR Slack!
We want to build a collection where we can disentangle what the LLM knows from what it learns from retrieval...but how can we know what an LLM knows? 🤔
With open LLMs + open corpora built on LLM pretraining data - like ClimbMix - NanoKnow lets us disentangle LLM knowledge 👇
🤔How do LLMs know what they know? 🧠
Pretraining data has long been a black box ◼️
But with @karpathy’s nanochat, we can finally peek inside 🔓
🔍 Introducing NanoKnow — a benchmark to disentangle parametric vs. external knowledge in LLMs.
🚨 Development topics for RAG 2026 are now out 🥳
You can begin testing your systems before the official evaluation topics are released in early July
So, tell your agents it's time to start building and evaluating! 🔥
💻github.com/TREC-RAG/trec-rag…
🔍 What about retrieval?
Worry not!
We also provide ☂️ relevance judgements for the TREC RAG 2025 topics over ClimbMix, making it easy to evaluate both retrieval and generation components of your pipeline 👌
Interested? Don’t forget to:
1️⃣ Sign up for TREC: trec.nist.gov/cfp.html
2️⃣ Join our mailing list: groups.google.com/g/trec-rag…
3️⃣ Join us on SIGIR Slack!
Search is becoming increasingly agentic: systems plan, search, synthesize, cite, and revise. But, how should we study and evaluate these systems? 🤔
In TREC RAG 2026, we want to build a reusable collection for this new reality
We’ve aligned on 4 core directions 🧵👇
Which research direction are you 🫵 most excited about?
What do you think are unanswered questions in current agentic search evaluations?
What do you want us to explore for TREC RAG 2026?
We welcome all feedback, and would love to hear from you!
Interested? Don’t forget to:
1️⃣ Sign up for TREC: trec.nist.gov/cfp.html
2️⃣ Join our mailing list: groups.google.com/g/trec-rag…
3️⃣ Join us on SIGIR Slack!