CS PhD @UWaterloo | Co-organizer of TREC RAG 2026 | Interests: NLP/IR Previously @ Doordash
Toronto, Ontario
Joined February 2026
- Tweets39
- Following57
- Followers27
- Likes65
Lingwei Gu retweeted
Recently revamped the FreshStack website and leaderboard! ๐
Added newer models, including recent releases: Cohere Embed v5, Google's EmbeddingGemma2 & Perplexity late-interaction models.
Use the benchmark to pick robust retrieval models and findings! ๐
Lingwei Gu retweeted
tl;dr - the team @Sahel_Sharify @Lingwei_Gu @nour_jedidi speedran nanochat by @karpathy using ClimbMix by @shizhediao et al. and then SFT'ed on relevance data from @beirmug @crystina_z to arrive at a competitive pointwise reranker: arxiv.org/abs/2610.11922
Lingwei Gu retweeted
What are we up to? ๐ก Project Greenhouse: Progress Toward Fully Open and Sovereign Agentic Search w/ @Sahel_Sharify @Lingwei_Gu @nour_jedidi arxiv.org/abs/2610.11922
Lingwei Gu retweeted
Excited to share that our paper was accepted to EMNLP Findings ๐
We ask a simple question: Does reasoning improve pointwise rerankers?
The answer? It can actually hurt... even for Rank1, a reranker trained to reason-then-answer ๐ฑ
๐ nourj98.github.io/papers/donโฆ
๐จDo passage rerankers really need explicit reasoning?๐คโMaybe Not!
Our findings:
โ๏ธStandard rerankers outperform those w/ step-by-step reasoning!
๐ซDisable reasoning from reasoning reranker actually improves reranking accuracy!๐คฏ
๐But, why?
๐ฐarxiv.org/abs/2505.16886
(1/6)
Lingwei Gu retweeted
To make an LLM a strong dense retriever, you often need trainingโฆ
But what if you only have a few-examples ๐ฅฉ๐ฅฆ and no training budget?
Just add RICE ๐, a simple way to turn your LLM into an effective dense retriever that requires zero training ๐จโ๐ณ
๐ arxiv.org/abs/2609.38099
Lingwei Gu retweeted
1/ ๐งต
BrowseComp-Plus (ACL 2026) set the standard for disentangled agentic search eval...
๐ ๐๐ซ๐จ๐ฐ๐ฌ๐๐๐จ๐ฆ๐ฉ-๐๐ฅ๐ฎ๐ฌ_๐๐ makes it even better!
๐ Same BCP questions grounded on ๐๐ฅ๐ข๐ฆ๐๐๐ข๐ฑ-๐๐๐๐ with ๐๐๐๐ ๐๐จ๐๐ฌ.
๐ป github.com/castorini/cmass
๐ Introducing BrowseComp-Plus: A More Fair and Transparent Evaluation Benchmark of Deep-Research Agent.
It is a new Deep-Research evaluation benchmark built on top of BrowseComp. It features
- ๐ a fixed, carefully curated corpus of web documents
- โ
human-verified positive documents
- โ๏ธ web-mined challenging hard negatives.
With BrowseComp-Plus, you can thoroughly evaluate and compare the performance of different components in a deep-research system. e.g. GPT-5 + Qwen3-Embedding.
Code, dataset, and leaderboard links are provided at the end of this thread.
Lingwei Gu retweeted
And with that, all TREC RAG 2026 submissions are in ๐
We had an incredible turnout this year, with 30 teams submitting more than 80 RAG runs and 110 retrieval runs!
A huge thank you to everyone who participated! We canโt wait to start evaluating your submissions ๐
Lingwei Gu retweeted
It's now officially ๐จโ๐ DR @xueguang_ma - who yesterday successfully defended his Ph.D. ๐ Congrats!
Lingwei Gu retweeted
๐ Thrilled that our ACL 2026 paper โTo Lie or Not to Lie?โ was selected as an ACL 2026 SAC Highlight! ๐
The work asks a simple but urgent question: are LLMs equally safe around the world? ๐งต๐
@aclmeeting #ACL2026NLP #NLProc
Lingwei Gu retweeted
๐จOfficial test topics for TREC RAG 2026 have been released! ๐จ
๐ Find them here: github.com/TREC-RAG/trec-ragโฆ
Time to see what your agents can do ๐
Lingwei Gu retweeted
Nuggets, rubrics, side-by-side comparisons, citation supportโฆ all useful ways to evaluate RAG, but these pieces are generally scattered across different tools and workflows
Introducing RAGDoll ๐, an end-to-end toolkit for evaluating long-form RAG answers ๐งต
Lingwei Gu retweeted
Disentangling retrievalโs role in RAG requires tighter NLP โ IR alignment - thatโs why weโre using @nvidia's ClimbMix ๐
But we donโt know which questions LLMs trained on ClimbMix "know"...and which they donโt
Now we have NanoKnow projections for NQ + SQuAD on ClimbMix ๐ฅ
Lingwei Gu retweeted
Agents can use tools to gather information, remember context, and act on your behalf.
That makes them useful. It also makes them dangerous. Agents can leak information they shouldnโt.
Introducing PrivacyAlign! PrivacyAlign uses human-annotation-grounded training and evaluation to cut privacy leaks by up to half and make automated privacy evaluation for agents more reliable.
Project Page: privacyalign.github.io/
๐งต(1/10)
Lingwei Gu retweeted
We want to build a collection where we can disentangle what the LLM knows from what it learns from retrieval...but how can we know what an LLM knows? ๐ค
With open LLMs + open corpora built on LLM pretraining data - like ClimbMix - NanoKnow lets us disentangle LLM knowledge ๐
๐คHow do LLMs know what they know? ๐ง
Pretraining data has long been a black box โผ๏ธ
But with @karpathyโs nanochat, we can finally peek inside ๐
๐ Introducing NanoKnow โ a benchmark to disentangle parametric vs. external knowledge in LLMs.
Lingwei Gu retweeted
๐จ Development topics for RAG 2026 are now out ๐ฅณ
You can begin testing your systems before the official evaluation topics are released in early July
So, tell your agents it's time to start building and evaluating! ๐ฅ
๐ปgithub.com/TREC-RAG/trec-ragโฆ
Lingwei Gu retweeted
Search is becoming increasingly agentic: systems plan, search, synthesize, cite, and revise. But, how should we study and evaluate these systems? ๐ค
In TREC RAG 2026, we want to build a reusable collection for this new reality
Weโve aligned on 4 core directions ๐งต๐
Lingwei Gu retweeted
Itโs incredibly rewarding to see ClimbMix become part of TREC RAG 2026.
The question of โdid the model retrieve it, or did it already know it?โ sits at the heart of modern RAG evaluation. Better controlled datasets and benchmarks are an important step toward answering that question scientifically.
Excited to see where this goes ๐
Does retrieval help RAG or did the LLM already memorize the answer? ๐ค Too often, the overlap between RAG corpora and what LLMs โknowโ is unclear
Better RAG evaluation needs tighter alignment between NLP and IR
๐ That's why for RAG 2026 we are using @nvidia's ClimbMix corpus
Lingwei Gu retweeted
Search is no longer just a ranked list...LLM agents can now query, inspect, reformulate, and decide when to stop ๐ค
At TREC RAG 2026, weโre introducing new metrics for agentic search: evaluating not only final results, but the search process itself ๐
Stay tuned!
Lingwei Gu retweeted
๐คจ Is your agent confused about what to build because it says there arenโt any guidelines?
Now your agent has no more excuses - track guidelines for TREC RAG 2026 are out ๐ฅ
And yes, theyโre available via SKILLz ๐
Tell your agents to showcase your agentic search system!