@bryanlics

Research Scientist @google, building efficient multi-{lingual,modal,task} information retrieval models for search | PhD from @penn | BA from @columbia

 New York
manestay.github.io
Joined May 2020
  • Tweets46
  • Following231
  • Followers175
21Media
In a globalized world, every event has ripple effects. When LLMs turn to the open web, how do they reflect state media? We release the WorldStateMedia dataset: 118M articles from 136 countries in 114 languages, and study their prevalence, narratives, and impact on LLMs. 1/6🧵
4
11
1
30
1,759
more replies
What happens when LLMs retrieve from state media on territorial disputes? RAG over state media sources decreases factuality vs. RAG over Wikipedia. And for multilingual queries, RAG over English sources increases divergent responses vs. over in-language sources. 5/6🧵
1
1
86
So glad to get this final project out the door, a year post-PhD graduation! Huge thanks to co-authors @samarhdr @_grace_kim @fionalluo @peterljq @lmgAboveFog @duncanjwatts @vidal_rene and Chris. 📜arxiv.org/abs/2610.24990 🤗huggingface.co/collections/m… 6/6🧵

The World as Viewed by 136 Countries: An Open, Curated, Annotated,...

State media outlets often project their governments' narratives to target both domestic and foreign audiences. Because this content is plentiful and free to access, it appears widely in the web...

arxiv.org
1
4
108
Given a request, AI systems generate long-form reports. Report evaluation relies on nuggets: essential & granular facts. Prior work on nugget generation treats them as short statements. Our #ICTIR2026 oral proposes DoGMaTiQ, an automated pipeline for Q&A nuggets. 1/6🧵
1
4
9
881
more replies
DoGMaTIQ scores for reports also best correlate with manual human scores, at system-level and at topic-level. More experiments are in our paper: circularity of LLMs for report generation + evaluation, as well as robustness to underperforming systems. 5/6🧵
1
31
Thanks to collaborators @pybeebee @houyu0930 @lawrie_dawn @EYangTW @lauradietz99 + more! At SIGIR #ICTIR2026 in Melbourne 🇦🇺 @EYangTW will present on July 24 at 14:00, or catch him at the poster session! 📜arxiv.org/abs/2605.04458 📊github.com/manestay/dogmatiq 6/6🧵

DoGMaTiQ: Automated Generation of Question-and-Answer Nuggets for...

Evaluation of long-form, citation-backed reports has lately received significant attention due to the wide-scale adoption of retrieval-augmented generation (RAG) systems. Core to many evaluation...

arxiv.org
2
37
In a world of geopolitical conflicts, how can AI help us navigate? Our #ACL2025-F work studies RAG robustness across 49 languages. TL;DR: 📈 boost robustness w/ multilingual RAG, 🤔 take care w/ low-resource citations 📜arxiv.org/abs/2410.01171 🤗huggingface.co/datasets/bord… 1/4 🧵
2
3
1
12
1,202
more replies
We study cross-lingual robustness over 4 LLMs and 2 IR models. We find A) multilingual RAG performs best; B) LLM’s citations varies widely across langs. Our further experiments investigate aspects of cross-lingual RAG from IR to LLM explanations. 3/4 🧵
1
118
This is the final paper of my PhD! Thanks to my many @upennnlp collaborators: @samarhdr, Chris, and the 7 wonderful students who I was fortunate to mentor. Please look out for our poster at ACL 2025 in Vienna. 4/4 🧵
3
122
🚀 How well can LLMs know you and personalize your response? Turns out, not so much! Introducing the PersonaMem Benchmark -- 👩🏻‍💻Evaluate LLM's ability to understand evolving persona from 180+ multi-session user-chatbot conversation history 🎯Latest models (GPT-4.1, GPT-4.5, o4-mini, Llama-4, Gemini 2.0, Deepseek-R1, Claude-3.7) all struggle in personalization! 🎨7 personalization skills tested in 15 scenarios 🌟Realistic long-context evaluation up to 1M tokens 👇 Check out what we discovered… (1/6)
Fig 1: Overview of PersonaMem benchmark. Each benchmark sample is a user persona with static (e.g., demographic info.) and dynamic attributes (e.g., evolving preferences). Users engage with a chatbot in multi-session interactions across a variety of topics such as food recommendation, travel planning, and therapy consultation. As the user’s preferences evolve over time, the benchmark offers annotated questions assessing whether models can track and incorporate the changes into their responses. Fig 2: Model performances by number of sessions elapsed since most recent preferences were mentioned in long context. Top: up to 20 sessions/128k tokens; Bottom: up to 60 sessions/1M tokens. Long-context retrieval is important for personalization in practice.

ALT Fig 1: Overview of PersonaMem benchmark. Each benchmark sample is a user persona with static (e.g., demographic info.) and dynamic attributes (e.g., evolving preferences). Users engage with a chatbot in multi-session interactions across a variety of topics such as food recommendation, travel planning, and therapy consultation. As the user’s preferences evolve over time, the benchmark offers annotated questions assessing whether models can track and incorporate the changes into their responses. Fig 2: Model performances by number of sessions elapsed since most recent preferences were mentioned in long context. Top: up to 20 sessions/128k tokens; Bottom: up to 60 sessions/1M tokens. Long-context retrieval is important for personalization in practice.

2
12
4
34
4,855
Externally retrieving knowledge empowers LLMs for domain-adapted MT ⚖️🩺. But how is knowledge best represented, and how viable is generating it from an LLM itself? Our @GoogleAI paper investigates these questions through a careful experimental setup 📜. arxiv.org/abs/2503.05010
1
3
6
464
TL;DR - translation pairs > bilingual terminologies, generation especially boosts translations for small LLMs Our ablations highlight the need for more challenging domain-adapted MT datasets with modern LLMs. Thanks to collaborators Jiaming, @ebriakou & @ColinCherry!
94
We share Code-Guided Synthetic Data Generation: using LLM-generated code to create multimodal datasets for text-rich images, such as charts📊, documents📄, etc., to enhance Vision-Language Models. Website: yueyang1996.github.io/cosyn/ Dataset: huggingface.co/datasets/alle… Paper: arxiv.org/pdf/2502.14846 Code: github.com/allenai/pixmo-doc…
Average performance on 7 text-rich benchmarks: ChartQA, DocVQA, InfoVQA, TableVQA, AI2D, TextVQA, ScreenQA.

ALT Average performance on 7 text-rich benchmarks: ChartQA, DocVQA, InfoVQA, TableVQA, AI2D, TextVQA, ScreenQA.

6
46
4
190
23,211
Load more