
Research Scientist working on RL environments and evals @PatronusAI | ex-Research @USC_ISI
San Francisco, CA
Joined July 2020
- Tweets190
- Following85
- Followers281
- Likes558
Darshan Deshpande retweeted
ICLR reviews are really instructive. Now I'm thinking, what a shame I didn't run multiple seeds for every decision I've made in life to make sure the results were statistically convincing.
Prompting LLMs to behave like people can improve simulations, but how should we design this? What info is needed, and how much?
With @Kehang_Zhu @locross @bakkermichiel, we build โbehavioral profilesโ that help LLMs simulate human behavior under distribution shifts. 1/8 ๐งต
Darshan Deshpande retweeted
Today, we're open sourcing the first computer-use dataset for design.
We captured 200+ hours and 3400+ trajectories of real design work in Figma, recording step by step how designers execute complex, long-horizon tasks.
FigmaTrace dataset, paper, and blog below :)
@PatronusAI
Darshan Deshpande retweeted
๐ Folks at #NEURIPS2025, come check out & stop by the poster of our Memtrack env at the SEA workshop happening at Upper Level 23ABC, 3:50pm onwards.
Our env studies how well an agent dropped into a workplace can context engineer by composing tool calls to access intertwined slack, linear & git timelines in pursuit of answering a battery of related questions.
Full paper arxiv: arxiv.org/abs/2510.01353
๐จWe will be presenting Memtrack today at the SEA workshop from 3:50pm onwards at #NeurIPS2025
Memtrack is a SoTA eval env to study an agent's ability to memorize and retrieve facts using exploration over interleaved enterprise slack, linear and git threads in a multi-QA setting
Darshan Deshpande retweeted
Thank you, @BerkeleyRDI, for hosting the Agentic AI Summit and having us!
@getdarshan, one of our research scientists, who leads agent evaluation here at Patronus, presented at the summit!
Here are a few takeaways:
* Given context explosion and increasing domain depth and specificity, we are approaching a ๐ป๐ฒ๐ ๐ฎ๐ด๐ฒ ๐ผ๐ณ ๐ฐ๐ผ๐ป๐๐ฒ๐
๐๐๐ฎ๐น ๐ฏ๐ฒ๐ป๐ฐ๐ต๐บ๐ฎ๐ฟ๐ธ๐.
* As the AI we work with becomes exponentially better, nuances become more important, as does the ๐ฐ๐ฟ๐ฒ๐ฎ๐๐ถ๐ผ๐ป ๐ผ๐ณ ๐ด๐ฟ๐ผ๐๐ป๐ฑ๐ฒ๐ฑ ๐ณ๐ฒ๐ฒ๐ฑ๐ฏ๐ฎ๐ฐ๐ธ-๐ฑ๐ฟ๐ถ๐๐ฒ๐ป ๐ฒ๐ป๐๐ถ๐ฟ๐ผ๐ป๐บ๐ฒ๐ป๐๐.
* Agent evaluation and ๐ฒ๐
๐ฝ๐น๐ฎ๐ถ๐ป๐ฎ๐ฏ๐น๐ฒ ๐๐ go hand-in-hand. Explainable agents are optimal for understanding agent workflows, fixing errors, and improving trajectories.
* Our team has seen success in developing ๐ต๐ฎ๐ฟ๐ฑ, ๐ฑ๐ผ๐บ๐ฎ๐ถ๐ป-๐๐ฝ๐ฒ๐ฐ๐ถ๐ณ๐ถ๐ฐ, ๐ฎ๐ป๐ฑ ๐ป๐ผ๐๐ฒ๐น ๐ฏ๐ฒ๐ป๐ฐ๐ต๐บ๐ฎ๐ฟ๐ธ๐ to rigorously evaluate AI performance.
๐ ๐ฐ๐ถ ๐ค๐ข๐ฏ ๐ณ๐ฆ๐ข๐ฅ ๐ฎ๐ฐ๐ณ๐ฆ ๐ข๐ฃ๐ฐ๐ถ๐ต ๐๐ข๐ณ๐ด๐ฉ๐ข๐ฏโ๐ด ๐ณ๐ฆ๐ค๐ฆ๐ฏ๐ต ๐ธ๐ฐ๐ณ๐ฌ ๐ฉ๐ฆ๐ณ๐ฆ:
* ๐ง๐ฅ๐๐๐: ๐ ๐๐ฒ๐ป๐ฐ๐ต๐บ๐ฎ๐ฟ๐ธ ๐ณ๐ผ๐ฟ ๐๐ด๐ฒ๐ป๐๐ถ๐ฐ ๐๐๐ฎ๐น๐๐ฎ๐๐ถ๐ผ๐ป
patronus.ai/blog/introducingโฆ
* ๐๐๐จ๐ฅ: ๐ ๐๐ฒ๐ป๐ฐ๐ต๐บ๐ฎ๐ฟ๐ธ ๐ณ๐ผ๐ฟ ๐ง๐ถ๐ฝ-๐ผ๐ณ-๐๐ต๐ฒ-๐ง๐ผ๐ป๐ด๐๐ฒ ๐ฆ๐ฒ๐ฎ๐ฟ๐ฐ๐ต ๐ฎ๐ป๐ฑ ๐ฅ๐ฒ๐ฎ๐๐ผ๐ป๐ถ๐ป๐ด
patronus.ai/blog/the-blur-beโฆ
* ๐๐๐๐๐๐ฅ: ๐ฆ๐ผ๐ง๐ ๐ฆ๐๐ ๐๐๐ฑ๐ด๐ฒ
patronus.ai/blog/glider-statโฆ
Reach out if youโre interested in chatting more about agent evals and how we can collaborate!
#BerkeleyRDI #AgenticAISummit
Darshan Deshpande retweeted
Check out the very cool work from our friends @PatronusAI ๐ฅ work here!
huggingface.co/spaces/Patronโฆ
Darshan Deshpande retweeted
We're excited to introduce the BLUR Leaderboard on @huggingface ๐ฅ
Earlier today, we open sourced BLUR: the first agent benchmark for tip-of-the-tongue search and reasoning. It measures how effectively agents can help you identify something you vaguely remember, but canโt quite name.
Check out the leaderboard on @huggingface to see how SOTA systems perform on BLUR!
Most systems score below 50% ๐ฒ
huggingface.co/spaces/Patronโฆ
Darshan Deshpande retweeted
1/ Ever tried to remember the name of a movie youโve seen โ you can picture the scenes clearly, but the movie name wonโt come to you?
Introducing BLUR: the first agent benchmark for tip-of-the-tongue search and reasoning ๐ฅ
We benchmarked SOTA agents and found that the best-performing agent only scored 56% on BLUR, while humans scored nearly perfectly at 98%! ๐คฏ
- OpenAI Operator: 54%
- Perplexity Pro: 27%
- ChatGPT-4o: 49%
- DeepSeek-R1: 41%
ArXiv: arxiv.org/pdf/2503.19193
HuggingFace data sample: huggingface.co/datasets/Patrโฆ
Patronus dataset: app.patronus.ai/datasets
Learn about our approach below ๐
Darshan Deshpande retweeted
1/ Introducing Lynx v2.0: an 8B State-of-the-Art RAG hallucination detection model ๐
- Beats Claude-3.5-Sonnet on HaluBench by 2.2%
- 3.4% higher accuracy than Lynx v1.1 on HaluBench
- Optimized for long context use cases
- Detects 8 types of common hallucinations, including Coreference Errors, Calculation Errors, CoT hallucinations
Try the Patronus API: app.patronus.ai
Read the docs: docs.patronus.ai/docs/hallucโฆ
Read the Lynx arXiv paper: arxiv.org/abs/2407.08488
Read the Lynx blog: patronus.ai/blog/lynx-state-โฆ
Darshan Deshpande retweeted
Prototype-based networks can greatly enhance the robustness of #languagemodels in text classification, addressing real-world needs by combining robustness & interpretability for #trustworthyAI. Learn how in our Findings of #EMNLP24 accepted paper. neclab.eu/research-groups/huโฆ #NECLabs

ALT Prototype-based networks can greatly enhance the robustness of #languagemodels in text classification, addressing real-world needs by combining robustness & interpretability for #trustworthyAI. Learn how in our Findings of #EMNLP24 accepted paper. https://neclab.eu/research-groups/human-centric-ai/publications#publications-2001 #NECLabs
Darshan Deshpande retweeted
Llama Guard is Off Duty ๐ฒย Itโs weak at toxicity detection!
We benchmarked popular toxicity datasets spanning languages like Portuguese, Ukrainian, and Turkish, and found that Llama Guard has a very high false negative rate for toxic content! We found that base models like Llama 3.1 do all the heavy lifting on toxicity filtering, and that the joint usage of Llama Guard might be redundant. ๐ค
Itโs time for a thread ๐งต
Darshan Deshpande retweeted
1/ Introducing Lynx v1.1: an 8B State-of-the-Art RAG hallucination detection model ๐
- Beats Claude-3.5-Sonnet on HaluBench by 3.0%
- Outperforms GPT-4o on medical questions and answers by 6.8%
- 1.4% higher accuracy than Lynx v1.0 on HaluBench
Try it out on HuggingFace Spaces: huggingface.co/spaces/Patronโฆ
Download Lynx v1.1 on HuggingFace: huggingface.co/PatronusAI/Llโฆ
huggingface.co/PatronusAI/Llโฆ
Read the ArXiv paper: arxiv.org/abs/2407.08488
Darshan Deshpande retweeted
Hello Tensors,
We have been waiting for this day for quite a while now....
Our first offline in person meetup is scheduled at 4th June 2022.
Stay tune for more updates๐
#TensorFlow #tfugMumbai โ at Mumbai, India
Darshan Deshpande retweeted
Weโve just open-sourced AugLy, a new #Python library that will help AI researchers use data augmentations to evaluate and improve the robustness of their machine learning models. Read more: ai.facebook.com/blog/augly-aโฆ
Darshan Deshpande retweeted
The first part of the Hugging Face Course is finally out!
Come learn how the ๐ค Ecosystem works ๐ฅณ: Transformers, Tokenizers, Datasets, Accelerate, the Model Hub!
Share with your friends who want to learn NLP, it's free!
Come join us at hf.co/course
Darshan Deshpande retweeted
BERT works for proteins! We tried it on one of the most important protein tasks, predicting signal peptides, and it beats the previous state-of-the-art. We present SignalP 6.0 biorxiv.org/content/10.1101/โฆ , thread below 1/5
Darshan Deshpande retweeted
Today's featured report is a fascinating look (with copious code and math!) at imaginary variables by @getdarshan wandb.ai/darshandeshpande/coโฆ
Darshan Deshpande retweeted
In today's featured report, @getdarshan explains how to use @huggingface to train a DistilBERT model for Devanagari languages like Marathi.
๐: wandb.ai/darshandeshpande/maโฆ























