Darshan Deshpande (@getdarshan) profile picture

Research Scientist working on RL environments and evals @PatronusAI | ex-Research @USC_ISI

San Francisco, CA
Joined July 2020
Photo by Darshan Deshpande (@getdarshan)Photo by Darshan Deshpande (@getdarshan)Photo by Darshan Deshpande (@getdarshan)Photo by Darshan Deshpande (@getdarshan)Photo by Darshan Deshpande (@getdarshan)Photo by Darshan Deshpande (@getdarshan)Photo by Darshan Deshpande (@getdarshan)Photo by Darshan Deshpande (@getdarshan)Photo by Darshan Deshpande (@getdarshan)Photo by Darshan Deshpande (@getdarshan)
Darshan Deshpande retweeted
ICLR reviews are really instructive. Now I'm thinking, what a shame I didn't run multiple seeds for every decision I've made in life to make sure the results were statistically convincing.
22
56
4
974
56,879
Darshan Deshpande retweeted
Prompting LLMs to behave like people can improve simulations, but how should we design this? What info is needed, and how much? With @Kehang_Zhu @locross @bakkermichiel, we build โ€œbehavioral profilesโ€ that help LLMs simulate human behavior under distribution shifts. 1/8 ๐Ÿงต
7
24
2
147
13,966
Darshan Deshpande retweeted
Today, we're open sourcing the first computer-use dataset for design. We captured 200+ hours and 3400+ trajectories of real design work in Figma, recording step by step how designers execute complex, long-horizon tasks. FigmaTrace dataset, paper, and blog below :) @PatronusAI
41
73
19
733
138,034
Darshan Deshpande retweeted
๐Ÿ‘‹ Folks at #NEURIPS2025, come check out & stop by the poster of our Memtrack env at the SEA workshop happening at Upper Level 23ABC, 3:50pm onwards. Our env studies how well an agent dropped into a workplace can context engineer by composing tool calls to access intertwined slack, linear & git timelines in pursuit of answering a battery of related questions. Full paper arxiv: arxiv.org/abs/2510.01353
๐ŸšจWe will be presenting Memtrack today at the SEA workshop from 3:50pm onwards at #NeurIPS2025 Memtrack is a SoTA eval env to study an agent's ability to memorize and retrieve facts using exploration over interleaved enterprise slack, linear and git threads in a multi-QA setting
3
8
919
Darshan Deshpande retweeted
Thank you, @BerkeleyRDI, for hosting the Agentic AI Summit and having us! @getdarshan, one of our research scientists, who leads agent evaluation here at Patronus, presented at the summit! Here are a few takeaways: * Given context explosion and increasing domain depth and specificity, we are approaching a ๐—ป๐—ฒ๐˜„ ๐—ฎ๐—ด๐—ฒ ๐—ผ๐—ณ ๐—ฐ๐—ผ๐—ป๐˜๐—ฒ๐˜…๐˜๐˜‚๐—ฎ๐—น ๐—ฏ๐—ฒ๐—ป๐—ฐ๐—ต๐—บ๐—ฎ๐—ฟ๐—ธ๐˜€. * As the AI we work with becomes exponentially better, nuances become more important, as does the ๐—ฐ๐—ฟ๐—ฒ๐—ฎ๐˜๐—ถ๐—ผ๐—ป ๐—ผ๐—ณ ๐—ด๐—ฟ๐—ผ๐˜‚๐—ป๐—ฑ๐—ฒ๐—ฑ ๐—ณ๐—ฒ๐—ฒ๐—ฑ๐—ฏ๐—ฎ๐—ฐ๐—ธ-๐—ฑ๐—ฟ๐—ถ๐˜ƒ๐—ฒ๐—ป ๐—ฒ๐—ป๐˜ƒ๐—ถ๐—ฟ๐—ผ๐—ป๐—บ๐—ฒ๐—ป๐˜๐˜€. * Agent evaluation and ๐—ฒ๐˜…๐—ฝ๐—น๐—ฎ๐—ถ๐—ป๐—ฎ๐—ฏ๐—น๐—ฒ ๐—”๐—œ go hand-in-hand. Explainable agents are optimal for understanding agent workflows, fixing errors, and improving trajectories. * Our team has seen success in developing ๐—ต๐—ฎ๐—ฟ๐—ฑ, ๐—ฑ๐—ผ๐—บ๐—ฎ๐—ถ๐—ป-๐˜€๐—ฝ๐—ฒ๐—ฐ๐—ถ๐—ณ๐—ถ๐—ฐ, ๐—ฎ๐—ป๐—ฑ ๐—ป๐—ผ๐˜ƒ๐—ฒ๐—น ๐—ฏ๐—ฒ๐—ป๐—ฐ๐—ต๐—บ๐—ฎ๐—ฟ๐—ธ๐˜€ to rigorously evaluate AI performance. ๐˜ ๐˜ฐ๐˜ถ ๐˜ค๐˜ข๐˜ฏ ๐˜ณ๐˜ฆ๐˜ข๐˜ฅ ๐˜ฎ๐˜ฐ๐˜ณ๐˜ฆ ๐˜ข๐˜ฃ๐˜ฐ๐˜ถ๐˜ต ๐˜‹๐˜ข๐˜ณ๐˜ด๐˜ฉ๐˜ข๐˜ฏโ€™๐˜ด ๐˜ณ๐˜ฆ๐˜ค๐˜ฆ๐˜ฏ๐˜ต ๐˜ธ๐˜ฐ๐˜ณ๐˜ฌ ๐˜ฉ๐˜ฆ๐˜ณ๐˜ฆ: * ๐—ง๐—ฅ๐—”๐—œ๐—Ÿ: ๐—” ๐—•๐—ฒ๐—ป๐—ฐ๐—ต๐—บ๐—ฎ๐—ฟ๐—ธ ๐—ณ๐—ผ๐—ฟ ๐—”๐—ด๐—ฒ๐—ป๐˜๐—ถ๐—ฐ ๐—˜๐˜ƒ๐—ฎ๐—น๐˜‚๐—ฎ๐˜๐—ถ๐—ผ๐—ป patronus.ai/blog/introducingโ€ฆ * ๐—•๐—Ÿ๐—จ๐—ฅ: ๐—” ๐—•๐—ฒ๐—ป๐—ฐ๐—ต๐—บ๐—ฎ๐—ฟ๐—ธ ๐—ณ๐—ผ๐—ฟ ๐—ง๐—ถ๐—ฝ-๐—ผ๐—ณ-๐˜๐—ต๐—ฒ-๐—ง๐—ผ๐—ป๐—ด๐˜‚๐—ฒ ๐—ฆ๐—ฒ๐—ฎ๐—ฟ๐—ฐ๐—ต ๐—ฎ๐—ป๐—ฑ ๐—ฅ๐—ฒ๐—ฎ๐˜€๐—ผ๐—ป๐—ถ๐—ป๐—ด patronus.ai/blog/the-blur-beโ€ฆ * ๐—š๐—Ÿ๐—œ๐——๐—˜๐—ฅ: ๐—ฆ๐—ผ๐—ง๐—” ๐—ฆ๐—Ÿ๐—  ๐—๐˜‚๐—ฑ๐—ด๐—ฒ patronus.ai/blog/glider-statโ€ฆ Reach out if youโ€™re interested in chatting more about agent evals and how we can collaborate! #BerkeleyRDI #AgenticAISummit
1
2
333
Darshan Deshpande retweeted
We're excited to introduce the BLUR Leaderboard on @huggingface ๐Ÿ”ฅ Earlier today, we open sourced BLUR: the first agent benchmark for tip-of-the-tongue search and reasoning. It measures how effectively agents can help you identify something you vaguely remember, but canโ€™t quite name. Check out the leaderboard on @huggingface to see how SOTA systems perform on BLUR! Most systems score below 50% ๐Ÿ˜ฒ huggingface.co/spaces/Patronโ€ฆ
2
11
2
43
18,102
Darshan Deshpande retweeted
1/ Ever tried to remember the name of a movie youโ€™ve seen โ€“ you can picture the scenes clearly, but the movie name wonโ€™t come to you? Introducing BLUR: the first agent benchmark for tip-of-the-tongue search and reasoning ๐Ÿ”ฅ We benchmarked SOTA agents and found that the best-performing agent only scored 56% on BLUR, while humans scored nearly perfectly at 98%! ๐Ÿคฏ - OpenAI Operator: 54% - Perplexity Pro: 27% - ChatGPT-4o: 49% - DeepSeek-R1: 41% ArXiv: arxiv.org/pdf/2503.19193 HuggingFace data sample: huggingface.co/datasets/Patrโ€ฆ Patronus dataset: app.patronus.ai/datasets Learn about our approach below ๐Ÿ‘‡
1
6
3
46
11,152
Darshan Deshpande retweeted
1/ Introducing Lynx v2.0: an 8B State-of-the-Art RAG hallucination detection model ๐Ÿš€ - Beats Claude-3.5-Sonnet on HaluBench by 2.2% - 3.4% higher accuracy than Lynx v1.1 on HaluBench - Optimized for long context use cases - Detects 8 types of common hallucinations, including Coreference Errors, Calculation Errors, CoT hallucinations Try the Patronus API: app.patronus.ai Read the docs: docs.patronus.ai/docs/hallucโ€ฆ Read the Lynx arXiv paper: arxiv.org/abs/2407.08488 Read the Lynx blog: patronus.ai/blog/lynx-state-โ€ฆ
3
8
1
21
5,769
Prototype-based networks can greatly enhance the robustness of #languagemodels in text classification, addressing real-world needs by combining robustness & interpretability for #trustworthyAI. Learn how in our Findings of #EMNLP24 accepted paper. neclab.eu/research-groups/huโ€ฆ #NECLabs
4
7
327
Darshan Deshpande retweeted
Llama Guard is Off Duty ๐Ÿ˜ฒย Itโ€™s weak at toxicity detection! We benchmarked popular toxicity datasets spanning languages like Portuguese, Ukrainian, and Turkish, and found that Llama Guard has a very high false negative rate for toxic content! We found that base models like Llama 3.1 do all the heavy lifting on toxicity filtering, and that the joint usage of Llama Guard might be redundant. ๐Ÿค” Itโ€™s time for a thread ๐Ÿงต
1
2
2
18
2,736
Darshan Deshpande retweeted
1/ Introducing Lynx v1.1: an 8B State-of-the-Art RAG hallucination detection model ๐Ÿš€ - Beats Claude-3.5-Sonnet on HaluBench by 3.0% - Outperforms GPT-4o on medical questions and answers by 6.8% - 1.4% higher accuracy than Lynx v1.0 on HaluBench Try it out on HuggingFace Spaces: huggingface.co/spaces/Patronโ€ฆ Download Lynx v1.1 on HuggingFace: huggingface.co/PatronusAI/Llโ€ฆ huggingface.co/PatronusAI/Llโ€ฆ Read the ArXiv paper: arxiv.org/abs/2407.08488
4
24
3
127
27,318
Darshan Deshpande retweeted
Hello Tensors, We have been waiting for this day for quite a while now.... Our first offline in person meetup is scheduled at 4th June 2022. Stay tune for more updates๐Ÿ™‚ #TensorFlow #tfugMumbai โ€“ at Mumbai, India
1
12
2
37
Darshan Deshpande retweeted
Meet GitHub Copilot - your AI pair programmer. copilot.github.com
733
7,623
6,076
27,317
Darshan Deshpande retweeted
Weโ€™ve just open-sourced AugLy, a new #Python library that will help AI researchers use data augmentations to evaluate and improve the robustness of their machine learning models. Read more: ai.facebook.com/blog/augly-aโ€ฆ
15
279
18
928
Darshan Deshpande retweeted
The first part of the Hugging Face Course is finally out! Come learn how the ๐Ÿค— Ecosystem works ๐Ÿฅณ: Transformers, Tokenizers, Datasets, Accelerate, the Model Hub! Share with your friends who want to learn NLP, it's free! Come join us at hf.co/course
23
469
70
1,459
Darshan Deshpande retweeted
BERT works for proteins! We tried it on one of the most important protein tasks, predicting signal peptides, and it beats the previous state-of-the-art. We present SignalP 6.0 biorxiv.org/content/10.1101/โ€ฆ , thread below 1/5
1
31
3
136
Today's featured report is a fascinating look (with copious code and math!) at imaginary variables by @getdarshan wandb.ai/darshandeshpande/coโ€ฆ
1
1
In today's featured report, @getdarshan explains how to use @huggingface to train a DistilBERT model for Devanagari languages like Marathi. ๐Ÿ“: wandb.ai/darshandeshpande/maโ€ฆ
3
9