Ojasee Bapat

🔮Demystifying Coordination in #LLMs: We are excited to announce LLM-Coordination, a new benchmark for the evaluation and analysis of #LLMs in Multi-agent coordination tasks. Our benchmark studies two task settings: 1. 🤖 Agentic Coordination, where LLMs act as proactive participants for cooperation in 4 pure coordination games; 2. ❓Coordination Question Answering (CoordQA), where LLMs are prompted to answer questions from coordination games for evaluation of three key reasoning abilities: Environment Comprehension, #ToM Reasoning, and Joint Planning. Interestingly, we find that LLM agents excel at coordination games where the primary challenge is common-sense reasoning about the environment 🌎 and following the rules of the game. However, they struggle at games requiring an advanced Theory Of Mind🤔 #ToM, which is the ability to reason about the beliefs and intentions of their partners! Keep Reading 🧵 for the details and discoveries! Website: eric-ai-lab.github.io/llm_co… 📜paper: arxiv.org/abs/2310.03903 🔗data & code: github.com/eric-ai-lab/llm_c…