Can #LLMs replace developers? Introducing RepoCod-Lite 🐟 for faster evaluation to answer this: 200 of the toughest #RepoCod #code-generation tasks: - GPT-4o and other LLMs have < 10% accuracy/pass@1 on RepoCod-Lite tasks - Leaderboard lt-asset.github.io/REPOCOD/ - 67 repository-level, 67 file-level, and 66 self-contains tasks - Detailed problem descriptions (967 tokens) and long canonical solutions (918 tokens) - Dataset: huggingface.co/datasets/lt-a… Thanks to the great feedback from #swe-bench’s @OfirPress, here are some clarifications about #RepoCod (lt-asset.github.io/REPOCOD/): Compared to #SWE-Bench, RepoCod tasks are - General code generation tasks,  while SWE-Bench tasks resolve pull requests from GitHub issues - With 2.6X more tests per task (313.5 compared to SWE-Bench’s 120.8) Compared to #HumanEval, #MBPP, #CoderEval, and #ClassEval, RepoCod has 980 instances from 11 Python projects, with - Whole function generation - Repository-level context - Validation with test cases, and  - Real-world complex tasks: longest average canonical solution length (331.6 tokens) and the highest average cyclomatic complexity (9.00) #LLMs #LLM4Code #security #codegen