Dustin - We keep things moving

Today we are announcing our latest project, an effort to provide a new evaluation system for open source models. Traditional benchmarking leans heavily on public datasets which can be easy to game and often lead to superficial score improvements that mask true model capabilities (or lack thereof), misdirecting efforts towards meaningless score chasing. To achieve this, we built a subnet on Bittensor, a decentralized network for AI projects. This system allows you to submit your finetuned model, and then validators will evaluate all models submitted against freshly generated data from the Cortex subnet. The Cortex subnet is a dynamic source of fresh synthetic data, continuously generated by GPT-4, which allows for fresh and unpredictable high quality data to test models against, and ensures a fair and accurate consensus on which model most closely mirrors GPT-4's performance. This new system aims to celebrate and reward creators of public, open source models that genuinely meet user needs, through the incentive structures built into Bittensor. We plan to continually evolve and expand the diversity and quality of that continuous data stream over time, to keep the evaluations fresh, challenging, and more and more representative of user needs in every new iteration. Learn more on the following pages: Nous Subnet Leaderboard - huggingface.co/spaces/NousRe… Nous Subnet Repository - github.com/NousResearch/fine… Bittensor - bittensor.com/ Cortex Subnet Repository - github.com/corcel-api/cortex…