Zeno (@try_zeno) header
Zeno (@try_zeno) profile picture

AI evaluation platform

San Francisco, CA
Joined January 2023
Photo by Zeno (@try_zeno)Photo by Zeno (@try_zeno)Photo by Zeno (@try_zeno)Photo by Zeno (@try_zeno)
@a13xba will give a presentation about @try_zeno, an interactive AI evaluation platform for exploring, debugging, and sharing how your AI systems perform. (co-founded with @a_a_cabrera) teapawt.net/CarnegieMellon/status/…
An @SCSatCMU team has released a new interactive platform for data management and machine learning (ML) evaluation called Zeno. It empowers users to explore, visualize and analyze data and ML model performance across custom use cases. cmu.is/zeno
1
3
6
813
Lots of predictions of synthetic data for AI being big this year. Decided to look at the OG Alpaca dataset: hub.zenoml.com/project/f192e… Impressive for being GPT-4 generated w/ 1 prompt, but begs the question of how to generate diverse, OOD data
1
2
9
867
In case you missed it over the break - you can now visualize the outputs of any Eleuther LM Eval Harness run in @try_zeno with one command! 𝚙𝚢𝚝𝚑𝚘𝚗 𝚜𝚌𝚛𝚒𝚙𝚝𝚜/𝚣𝚎𝚗𝚘_𝚟𝚒𝚜𝚞𝚊𝚕𝚒𝚣𝚎.𝚙𝚢
We've teamed up with @AiEleuther to make it super easy to visualize your evaluation results in Zeno! Try it out the next time you run a benchmark: github.com/EleutherAI/lm-eva…
1
5
13
4,026
Google’s Gemini recently made waves as a major competitor to OpenAI’s GPT. Exciting! But we wondered: How good is Gemini really? At CMU, we performed an impartial, in-depth, and reproducible study comparing Gemini, GPT, and Mixtral. Paper: arxiv.org/abs/2312.11444 🧵
29
248
47
1,354
494,236
Since the initial release, we have significantly improved the usability of WebArena, accuracy of the evaluation, and provided interactive result analysis with @try_zeno I am attending #NeurIPS2023 , say hi 👋 if you are interested in AI agent, code gen and their evaluations!
🤖There have been recent exciting demos of agents that navigate the web and perform tasks for us. But how well do they work in practice? 🔊To answer this, we built WebArena, a realistic and reproducible web environment with 4+ real-world web apps for benchmarking useful agents🧵
2
9
45
9,965
Zeno retweeted
Since some of you might be wondering whether Mamba 2.8B can serve as a drop-in replacement of some of the larger models, we've compared the Mamba model family to some of the most popular 7B models in @try_zeno Report: hub.zenoml.com/report/2443/M… 🧵 1/5
Quadratic attention has been indispensable for information-dense modalities such as language... until now. Announcing Mamba: a new SSM arch. that has linear-time scaling, ultra long context, and most importantly--outperforms Transformers everywhere we've tried. With @tri_dao 1/
3
22
2
125
98,618
Recently there were some great results from the new Mamba architecture (arxiv.org/abs/2312.00752) by @_albertgu and @tri_dao. We did a bit of third-party validation, and 1. The results are reproducible 2. Mamba 2.8B is competitive w/ some 7B models (!) 3. Mistral is still strong
Since some of you might be wondering whether Mamba 2.8B can serve as a drop-in replacement of some of the larger models, we've compared the Mamba model family to some of the most popular 7B models in @try_zeno Report: hub.zenoml.com/report/2443/M… 🧵 1/5
3
30
2
264
48,435
Useful thread wrt understanding Google's new Gemini reported results.
Google just released 𝑮𝒆𝒎𝒊𝒏𝒊, their long-awaited GPT-4 competitor. Their report shows comparison across multiple common benchmarks, but 𝐡𝐨𝐰 𝐫𝐞𝐥𝐢𝐚𝐛𝐥𝐞 𝐚𝐫𝐞 𝐭𝐡𝐞𝐬𝐞 𝐫𝐞𝐬𝐮𝐥𝐭𝐬? 🧵 on potential issues with the benchmark scores
1
5
17
6,322
Google just released 𝑮𝒆𝒎𝒊𝒏𝒊, their long-awaited GPT-4 competitor. Their report shows comparison across multiple common benchmarks, but 𝐡𝐨𝐰 𝐫𝐞𝐥𝐢𝐚𝐛𝐥𝐞 𝐚𝐫𝐞 𝐭𝐡𝐞𝐬𝐞 𝐫𝐞𝐬𝐮𝐥𝐭𝐬? 🧵 on potential issues with the benchmark scores
6
28
15
173
76,308
Zeno retweeted
Awesome blogpost by our friends @huggingface and @AiEleuther and a demonstration of how @try_zeno can be used to systematically spot issues with benchmark results. Give it a read! Also: Zeno Report: hub.zenoml.com/report/1255/D… Zeno Project: hub.zenoml.com/project/2f5de…
⚠️ We are removing DROP from the Open LLM Leaderboard! With leaderboard evaluation data openly shared on 2000+ models, we did a deep dive with our friends @AiEleuther and @try_zeno, & found out that its original implementation is unfair to many models 😱 huggingface.co/blog/leaderbo…
1
7
702
Lots of info here on the several problems benchmarks can face
We loved collaborating with the @huggingface and @AiEleuther teams to investigate the odd behavior on the DROP benchmark! Check out the blog post and supporting Zeno report & project: Report: hub.zenoml.com/report/1255/D… Project: hub.zenoml.com/project/2f5de…
1
3
46
7,418
We loved collaborating with the @huggingface and @AiEleuther teams to investigate the odd behavior on the DROP benchmark! Check out the blog post and supporting Zeno report & project: Report: hub.zenoml.com/report/1255/D… Project: hub.zenoml.com/project/2f5de…
⚠️ We are removing DROP from the Open LLM Leaderboard! With leaderboard evaluation data openly shared on 2000+ models, we did a deep dive with our friends @AiEleuther and @try_zeno, & found out that its original implementation is unfair to many models 😱 huggingface.co/blog/leaderbo…
11
1
60
22,716
⚠️ We are removing DROP from the Open LLM Leaderboard! With leaderboard evaluation data openly shared on 2000+ models, we did a deep dive with our friends @AiEleuther and @try_zeno, & found out that its original implementation is unfair to many models 😱 huggingface.co/blog/leaderbo…
8
31
10
144
84,788
𝐂𝐚𝐧 𝐀𝐈 𝐝𝐨 𝐲𝐨𝐮𝐫 𝐭𝐚𝐱𝐞𝐬? Probably not 💸 Quick @try_zeno report on @danielgross' benchmark of tax Qs. LLMs struggle with any math, and it's hard to validate text answers without external references Report: hub.zenoml.com/report/1717/C… Explore: hub.zenoml.com/project/e7519…
This quoted post is unavailable.
1
5
11
5,679
Are the new @OpenAI Whisper models, including 𝗹𝗮𝗿𝗴𝗲 𝘃𝟯 and 𝗱𝗶𝘀𝘁𝗶𝗹, really better & faster than the original? We find that large v3 is better at identifying English audio & that distil models are competitive across accents. Full report: hub.zenoml.com/report/1123/C…
1
8
1
32
39,250
This is our thesis @try_zeno! Let the AI engineers design the evaluation but empower them to visualize the data, discover behaviors, and create & share evaluation reports. Essentially a BI/product analytics platform for AI. Would love to chat more! hub.zenoml.com
1
2
4
931
🤗 @huggingface released a 𝐦𝐚𝐬𝐬𝐢𝐯𝐞 update to their OpenLLM leaderboard with 3 new tasks. Using @try_zeno we explore each task and discuss what this means for the leaderboard's validity. TL;DR it's a 𝐦𝐮𝐜𝐡 better measure of performance. Report: hub.zenoml.com/report/cabrer…
Ready for the *biggest update* of the Open LLM Leaderboard yet? We just spent A YEAR of GPU time to make it more interesting and fairer! 🤯 How? With @nathanhabib1011, we added 3 new evals from the great EleutherAI harness 💥 and re-ran 2000+ models! 🚀 So, what changes? 🧵
3
15
3
52
40,455
Zeno retweeted
Heard of the @AiEleuther LLM Evaluation Harness? We've made it easy to upload harness results to @try_zeno to explore, compare, and visualize them! Check out our example exploring some of the BigBench Tasks! Project: hub.zenoml.com/project/a13x/… Notebook: github.com/zeno-ml/zeno-buil…
5
8
2,272
The recording of my talk on @try_zeno and generative AI evaluation from the @mlopscommunity conference is up! See how you can author interactive reports of AI behavior for everything from model debugging to auditing. youtube.com/watch?v=fS3taBVK…
2
10
1,163