Benjamin Feuer retweeted
How can we train small agentic models that are highly capable of terminal use and coding?
Announcing OpenThoughts-Agent + OpenThinkerAgent-32B, the strongest Qwen-3 based open-data agentic model: 44.8% avg across 7 agentic benchmarks! (1/n)
Benjamin Feuer retweeted
If you are attending #ICML2025, check out our DataWorld workshop on Sat July 19. We have updated the website with more info on speakers & accepted papers! dataworldicml2025.github.io/
Also happy to chat offline about all things ✨ data ✨
Benjamin Feuer retweeted
Announcing OpenThinker3-7B, the new SOTA open-data 7B reasoning model: improving over DeepSeek-R1-Distill-Qwen-7B by 33% on average over code, science, and math evals.
We also release our dataset, OpenThoughts3-1.2M, which is the best open reasoning dataset across all data scales. Full details are in our ✨new paper✨ - below we share the highlights:
BTW, it also works on non-Qwen models😉 (1/N)
Benjamin Feuer retweeted
📣We are extending our deadline to May 31st!📣
Looking forward to seeing everyone's submissions :)
📢 Announcing our data-centric workshop at ICML 2025 on unifying data curation frameworks across domains!
📅 Deadline: May 24, AoE
🔗 Website: dataworldicml2025.github.io/
We have an amazing lineup of speakers + panelists from various institutions and application areas.
Benjamin Feuer retweeted
📢 Announcing our data-centric workshop at ICML 2025 on unifying data curation frameworks across domains!
📅 Deadline: May 24, AoE
🔗 Website: dataworldicml2025.github.io/
We have an amazing lineup of speakers + panelists from various institutions and application areas.
Benjamin Feuer retweeted
If AI isn’t truly open, it will fail us. We can’t close in a black box our greatest invention yet just so that a few can freely monetize. AI needs its Linux moment, and so we started working towards it. This can only succeed if we all work together!
#oumi #opensource #collaboration
@rsalakhu @svlevine @larry_heck
@karpathy @atalwalkar @prfsanjeevarora @tsvetshop
@hhexiy @sainingxie @larry_heck @AnimaAnandkumar
@JunjieHu12 @georgiagkioxari @profjoeyg @pliang279
@danqi_chen @ChrisGPotts @BillMacCartney @vinodv
@tur_gokhan @dilekhakkanitur @j_foerst @gingsmith23 @kahinish @jamesjoaquin @gan3sh @ethanjb @kirbywinfield @egonzdp
Benjamin Feuer retweeted
Oumi is a fully open-source platform to help you build state-of-the-art foundation models, end-to-end.
Benjamin Feuer retweeted
🐞 Check out this zero-shot AI model dataset with 6M images of important species that is vital to farming and environmental research! Learn more: ow.ly/Kzig50TRSaC
#ZeroShotLearning #AIModels #AgriculturalTech #EnvironmentalResearch
@chegday @FeuerBenjamin @AII4RA
Benjamin Feuer retweeted
Micah's an incredible mentor and one of the most creative early-career AI/ML thinkers out there -- if you're looking for a potential PhD advisor, can't imagine a better choice!
Benjamin Feuer retweeted
In our new paper, we find that LLMs can efficiently do RLHF in-context!
Our method, in-context preference learning (ICPL), iterates LLMs writing reward functions, training agents, and putting preferences into context. We see a 30x boost in query efficiency over baseline RLHF!
Benjamin Feuer retweeted
👉 NeurIPS paper w @fjzzq2002!
The text below was generated by a random transformer in which *only embeddings* were trained on real data.
How effectively can you steer a random transformer through embeddings alone? What else can these models do?
arxiv.org/abs/2410.04368
Benjamin Feuer retweeted
Thrilled that Arboretum* was accepted to #NeurIPS2024 (D&B) as a spotlight! Congrats to the whole team.
USDA/NIFA recently highlighted this work here:
nifa.usda.gov/about-nifa/imp…
* Several updates are in the pipeline, including a name change (!!). Stay tuned...
Benjamin Feuer retweeted
Evaluating LLMs is hard and important. Leaderboards drive public discussion of new models & private decisions about which models to test or roll to prod.
LLM judges - Arena-Hard-Auto, MT-bench, etc - are used to automatically evaluate model quality. But they're not perfect!
1/
Benjamin Feuer retweeted
Finally, a nice paper that evaluates whether o1 can plan.
Here is a quick summary of the interesting bits:
A domain-independent planner (Fast Downward) can solve all instances of Mystery Blocksworld. LLMs struggle, even on small instances. But Large Reasoning Models (LRMs) like o1-preview are effective but tend to degrade in performance as plan length increases.
Now look at the drill down on the performance of the o1-preview and o1-mini models compared to Fast Downward:
Accuracy/costs Tradeoffs: The authors claim that "while o1-preview may provide higher accuracy than LLMs, it still fails to provide any correctness guarantees, and it is unclear whether it is at all cost-effective."
An interesting comment from the paper on o1's creative justifications: "While our main focus has been on providing a quantitative evaluation of o1’s performance on PlanBench, we have also noticed an o1 idiosyncrasy that is worth commenting on. When the model gives an incorrect answer, it also sometimes provides a creative, but nonsensical, justification for its decision. It is almost as if o1 has gone from hallucinating to gaslighting!"
The conclusion in the paper is that while o1 shows a bit of progress on more challenging planning problems, the accuracy gains cannot be considered general or robust (particularly, on longer problems and unsolvable instances).
Benjamin Feuer retweeted
Answer the questions that people aren’t already asking but should be asking.
Benjamin Feuer retweeted
(3) o1 is extremely strong, but it's not perfect! In code generation (based on LeetCode and AtCoder), o1-mini **becomes the new SotA**. But in code completion, it performs worse than gpt-4o. Full livebench.ai results coming soon!
(3 / ?)
Benjamin Feuer retweeted
We just rolled out prompt caching in the Anthropic API.
It cuts API input costs by up to 90% and reduces latency by up to 80%.
Here's how it works:
Benjamin Feuer retweeted
LiveBench Benchmarks are for the new GPT-4o-2024-08-06
Claude-3.5 still at no-1
Benjamin Feuer retweeted
Now that we have a frontier model that's open-weight (not open-source), it's time to go back to all those ambitious use cases where open-weight models failed to deliver (agents) and try again, so we can have reproducible science and not worry about API models getting deprecated.
Benjamin Feuer retweeted
Llama-3 405B is now openly available and beats GPT-4o on most metrics!
- Sonnet 3.5 barely inches out 405-b!
- It's the best open-source model in the world
- At 128K context open source length, it is a pretty good model for RAG
Thanks to Meta for this excellent model!! It can be applied to a myriad of real-world use cases.
Finally, open-source has officially and fully caught up with Closed Source!






































