Research @EngramLab Prev: PhD @Princeton
Joined August 2013
- Tweets225
- Following1.5K
- Followers1.5K
- Likes3.8K
Pinned Tweet
AI agents now seem to solve the hardest problems in math and science, but they haven't yet deepened our understanding at the same rate. They are great problem-solving machines when the objective and optimization process are clear, but discovering insights is a lot fuzzier!
We build EurekaBench to measure this gap using 6 natural science domains (neuroscience, geophysics, plasma physics, astrophysics, computer science, and chemistry). We test if AI agents can complete the whole scientific discovery loop from understanding the observations, explaining the data by describing the mechanism (e.g., equations), and eventually discovering insights.
Results show that the agents don't advance on both improving accuracy and finding insights at the same pace! A lot more details in the thread. Fantastic work led by @JiayiiGeng.
AI scientist agents are great at optimizing and finding solutions through endless trial and error. But is that really what science is about?
As Terrance Tao eloquently put, the role of math and science should be more than that – they are "lighthouses" that guide and inspire exploration through understanding and insights. Can AI agents discover novel insights through experimentation?
To study these gaps, we worked with domain experts to introduce EurekaBench, a benchmark spanning 6 science domains that evaluates an agent’s ability to discover genuinely new scientific insights.
(1/n)
Howard Chen retweeted
We’re planning to expand EurekaBench, an expert-verified benchmark that tests whether AI agents can go beyond simply finding solutions to discover novel scientific insights.
We’re looking to collaborate with domain experts on two types of contributions:
1. Additional insight questions for existing problems: Different approaches to the same science problem can yield different scientific insights. If you've worked on a problem already included in EurekaBench but approached it differently, we’d love to hear what additional insights you think an AI agent could discover.
2. New scientific problems: We’d like to include more open scientific problems across our six existing domains, as well as in new domains. These may be recently published science problems that remain only partially solved, or unpublished problems.
If you’re interested in contributing, please check out our contribution guidelines for more details: eurekabench.github.io/contri….
AI scientist agents are great at optimizing and finding solutions through endless trial and error. But is that really what science is about?
As Terrance Tao eloquently put, the role of math and science should be more than that – they are "lighthouses" that guide and inspire exploration through understanding and insights. Can AI agents discover novel insights through experimentation?
To study these gaps, we worked with domain experts to introduce EurekaBench, a benchmark spanning 6 science domains that evaluates an agent’s ability to discover genuinely new scientific insights.
(1/n)
Howard Chen retweeted
The most profound paper of my PhD so far. We truly did something special to make LMs play/explain chess, needing both architectural & algorithmic innovation (natural-language analogue of the Alphazero algorithm) + applicable to many other domains. Please read on and share!
1/10
Howard Chen retweeted
Prediction ≠ Insights. Despite the fact that AI agents excel at well-defined prediction tasks, they still struggle to distill insights from predictions like human scientists do, across a variety of domains. Check out this paper by @JiayiiGeng that asks this important question.
AI scientist agents are great at optimizing and finding solutions through endless trial and error. But is that really what science is about?
As Terrance Tao eloquently put, the role of math and science should be more than that – they are "lighthouses" that guide and inspire exploration through understanding and insights. Can AI agents discover novel insights through experimentation?
To study these gaps, we worked with domain experts to introduce EurekaBench, a benchmark spanning 6 science domains that evaluates an agent’s ability to discover genuinely new scientific insights.
(1/n)
Howard Chen retweeted
Scientific discovery is very different from solving fixed-procedure tasks.
Agents are getting really good at brute-force search and optimization. But science is about more: finding mechanisms, connecting the dots, and uncovering insights that open new directions.
EurekaBench shows this gap clearly: agents can beat humans at prediction, but still struggle to generate the insights that keep science moving forward.
The goal isn't just to search faster, but to figure out what’s worth searching for next!
AI scientist agents are great at optimizing and finding solutions through endless trial and error. But is that really what science is about?
As Terrance Tao eloquently put, the role of math and science should be more than that – they are "lighthouses" that guide and inspire exploration through understanding and insights. Can AI agents discover novel insights through experimentation?
To study these gaps, we worked with domain experts to introduce EurekaBench, a benchmark spanning 6 science domains that evaluates an agent’s ability to discover genuinely new scientific insights.
(1/n)
Howard Chen retweeted
AI scientist agents are great at optimizing and finding solutions through endless trial and error. But is that really what science is about?
As Terrance Tao eloquently put, the role of math and science should be more than that – they are "lighthouses" that guide and inspire exploration through understanding and insights. Can AI agents discover novel insights through experimentation?
To study these gaps, we worked with domain experts to introduce EurekaBench, a benchmark spanning 6 science domains that evaluates an agent’s ability to discover genuinely new scientific insights.
(1/n)
Howard Chen retweeted
Work from @EngramLab on study compute reducing inference tokens while simultaneously increasing their quality👇
Howard Chen retweeted
this is a crazy chart. having a model study its environment is probably how you get a big capability jump on the long tail of real-world tasks
Howard Chen retweeted
Awesome research by @engramlab using the synthetic law firm we built.
They trained a 27B Qwen model on synthetic client matters and used the learned knowledge to do online search more efficiently. The trained model outperforms frontier models at 10x lower cost per query.
The results are promising for scaling up enterprise search and personalization.
Howard Chen retweeted
We had Qwen 3.8 study all of the docs in Harvey’s synthetic law firm - the more it studies, the fewer tokens it needs at inference time to respond to the queries
The paradigm in the last few years has been scaling reasoning to get better performance. It will remain to be extremely important (yet costly).
There's another axis to scale: through study, our agent internalizes the env and navigates it with familiarity and intuition. We are seeing sparks of something different, and it's just the beginning.
Howard Chen retweeted
So excited to share our first result! We get agents to study and deeply understand the context of a synthetic law firm we built with @Harvey. By combining text memory and parametric memory in training, we solve complex search and knowledge tasks over the law firm with a 3.3x token reduction while outperforming Opus 4.8 X-high.
Excitingly, we also see that studying unlocks unique agent behaviors. It allows the agent to weave knowledge into its behavior — not only recalling things off the top of its head, but also using this to take more informed actions in the workspace.
Howard Chen retweeted
This is a good time to share that I’m taking a sabbatical to work at Engram full time!
Our blog post highlights some of the directions we’re taking, building agents that study and learn with complementary memory systems.
There's lots of research still to be done, and I’m excited to help lead these efforts!
Howard Chen retweeted
Before my first day at @EngramLab, I preempted many of my own questions by staying up and reading internal documentation. Our new models with @harvey scale a larger version of this process to study the entirety of a law firm. These models display qualitatively new behaviors and show promising returns from scale.
If you are interested in reading the best research docs in SF and scaling this new paradigm of informed action, we are hiring!
Howard Chen retweeted
The best frontier models know the law. The hard part is knowing the firm and the knowledge that compounds over thousands of hours and matters. So we placed our agent in the synthetic law firm that we built together with our friends at @harvey, and had it study the 100M tokens of legal work in Calderwood & Harkness!
Interesting questions emerge when an agent understands your workspace. Knowledge and memory shape how it navigates and searches the firm’s corpus — it is able to cut through the noise. Having studied the firm, our agent knows where to look, when search misses a matter it remembers, and how to search with terms that appear nowhere in the query but it knows are relevant from memory.
Agents should get better in a workspace the longer they work in it. This is what a first step towards that looks like.
Howard Chen retweeted
Like real people, Engram's models use both neural memories and written notes to tackle hard tasks.
Many legal tasks require reingesting the same persistent context: the firm filesystem. Scaling compute preemptively on that filesystem can obviate many challenges at inference time and unlock new types of behaviors.
Howard Chen retweeted
very proud to share some of our first work at Engram!
These days, we have many tools for models to learn and remember: learning knowledge in weights, writing text files, practice with RL, on-policy distillation, etc.
We've been thinking about what it would look like to have an agent that uses many kinds of memory natively, the same way human experts learn over time. it doesn't look like memorizing everything in weights, or grooming an elaborate system of notes, but naturally integrating all of these together to familiarize yourself in the same context day after day.
We put our agent to work in Calderwood & Harkness, a 100M token synthetic law firm from our friends at Harvey, and are super excited to show a first preview of what this kind of system could look like!