Pinned Tweet
Can agents use experimental feedback to establish new material capabilities that hardware teams can reliably design around?
Today we’re releasing SDL-2, an open evaluation of how agents choose experiments and decide whether a material process will reliably meet a requirement.
We’re interested in the full process of scientific investigation. That means discovering which relationships matter, choosing experiments that distinguish competing explanations, revising an explanation when results surprise and knowing when the evidence is strong enough to act.
The experiments run in a digital twin of magnet processing, calibrated on measurements from our campaign with Rensselaer Polytechnic Institute (@rpi). Agents investigate unfamiliar powders with an uncertain physics model, a limited experimental budget and results that vary from batch to batch.
Across 32 simulated worlds, frontier models reached correct decisions supported by their own measurements in 44–50% of investigations, compared with 25–28% for other language models and classical experiment design. One in three of their correct answers were unsupported.
Our larger ambition is for materials development to keep pace with hardware design, so teams can design around the capabilities they request, not only those that already exist. SDL-2 makes one part of that measurable, how agents investigate.
We're releasing the evaluation code, harness, digital twin and all 256 investigation traces.
Technical Report: dynamicalsystems.ai/blog/sdl…
Code: github.com/Dynamical-Systems…
Dataset: huggingface.co/datasets/Dyna…
We're opening our Lab Notebook, an open science series sharing our work in progress.
Our notebooks share which material capabilities could change a design, how we investigate them, and what it would take to make them reliably.
Notebook 001 - Can a thinner magnet make a propulsion motor lighter?
Removing magnet material saves rotor mass. Keeping the same torque can mean more current, copper and heat. The hard part is that these constraints are intertwined. Change the magnet and the current, copper, heat and fault response all move with it.
We use a public motor model to explore how the material, winding and motor construction affect that tradeoff.
dynamicalsystems.ai/lab-note…
In the reference calculation, thinning the magnets saved 8.5 g of rotor mass. Keeping the same torque took 11.5% more current. Adding enough copper to hold copper loss constant left the rotor and winding 4.0 g heavier.
The ring behind the magnets changed the answer, too. With the alternative assumed steel, keeping the original magnets saved more mass than thinning them. Changing the winding also changed the field opposing the magnets during a short circuit.
1
2
18
Our efforts in these investigations are to make material requirements more specific. A useful magnet has to stay strong at operating temperature, resist the opposing fields it meets under load and during a fault, and not cost more in copper, cooling and retention than it saves.
Catalog grades may meet the magnetic requirement on paper. The open question is the finished part. Finishing, coating and magnetization can all change what a magnet delivers when hot.
If you’re interested in pursuing these open questions with us, reach out. We’d love to chat.
dynamicalsystems.ai/lab-note…
2
17
Dynamical Systems retweeted
Why do we need more benchmarks for frontier models? What you can evaluate and understand, you can optimize. Excited to share some of our work on connecting scientific reasoning, experiments, measurements and process decisions.
@dynamical__ SDL-2 is an open materials science benchmark that follows the whole investigation, from discovering which relationships matter and choosing experiments that distinguish competing explanations to revising an explanation when results surprise and knowing when the evidence is strong enough to act.
It asks agents to infer how much results vary from a few observations and choose experiments that identify which uncertainty would change the decision.
Code, harness, environments and all traces are open!
Can agents use experimental feedback to establish new material capabilities that hardware teams can reliably design around?
Today we’re releasing SDL-2, an open evaluation of how agents choose experiments and decide whether a material process will reliably meet a requirement.
We’re interested in the full process of scientific investigation. That means discovering which relationships matter, choosing experiments that distinguish competing explanations, revising an explanation when results surprise and knowing when the evidence is strong enough to act.
The experiments run in a digital twin of magnet processing, calibrated on measurements from our campaign with Rensselaer Polytechnic Institute (@rpi). Agents investigate unfamiliar powders with an uncertain physics model, a limited experimental budget and results that vary from batch to batch.
Across 32 simulated worlds, frontier models reached correct decisions supported by their own measurements in 44–50% of investigations, compared with 25–28% for other language models and classical experiment design. One in three of their correct answers were unsupported.
Our larger ambition is for materials development to keep pace with hardware design, so teams can design around the capabilities they request, not only those that already exist. SDL-2 makes one part of that measurable, how agents investigate.
We're releasing the evaluation code, harness, digital twin and all 256 investigation traces.
Technical Report: dynamicalsystems.ai/blog/sdl…
Code: github.com/Dynamical-Systems…
Dataset: huggingface.co/datasets/Dyna…
As a process moves from the lab to manufacturing, new powders, equipment and operating conditions reopen the reliability question.
Agents must infer where variation comes from, choose experiments that separate those sources and commit only as far as the evidence allows. Our broader hypothesis is that these judgments can be learned through experimental experience and remain useful when the material or conditions change.
Our next research objective is to identify what experience and learning signal would make an investigator better at discovering, testing and acting on scientific structure in unfamiliar worlds. Progress means more discovery and more correct, supported decisions on combinations and conditions the agent has not seen.
Our goal is to establish new material capabilities with less wasted experimentation, even when the previous recipe does not transfer.
1
1
19
The harness, grader, digital twin and its virtual instruments, all 32 evaluation worlds, and all 256 investigations with scores and traces are public.
Run an agent, inspect a failure, test a learning method, or help us extend the environment.
Code: github.com/Dynamical-Systems…
Dataset: huggingface.co/datasets/Dyna…
1
3
29




