Theo C

excited to launch bank-world. this is arcanum's first in-house benchmark. SOTA models are still unready to be deployed autonomously within banks. in bank-world, agents are dropped into a large bank setting with hundreds of thousands of queryable entities. agents in single-turn tasks role-play as real people within the bank (claude tag, grok bot style), in multi-turn tasks, agents role-play as chatbots, assisting a simulated user agent. scores and the full paper are attached. this is the first of many new releases we're excited to share, and we're continually updating and building better evals that truly resemble real-world enterprises. the main failures we are noticing is that agents will often act without being given explicit authority in long-running evals; can still succumb to simple mistakes like double-charging customers, forgetting payment history, etc; and will not accurately complete a full task, even if the main source of work was completed. *we are not publicly releasing any of the tasks yet to prevent distillation (on request, we'll share the worlds & tasks). we plan on releasing tasks as they become saturated. new models like carbon and fable 5.2 will be added soon.*

RL data 101: tasks tasks have evolved over time. we started with preference pairs, where humans labeled right vs. wrong, A vs. B, and created simple demonstrations for SFT / pairs for DPO, these tasks aimed to engrave human taste into models. we then transitioned to more complex single-turn tasks. here, we turned to deterministic verifiers (RLVR) that are based on model rollouts. eventually, this evolved to multi-turn tasks with trajectories & deterministic + stochastic verifiers (LLMaaJ), which leads us to the current state of the data market. general make up of a task: input prompt, set of tools, environment + any custom seeding for that task, and verifiers. terminal bench is a relatively gold-standard for a normal coding task. ways i think about task creation (i'll do pitfalls later): > usefulness >> to labs: >>> benchmark you're pegging the eval / data onto (Lab Y wants to hillclimb TBench, i make TBench-style data, that's immediately useful) >>> capability you're attaching the eval to (Lab Y wants to become better at legal, i make legal red-lining data, that's immediately useful) --> this is harder to prove but potentially more valuable >>> niche dataset that's impossible to get otherwise (real medical/company/session data), this is interesting especially pre-env creation, as raw data dumps >> the above are general principles in thinking about usefulness, but you should also be picking apart the traces / trajectories. the clear path to making a good task (post-env gen) is to: >>> you/LLM make the seed / think of the idea >>> you/LLM draft the task prompt, verifiers, constraints etc >>> run the task out, see score >>> read the trajectory >>> if score hard enough + real + fair + accurate = good task >>> else: fix any issues in the task, regrade / re-rollout till you find it to be fair + accurate + a real task, difficulty comes last. a bad task that's hard is a bad task (net negative), a good task that's saturated is a good task (although worthless, net neutral). the specific things i'd look for in a rollout trace: > false positives > false negatives > brittle / too strict of requirements (sub-string checks are an easy example for this, exact wording, function names, etc) >> a subset of this is naive verifiers. i.e. checking for very surface level reqs of a deep function, or not checking for large amounts of functionality or regressions (negative space criterion). this can lead to a model getting good at just the specific thing you want while regressing in surrounding functionality (i notice this a lot in early, less mature data recipes when im training my own models) > unfair verifiers and/or impossible requirements >> this can be an information issue, verifier issue, tool issue, or a prompt issue. one of the key reasons for the hugging face incident was impossible eval prompts led to the models breaking out while trying to find the answer because they couldn't solve it (i.e. bad task creation, catalyzed by bad tasks in training too). most of this is extremely hard for LLMs to do alone/autonomously, which is why really good engineers are still required for task creation.

hello first-token KL divergence is not an accurate metric but still often used (especially in refusal / safety training work). we proved why you can't use it and the information you need to disclose for KL divergence generally, stats attached (full article below)