Alireza Bashiri

A coding agent can optimize a security score without fixing the vulnerability. A new paper introduces FALCON, a verification system that grounds security feedback in actual program execution. The problem with some security rewards is that the model learns to satisfy the evaluator. That doesn't always mean the generated code becomes safer. FALCON separates attack exploration from behavioral verification. It uses execution traces to check whether an attack actually succeeds. The researchers tested it during code repair and reinforcement learning. They report stronger held-out functional-security results than the evaluated static-analysis and learned-reward baselines. The important part is the verification signal. If a model can improve its reward without fixing the underlying behavior, the reward is teaching the wrong lesson. For coding-agent harnesses, I'd rather verify exploitability through execution than rely only on a model judging the source code.

Your coding agent shouldn't be allowed to declare success without evidence. Found an interesting paper called Agentic Interpretation. Instead of keeping every finding inside a long conversation, the system stores typed evidence records. The model proposes evidence. The harness checks it, stores it, and tracks what remains unresolved. Only then can the agent make certain conclusions. The researchers tested this on patch acceptance and bug localization. With GLM-5, reject-class F1 on SWE-bench-Live improved from 0.256 to 0.543. On Defects4J, All@1 improved from 0.140 to 0.400. The paper also reports 35% fewer executions and 59% lower cost per candidate across its evaluated efficiency measurements. I like the architecture more than the headline scores. The model can investigate freely. But the harness controls which findings count as verified evidence. That's a useful pattern for code review, debugging, and autonomous releases.

A 9B coding model improved without keeping every training skill at deployment. Researchers introduced HarnessAdapt-RL to train the model and adapt its harness together. Instead of freezing the agent runtime during RL, the system changes tools and hooks as training progresses. The reported improvements over GRPO: SWE-bench Verified: +9.8 points SWE-bench Multilingual: +12.6 points SWE-bench Pro: +6.4 points Here's the interesting part. The researchers generated task-specific skills during training. The model retained some of those gains even after the skills were removed at evaluation. But removing the evolved tools and hooks at deployment reduced performance by up to 7.7 points. That suggests different harness components play different roles. Some training guidance can become part of the model's learned behavior. Other runtime capabilities still need to exist when the model executes. I think this matters for teams training coding models inside custom harnesses. Don't treat the model checkpoint as the entire trained system.

If your coding agent compacts context, read this before trusting its memory. A new security paper found that some harnesses can turn attacker-controlled text into apparent user instructions during compaction. The problem starts when a harness converts structured conversation messages into plain text. An attacker can place a fake user message inside material the agent reads. When the harness summarizes that material, it may lose the original message's trust boundary. The researchers tested nine harnesses. In five harnesses that flattened message roles, the attack survived compaction in 87–100% of their tests. In four harnesses that preserved structured roles, the same attack succeeded 0% of the time. That doesn't mean every installation is vulnerable in every configuration. But the failure mode matters. A prompt cannot reliably restore provenance after the harness has already discarded it. If you're building a coding-agent runtime, preserve message roles through compaction. Don't let untrusted tool output become a user instruction.

11 coding-agent configurations. 43 enterprise tasks. The best complete-delivery rate was just 14%. A new paper called PACE tested whether coding agents could implement complete product requirements inside mature repositories. The benchmark contains 750 expert-audited acceptance criteria. The agents often built individual components successfully. Then they missed the integration work. A feature existed, but the full workflow wasn't complete. This is a problem I keep coming back to with coding agents. A passing test doesn't mean the product requirement is done. If you're running coding agents on real projects, I'd check three things: • Does the final behavior match the original requirement? • Do the components work together? • Can the harness produce evidence for the complete workflow? The researchers also improved average benchmark performance from 59.4 to 63.0 by refining the harness while keeping the model and task suite fixed. The useful lesson is to verify delivery, not just code generation.

Self-improving agent harnesses can overfit just like models. A new paper on VideoHarness-SRSI studied more than 4,000 evolved harness candidates. The researchers found three recurring problems. Validation overfitting: The harness improves on the evaluation data but doesn't generalize. Search-budget dilution: The system spends compute exploring too many weak candidates. Early-experience trap: Initial feedback pushes the search toward approaches that may not work elsewhere. The researchers traced these problems to how the system reused imperfect validation evidence. They changed the search process. The new approach limits the feedback shown to the proposer, uses warm-up experiments, and tests mechanisms in isolated branches before combining them. Across four video-understanding benchmarks, the revised system improved the average held-out score by 6.2% relative to the earlier approach. It also reduced the validation-test gap by 37.9%. I think this matters beyond video agents. We're starting to let agents modify their own prompts, tools, memory, and execution rules. But a harness that improves on ten familiar tasks might become worse on unfamiliar ones. I'd want separate validation and held-out tasks, rollback support, and explicit regression budgets. Self-improvement only helps if the changes generalize.

Researchers improved agent harnesses by diagnosing failures before changing the architecture. A paper called Trace-to-Architecture (T2A) studies how to identify which part of a harness needs improvement. The problem is that similar agent failures can have different causes. A failed task might come from poor context selection, missing tools, weak planning, or incorrect recovery behavior. Changing the system prompt won't necessarily fix those problems. T2A uses execution traces from successful and failed tasks to diagnose bottlenecks. It then revises the relevant harness components. The system checks three things: Did the failed tasks improve? Did previously successful tasks remain successful? Did the new harness component actually run? Across ALFWorld, ScienceWorld, and AppWorld, T2A outperformed the strongest evaluated baselines by 6.2–30.3 points. The researchers also integrated its diagnosis process into Meta-Harness and improved ScienceWorld performance by 21.6 points. I like the idea of checking whether a harness change actually affected execution. Otherwise, it's easy to add instructions, tools, or middleware that the agent never uses. I'd want every harness revision to include a measurable hypothesis and a regression check.

The weakest model beat the strongest model when researchers gave it a better harness. A paper called "A Strong Harness Narrows the Model Gap" tested this on legislative document conversion. The researchers compared three Claude models across three execution setups: • Direct zero-shot generation • A single tool-using agent • A decomposed multi-agent pipeline They used 153 legislative documents and measured how accurately the systems converted them into structured XML. The result: Weakest model with the full harness: 0.948 Strongest model running alone: 0.806 The two agentic setups used identical tools. The difference came from orchestration, persistent state, and how the system divided the work. One detail caught my attention. Citation normalization remained difficult because the system needed exact identifiers from an external authority. A deterministic lookup tool helped solve that problem. I think this is a useful example of where harness engineering adds value. The model doesn't need to memorize every external identifier. It needs the right tool and a reliable way to use the result. The study covers a specialized document-conversion task, not general coding benchmarks. But the controlled comparison is worth studying.

Long-running coding agents need project state, not just longer conversation history. A paper called StateHarness explores this problem. The researchers built a runtime that tracks what the agent is doing across long development tasks. It records: • Current problems and hypotheses • Important findings • Repository dependencies • Previous decisions • Unfinished work The harness also checks which callers, tests, and documentation may be affected when the agent changes code. When enough execution evidence accumulates, a separate diagnostic session reviews the history for missed information. The reported results on RefactorBench: Full-chain completion: 29.4% → 35.3% Cleanup accuracy: 87.0% → 95.4% On LHTB, mean reward improved from 0.427 to 0.576. The researchers also reported improvements on ChainSWE, although some regressions increased. I think this is worth studying. Most coding agents rely heavily on conversation history to remember what happened. But a long-running task needs more structured information. What changed? Why did it change? What still needs work? What other files depend on it? I'd like to see more harnesses maintain this state explicitly instead of repeatedly summarizing the conversation.

An AI agent benchmark reported 94.3% success. After researchers fixed its evaluation logic, that dropped to 65.5%. A paper called "When Agent Evaluation Becomes Unreliable" audited the software responsible for scoring agent tasks. The researchers discovered a reward-function bug. The evaluation code changed the environment state before measuring the final result. That made some failed executions appear successful. Across 4,320 episodes, correcting the scoring logic reduced measured success from 94.3% to 65.5%. They also tested how configuration changes affected results. Tool descriptions, policy documents, and error messages changed observed trajectory reliability. This is something I think agent benchmark discussions often miss. The benchmark harness is software too. It can have bugs. It can change the environment. It can reward the wrong behavior. Before trusting an agent score, I'd want to know: • Does the evaluator modify task state? • Can known failures pass? • Can known successes fail? • Can the results be reproduced? • Does the same configuration produce consistent labels? A benchmark score is only as reliable as the system that measures it.