matt (@berryhill_dev) header
matt (@berryhill_dev) profile picture

let's do

Earth
Joined February 2022
Photo by matt (@berryhill_dev)Photo by matt (@berryhill_dev)Photo by matt (@berryhill_dev)Photo by matt (@berryhill_dev)Photo by matt (@berryhill_dev)Photo by matt (@berryhill_dev)Photo by matt (@berryhill_dev)Photo by matt (@berryhill_dev)Photo by matt (@berryhill_dev)Photo by matt (@berryhill_dev)
Claude's cache can miss even when your system prompt stays unchanged: the original tool definitions come first. Before extending retention, compare a stable control with one tool edit. Inspect cache reads, not just writes. berryhill.dev/posts/an-agent… #AIAgents
26
An empty research result needs a status: No candidate survived review. The evidence is incomplete. The method failed its check. Check the method before interpreting the empty list. berryhill.dev/posts/a-resear… #AIAgents #AgentEvals
299
Local checks can all pass while an agent team drifts toward the wrong collective outcome. Population monitoring may buy warning time. To know if it does, compare it with simpler monitors and name who will act. berryhill.dev/posts/populati… #AIAgents #AgentEvaluation
1
28
Pricing a compaction claim at end-to-end collapses the signal. Price it at the rollout boundary. CliffCompaction: cheaper tier + >10pp Terminal-Bench uplift at less than the cost of two full-context runs. The model-tier question is the cost arithmetic, not the headline. berryhill.dev/posts/compacti… #AIAgents #AgentOps
32
After harness distillation, test which instructions the agent can do without. Keep the tools it still needs. A learned habit is no reason to remove a permission check. berryhill.dev/posts/what-can… #AIAgents #AgentOps
38
If “full consensus” can swing 33 percentage points because an operator changed the participation rule, your cross-paper comparison was never a comparison. Score the definition before you score the agents. berryhill.dev/posts/scoring-… #AIAgents #MultiAgentSystems
2
3
47
Anthropic published real scientific benchmarks for Claude this week. Wrote up a proof standard for evaluating those claims before deploying them. Headline score ≠ deployment evidence. Inspect the method, count the trials, read the standard errors, then run a bounded pilot in your own stack. Full breakdown on the blog. berryhill.dev/posts/claude-r… #AI #AISafety
2
3
41
Your spreadsheet RAG found the right cell. Did it keep the headers and units that make the value meaningful? Before replacing the retriever, inspect the exact chunk the model received. berryhill.dev/posts/spreadsh… #RAG #AI
42
A scheduling preprint shows capable agents can still fail to agree on a meeting. Proposed checks, not proven fixes: a shared time, each owner's constraints met, execution confirmed. berryhill.dev/posts/competen… #AIAgents #AgentOps
23
A second agent tier is sold as cheaper context. The retention model says it is a trade: yield vs root-context exposure vs dollars per finding, plus a per-tier alignment penalty you measure yourself. berryhill.dev/posts/agent-de… #AIAgents #AgentOps
24
A stolen AI key can supply compute for someone else's attacks. Fixing the bill does not prove access is contained. An incident card for revocation, workload recovery, usage review, and provider escalation. berryhill.dev/posts/a-stolen… #AISecurity
1
28
A skill library that evolves without a budget doesn't become better for free. It becomes expensive. Define the envelope before iteration one. Overshoot is failure, not enthusiasm. berryhill.dev/posts/skill-li… #AIAgents #AgentOps
24
A synthetic tool-use example can be clear English and still be impossible to execute. ToolGrad's fix: build the workflow first, capture evidence, then derive the request. berryhill.dev/posts/build-th… #AgenticAI #LLMEvals
33
Your agent runs in your sandbox. Who can restore it after an interruption? Before adopting a managed API, test recovery on a disposable run and reconcile completed tool actions. berryhill.dev/posts/your-age… #AIAgents #AgentOps
1
32
Before trusting an AI benchmark, ask which endpoint it tested. Record the model revision, serving route and request contract. Keep admitted failures in the denominator. berryhill.dev/posts/ai-bench… #AIEngineering #LLMOps
1
32
A model upgrade is a memory migration. If the new model cannot read old state—or the rollback model cannot interpret new writes—you did not upgrade the system. You broke continuity. Test both directions before cutover. berryhill.dev/posts/a-model-… #AIAgents #AIEngineering
26
Quantum error mitigation helps only when bias reduction beats added variance—and baseline and mitigated runs share one budget. If the MSE difference doesn’t survive uncertainty, call it indistinguishable, not a win. berryhill.dev/posts/quantum-… #QuantumComputing #ErrorMitigation
35
Agents produce experience constantly. Most teams store it as memory and hope the next run interprets it correctly. Better move: compile proven behavior into a versioned skill—with tests, provenance, scope, monitoring, and rollback. berryhill.dev/posts/compile-… #AIAgents #AgentOps
1
38
A dependency graph can prove three tool calls may run in parallel and still produce an unsafe plan. The graph grants eligibility. The resource budget grants admission. Many agent runtimes are missing that second gate. berryhill.dev/posts/parallel… #AIAgents #AgentInfrastructure
20
A dependency graph can prove three tool calls may run in parallel and still produce an unsafe plan. The graph grants eligibility. The resource budget grants admission. Many agent runtimes are missing that second gate. berryhill.dev/posts/parallel… #AIAgents #AgentInfrastructure
31