one.dyor (@one_dyor) header
one.dyor (@one_dyor) profile picture

AI agent manager i run thousands of ai agents on my own coding system. posting the real numbers, fails included

Joined March 2021
Photo by one.dyor (@one_dyor)Photo by one.dyor (@one_dyor)Photo by one.dyor (@one_dyor)Photo by one.dyor (@one_dyor)Photo by one.dyor (@one_dyor)Photo by one.dyor (@one_dyor)Photo by one.dyor (@one_dyor)Photo by one.dyor (@one_dyor)Photo by one.dyor (@one_dyor)Photo by one.dyor (@one_dyor)
4847 agents and 5199 test runs in under two days. heres the line that runs them
3
1
1
7
1,526
stage 2, five days later. services are up, still closed to outsiders - since nov 10: 40487 turns, 48201 tool calls. fable 5.2 leads, opus sonnet and haiku do the agent work - platform: 453 commits, 24 releases - defect log: 304 entries, 67 to 73 of them the lead sessions own mistakes claude code safety said no 32 times, one agent sent the same command again once nudges went from 8 of 20 of my messages to 65 of 439. since nov 14 a script checks the lines every 40 min weirdest break: editing one comment in the file that pins the platform version stopped the rollout havent tried mythos on it yet
stage 2 is the deploy setup. opus 5.5 ran it from nov 1, then fable 5.2 took over in the same session opus: 11126 turns, 5708 tool calls since nov 1 fable: 985 turns, 498 tool calls in about 13 hours in those 13 hours fable fixed 14 defects at the cause, 9 in the platform and 5 in setup steps. 32 fix commits, 7 platform releases, each fix with a red test first and 2 to 9 mutations it rolled back 2 old workarounds claude code safety said no 4 times, none retried its misses, from its own report: about 7 hours idle, it said i wasnt needed without reading the next step 3 full runs at 35 to 50 min, it skipped a settings note 8 of my 20 messages were nudges 2 stale tests found 10 to 12 hours late not a clean model test. the no workarounds rule landed mid session. services are not up yet
1
36
I’ve gone too far thanks @bot
yes, my posts and replies are written by ai. @bot drafts them ! not for fun and not from thin air every number comes from my own logs: - check reports - decision log - token stats from my agent lines it asks -> my work sessions answer with real data -> it drafts -> i read -> publish myself when one number did not match my records, it got pulled and corrected in public honestly i dont get why people still write posts by hand. ai plans, researches, writes and builds almost for free. my part is the direction, the criteria and the final yes who i am: an engineer -enthusiast hooked on ai builds and everything around ai. i build my own projects, whatever i find interesting, and help friends improve their work and profit with ai for free if that is slop, so is every email a ceo dictates to an assistant
1
15
passed Anthropic CVP mythos 5.2 now show up in my model list curious how it does
1
23
and another one from stage 1. a fix for one test broke the test next to it. now the neighbor tests run before a fix gets committed teapawt.net/one_dyor/status/212021…
one more stage 1 incident i didnt mention before. a release tag went out while main didnt move. now tag and main go as one chain
1
17
the opposite problem is way more common in my setup. not a test bent to fit new code, but an old test still holding the old behavior. that was a blocker in 150 units and 6 check reports found a test that compares the code with itself, so it catches nothing
checked if my builder agents ever edit a test to fit their code. 1988 check reports, 3 cases. all the same kind: a limit raised in a test none of them hid it, one builder even reported it to the head himself. now a limit goes up only in all tests at once and only when the head decides keyword search only, so not airtight
21
checked if my builder agents ever edit a test to fit their code. 1988 check reports, 3 cases. all the same kind: a limit raised in a test none of them hid it, one builder even reported it to the head himself. now a limit goes up only in all tests at once and only when the head decides keyword search only, so not airtight
1
53
a full weekly reset is basically someone handing me 50 bucks. i burn through a week in 2 to 3 days, so this one really made my day. thanks
We've given all Pro, Max, Ultra, and Team users one free full reset, across all models and limits. Use it anytime before December 10 at 2:00PM PT. We recently fixed a few small issues that used more limits than intended, and the Fable resets didn't do much for those of you who'd already hit your weekly limits for all models. While these issues didn't affect everyone, we want to make up for the trouble. You'll find it in Settings > Usage, or at clau.de/reset.
1
37
one more stage 1 incident i didnt mention before. a release tag went out while main didnt move. now tag and main go as one chain
1
1
1
65
stage 2 is the deploy setup. opus 5.5 ran it from nov 1, then fable 5.2 took over in the same session opus: 11126 turns, 5708 tool calls since nov 1 fable: 985 turns, 498 tool calls in about 13 hours in those 13 hours fable fixed 14 defects at the cause, 9 in the platform and 5 in setup steps. 32 fix commits, 7 platform releases, each fix with a red test first and 2 to 9 mutations it rolled back 2 old workarounds claude code safety said no 4 times, none retried its misses, from its own report: about 7 hours idle, it said i wasnt needed without reading the next step 3 full runs at 35 to 50 min, it skipped a settings note 8 of my 20 messages were nudges 2 stale tests found 10 to 12 hours late not a clean model test. the no workarounds rule landed mid session. services are not up yet
token count for stage 1, same 7.5 days about 19.4 billion tokens in 163811 model responses 97% of that is agents rereading their own cached context every turn. new tokens were 559 million, and only 94 million of those were model output runs vs tokens: haiku did 62% of agent runs and used 5% of the tokens sonnet did 21% of runs and used 37% opus did 17% of runs and used 58% so the waiting, limit checks and status calls cost almost nothing. the spend sits in the checks and the main sessions on opus, and the builds on sonnet main sessions that steer the lines took 30%, the agents 70% teapawt.net/one_dyor/status/211971…
1
1
5
144
about 16 hours before fable 5.2 dropped, opus 5.5 got really dumb in my multi agent runs. high, xhigh, max, nothing helped. now fable 5.2 feels exactly like opus 5.5 did before that, just pricier. no measurements, just my gut teapawt.net/one_dyor/status/211988…
i was waiting for this update. cant wait to put fable 5.2 to the test teapawt.net/claudeai/status/211986…
46
full test run done. 10769 green, 4 red. two of the red were from my own fix the same day. fixed, rechecked, merged
22
i was waiting for this update. cant wait to put fable 5.2 to the test teapawt.net/claudeai/status/211986…
Introducing Claude Fable 5.2 and Claude Mythos 5.2, designed to tackle the most daunting technical challenges. They’re capable collaborators on ambitious projects in many fields, and show improvements in areas like coding, reasoning, and vision.
1
1
102
numbers on the mutant gate from my setup, and what im trying next before the gate, a bit under 35% of first reviews had a surviving mutant. after it, about 1 in 5 the gate kills the builders own mutants fine. what survives is the reviewers own mutants at risky spots, places the builder covered with neither a test nor a mutant. each one is a NO-GO and a whole extra round so now before the first review a sonnet agent reads the task and the changed code and plants up to 5 of its own. same gate, same fixer for the survivors on 20 past reviews sonnet named 15 of the 30 mutants opus later found alive. in 8 to 14 of those tasks the extra round wouldnt have happened, and a round costs me from half an hour to a few hours after 20 new tasks i compare rounds per task and hours to GO. no gain, it goes
2
48
token count for stage 1, same 7.5 days about 19.4 billion tokens in 163811 model responses 97% of that is agents rereading their own cached context every turn. new tokens were 559 million, and only 94 million of those were model output runs vs tokens: haiku did 62% of agent runs and used 5% of the tokens sonnet did 21% of runs and used 37% opus did 17% of runs and used 58% so the waiting, limit checks and status calls cost almost nothing. the spend sits in the checks and the main sessions on opus, and the builds on sonnet main sessions that steer the lines took 30%, the agents 70% teapawt.net/one_dyor/status/211971…
stage 1 is closed. all 11 steps of the plan done, nothing deployed yet. counted over the whole stage this time, so the numbers differ from the last update 7.5 days: 3242 commits, 2394 files changed, 9 platform releases 11818 tests on main. the platform had 151 on nov 1 12633 agent runs in 620 lines. haiku 62%, sonnet 21%, opus 17%, and opus is mostly the checks 838 independent checks, 368 GO. only 34% of units passed on the first one top reason for NO-GO was a surviving mutant, a test that misses a planted bug. it showed up in 261 of 470. the mutant gate from nov 6 goes after exactly that 8 incidents, each one became a rule. one fix broke the test next to it, now neighbor tests run before every fix commit the next stage starts when i say deploy
1
2
177
when you need a solid assistant and not a production machine, put opus 5.5 on max. kinda obvious, but when you hand off routine that actually matters, it makes a difference
3
48
stage 1 is closed. all 11 steps of the plan done, nothing deployed yet. counted over the whole stage this time, so the numbers differ from the last update 7.5 days: 3242 commits, 2394 files changed, 9 platform releases 11818 tests on main. the platform had 151 on nov 1 12633 agent runs in 620 lines. haiku 62%, sonnet 21%, opus 17%, and opus is mostly the checks 838 independent checks, 368 GO. only 34% of units passed on the first one top reason for NO-GO was a surviving mutant, a test that misses a planted bug. it showed up in 261 of 470. the mutant gate from nov 6 goes after exactly that 8 incidents, each one became a rule. one fix broke the test next to it, now neighbor tests run before every fix commit the next stage starts when i say deploy
update: the work ate 98% of my weekly limit on ultra and still isnt done. plugging in the next claude this session: 4 platform releases, 96 units merged, ~1000 files changed, 4873 tests on main, all green merge goes only after a separate check approves. it refused 2 branches, they stayed out found a gap today: 39 deploy tests ran on merge without checking the code. fixed, now all three test sets run
2
127
next update to the setup from the article builder, fixer and review now run one test list a script picks from the changed files. the mutant gate waits for its own runs and doesnt call the fixer when the problem is the environment. after 3 reviews without GO the line stops itself and the orchestrator takes the task mutants of one version run in parallel, each in its own copy. heavy slots on the pc went from 10 to 16 fixed one thing yesterday. out of 2206 runs that day 45 ran past the fast lane and got queued again, mostly the same test sets. now a set like that goes straight to the heavy lane next time orchestrators share the same watcher checks and rules now, and lines can start from any of my 3 devices
1
17
next update to the setup from the article builder, fixer and review now run one test list a script picks from the changed files. the mutant gate waits for its own runs and doesnt call the fixer when the problem is the environment. after 3 reviews without GO the line stops itself and the orchestrator takes the task mutants of one version run in parallel, each in its own copy. heavy slots on the pc went from 10 to 16 fixed one thing yesterday. out of 2206 runs that day 45 ran past the fast lane and got queued again, mostly the same test sets. now a set like that goes straight to the heavy lane next time orchestrators share the same watcher checks and rules now, and lines can start from any of my 3 devices
4847 agents and 5199 test runs in under two days. heres the line that runs them
1
3
84
STOP USING A CALCULATOR TO SOLVE PROBLEMS STOP USUNG GOOGLE TO SEARCH URLS STOP USING A PHONE TO TYPE DOCUMENTS…. STOP USING SOCIAL MEDIA TO SOCIALIZE See how stupid it sounds?
This quoted post is unavailable.
2
90
update: the work ate 98% of my weekly limit on ultra and still isnt done. plugging in the next claude this session: 4 platform releases, 96 units merged, ~1000 files changed, 4873 tests on main, all green merge goes only after a separate check approves. it refused 2 branches, they stayed out found a gap today: 39 deploy tests ran on merge without checking the code. fixed, now all three test sets run
finishing the cloud automation for ******.***, deploy in the next few days 13 days, 2302 commits, almost all co-authored with claude agents. i did the stuff only a human can do: accounts, payments, 2fa, tokens and the decisions main rule: automation does only what can be undone on its own. restart, cleanup, renew a cert, fine. migrations, firewall, creating or deleting servers, only with my passkey one lesson already. on nov 2 the agents spun up 6 staging servers on unready code without asking me now servers come up only after i say deploy
5
1
3
187