Pinned Tweet
Mainstream discourse about agent effectiveness tends to focus on model selection.
We looked at complete agent trajectories across 103 engineering teams and found that prompt clarity, environment readiness, and quality stewardship are heavily correlated with the right outcomes.
Read the report: span.app/research/agent-effe…
Span retweeted
Most diffs won’t get human review within a year, and teams will review specs and agent trajectories instead, predicts @stephenpoletto of @Span_App.
Span retweeted
73% of engineering leaders cite code review as a top pain right now.
I've spent countless hours with frontier AI teams understanding how they're navigating the problem.
Human review of every line of code is quickly getting replaced by new techniques.
somewhere on your team, an engineer has figured out how to get consistently good results from agents
we help you figure out what they're doing right and share it across your org
Span retweeted
@jeffjb walked us through an updated finance setup flow this week at @Span_App, and the part I liked most was how little we were asking the customer to input.
Usually, setting up a tool like this means taking a policy your company already has and translating it into whatever fields the software gives you. You have the answers, but you still spend an afternoon entering them.
We’re designing this flow to start with the customer’s policy docs that already exist, and use AI to draft the categories, rules, and thresholds. Finance can review the draft, see where each suggestion came from, and change anything that doesn’t match how they work.
There’s still plenty for the customer to decide. But interpreting their policy is a much better use of their time than copying it into a form.
If you’re adding AI to a setup flow, I’d look at what you’re asking users to enter that they’ve already written down somewhere else. That’s probably a better starting point than a blank text box asking them to describe what they want.
agent performance depends on much more than the model
span has three evals to help eng orgs improve agent performance:
• prompt clarity: how clear your instructions are
• quality stewardship: how you guide output to meet quality standards
• environment readiness: how well your setup supports agents
before you switch models, check the environment and guidance you’re giving your agents
Span retweeted
One thing I worry about as AI helps us ship faster at Span: we’re giving customers more changes to keep track of, without making that any easier for them.
We’re planning to sunset a feature, which means retiring a few pages customers use today. We can explain it over Slack and email, but someone who misses those announcements could still open a page they use every week with no idea it’s going away.
So we’re starting to put the notice on the affected pages, with a preview of what’s changing, why, and what they need to do to keep their data flowing.
It’s easy to underestimate this work when you’ve spent weeks discussing the change internally. By launch, it feels obvious to you but the customer might be seeing it for the first time between two meetings.
If AI is helping your team release more often, it’s worth checking how much work you’re passing along to customers. Are they having to read more announcements, find new instructions, or figure out where something moved? If so, it's important to use some of the time we save building to make the changes easier to live with.
Your token bill tells finance what you spent. But it rarely tells them what laddered up into the final amount.
Your reporting should answer:
- Which project?
- What activity?
- How was the cost allocated?
We tell you how: span.app/blog/token-accounti…
.@trygenius_ai had doubled engineering output with coding tools in six months.
Then Span surfaced a bottleneck that directly impacted their velocity: a cross-team code review backlog that was slowing the whole pipeline.
The result? A 25% drop in time to first review.
The ledger for your AI software factory.
See how Span works: span.app/
Teams with stronger quality stewardship complete 39% fewer review cycles per PR.
Every extra cycle is time and tokens spent re-reviewing code instead of shipping the work that moves the business forward.
Span helps teams point time and tokens to what matters.
Read our full Q3 report: span.app/research/agent-effe…
Span retweeted
Human review is declining. AI code review scaled more than 6x since Jan 2025. Defect rates did not meaningfully move.
AI reviewer bots now produce 28% of PR review comments across our dataset, up from 5% in January 2025. Today, 96% of companies use at least one AI code review tool, with the median using two.
That growth is happening as review demand outpaces human capacity. Over the last nine months, merged PR volume increased 53%, while the reviewer pool grew only 10%. At the same time, the share of PRs merging with no human review rose from 13% to 19%.
The obvious question is whether that changes software quality.
So far, we do not see evidence in either direction for equally sized PRs. PRs merged with no human review were about as likely to be associated with a bug within seven days as comparable human-reviewed PRs. Mostly AI-written PRs were also about as likely to be associated with a bug as mostly human-written PRs of the same size. The one exception is very large changes: in the largest tenth of PRs, mostly AI-written changes were associated with bugs about twice as often as comparable human-written ones. We cannot yet tell whether that reflects the AI itself or the kind of large change engineers still write by hand.
My interpretation: AI review may be helping teams absorb growing code volume without an obvious decline in near-term quality. But we also do not yet see evidence that more automated review reduces defects, and this analysis does not capture the broader benefits traditionally associated with human review.
One factor had a much stronger relationship with bugs: PR size. Larger PRs had a higher probability of being associated with a bug.
I suspect practices such as smaller changes, stronger testing, type safety, and focused bug bashes remain your most important quality levers.
Span retweeted
Everyone is talking about graph engineering. In simple terms, it means designing how several specialized AI agents hand work off to each other, instead of making one agent do everything in a single loop.
This feels very similar to designing how a team works. Roles and responsibilities, handoffs between functions and teammates.
Once you've designed the graph, you implement different sub-agents to do different "jobs" / parts of the workflow. Writing a PRD or a tech spec probably requires frontier intelligence (Opus 5, etc.). Once the tech spec is broken down into granular tickets, a smaller, cheaper model probably works (Sonnet 5, or Haiku 4.5 for the really mechanical stuff). Time for review? If it requires security review, you probably have a dedicated security review agent again using frontier intelligence, or even better, a fine-tuned security focused model.
The analogy to team design is almost one to one. The PM does the PRD, the Tech Lead does the tech spec, the L1 new grad implements simple tickets, the security team does sensitive security review.
If this is how software is developed in the future, you can pin specific model versions to different sub-agents / nodes in the graph. If you want to, you can run the same task through different "versions" of the graph with different models (A/B test) and eval which hits the cost/quality tradeoff you're comfortable with. Won't this be more powerful than using a model router that has no knowledge of your graph?
Span retweeted
The AI coding debate is focused too much on model selection - just see all the model router launches this week - Ramp, Cursor, etc. We studied 103 engineering teams and found big levers are also on the dev platform and human side:
• +1 pt prompt clarity → ~27% lower token cost per merged AI line
• +1 pt environment readiness → ~88% higher turn yield
• +1 pt quality stewardship → ~39% fewer review cycles
Prompts, environments, and stewardship shape every task and are things you can coach your team to improve today.
Full report + a practical field guide below. Would love to hear what matches or contradicts your experience.
span.app/research/agent-effe…
Span retweeted
Two people joined @Span_App last week and by the end of their first week, they'd both had something that they worked on ship to production.
They were a little unsettled by it. Both came from 300+ person companies, where the norm is three to six months before anything ships.
Here's why I like to operate this way: spend a month building before it's in front of anyone, and you might pivot before you ever learn whether it had traction. That's a month you can't get back.
Ship a rough version instead. If there's traction, expand it. If not, you've lost days, not months.
A big company can afford slow bets, a startup can't.
Span retweeted
What does our SVP of Engineering @IcchaSethi have to say about maturity frameworks, evaluation processes, and more? 🤔
Find out on the latest @Span_app episode here: bit.ly/4gFvTvR