Open infrastructure for building cloud software factories

Joined June 2021
Introducing Scorers: Agents that grade your agents. Use LLM-as-a-judge to grade past coding agent sessions on: • Quality • Efficiency • Compliance • Or any custom dimension Scores feed into performance measurements and automatic self-improvement for software factories
19
14
5
204
572,806
Reposting from our CEO:
On Friday, we became aware of what we believe to be an active and narrow-banded exploitation of a vendor in Warp's infrastructure. We detected the operation within hours and were able to remediate and mitigate the impact. We have been in touch with the vendor about the vulnerability. Everyone who uses Warp places a lot of trust in us and that’s not something we take lightly. I’m very sorry that this incident has occurred. The attack appears to have come from an agent swarm using a very sophisticated penetration and escalation technique, likely starting with a hole in a vendor dependency, and ultimately accessing a limited amount of agent trace and telemetry data. We have been working over the weekend and we have notified anyone who we have positively identified as impacted. As of today, we believe an upper bound of roughly 1% of Warp users may have been impacted. We are working closely with the team at Crowdstrike to do a more extensive and comprehensive investigation and have engaged law enforcement. Anyone who has been impacted will receive instructions from us on action they need to take. We will continue to post updates as we learn more.
1
1
8
4,825
On Friday, we became aware of what we believe to be an active and narrow-banded exploitation of a vendor in Warp's infrastructure. We detected the operation within hours and were able to remediate and mitigate the impact. We have been in touch with the vendor about the vulnerability. Everyone who uses Warp places a lot of trust in us and that’s not something we take lightly. I’m very sorry that this incident has occurred. The attack appears to have come from an agent swarm using a very sophisticated penetration and escalation technique, likely starting with a hole in a vendor dependency, and ultimately accessing a limited amount of agent trace and telemetry data. We have been working over the weekend and we have notified anyone who we have positively identified as impacted. As of today, we believe an upper bound of roughly 1% of Warp users may have been impacted. We are working closely with the team at Crowdstrike to do a more extensive and comprehensive investigation and have engaged law enforcement. Anyone who has been impacted will receive instructions from us on action they need to take. We will continue to post updates as we learn more.
6
8
7
88
19,231
Want better quality output from your agent? Give it a way to verify its own work. Love what Warp is doing here: every agent gets a computer and brings back a video of what it did. I do something similar in my workflow - on top of E2E tests, Claude gives me recorded videos + screenshots as part of the evidence work is "done". My bet is verification becomes one of the most important problems to work on this year.
verification is the biggest bottleneck for coding agents right now according to @addyosmani. we @warpdotdev have put a lot of thought into building a good computer-use tool for Warp Factories. every agents gets a computer to navigate the apps they built, take screenshots and recordings, and post them back for human review. 3-4mins of activity get edited down to 15 clips stitched together with cursor clicks, mouse animations, and key presses. we've been able to dramatically cut our time to merge and unlocked faster iteration + review cycles, especially for projects that are hard to mock out locally!
95
47
9
648
66,014
Stop asking agents for text summaries of a PR. They're capable of so much more: - Architecture diagrams to visually explain the change. Could be ascii, mermaid, HTML, or animated walkthroughs. /show-me is a great starting point. - Screen recordings verifying the change end-to-end. Most harnesses can do this now with Chrome DevTools or computer use. Warp Factories will auto-edit video and highlight cursor clicks as well. - Planning documents when humans were in the loop, checked in as code. This is optional, but a product spec explaining the purpose and expected user flow will a) guide the agent to better screen recording verification, and b) help the reviewer when looking at another engineer's work. Warp Factories shows all artifacts connected to an agent session to make these easy to find.
35
15
2
354
52,164
If an agent's PR doesn't have a screenshot verification, it's not worth reviewing. Add this to your PR requirements:
verification is the biggest bottleneck for coding agents right now according to @addyosmani. we @warpdotdev have put a lot of thought into building a good computer-use tool for Warp Factories. every agents gets a computer to navigate the apps they built, take screenshots and recordings, and post them back for human review. 3-4mins of activity get edited down to 15 clips stitched together with cursor clicks, mouse animations, and key presses. we've been able to dramatically cut our time to merge and unlocked faster iteration + review cycles, especially for projects that are hard to mock out locally!
3
1
19
3,277
verification is the biggest bottleneck for coding agents right now according to @addyosmani. we @warpdotdev have put a lot of thought into building a good computer-use tool for Warp Factories. every agents gets a computer to navigate the apps they built, take screenshots and recordings, and post them back for human review. 3-4mins of activity get edited down to 15 clips stitched together with cursor clicks, mouse animations, and key presses. we've been able to dramatically cut our time to merge and unlocked faster iteration + review cycles, especially for projects that are hard to mock out locally!
24
11
7
206
81,325
If you're setting up a software factory, use this configuration format:
Software factories should be configured as code. With Warp, your factory lives in a repo as markdown and YAML files, so your agents, automations, runners, benchmarks, and even custom model routers are all in one place. Here, I walk through a typical Warp factory config and show how it lays the foundation for self improvement! It's pretty awesome DX.
8
2,256
We made a skill for assessing the risk of merging a PR: /merge-risk This addresses the code review bottleneck by helping devs trust the agent for lower risk changes. It grades risk across 5 categories: alignment, impact, verification, rollback cost, and error detection. - Alignment: does code follow the PR description and related tickets? - Impact: what is the blast radius, and does it touch sensitive code paths? - Verification: does the change include end-to-end verification? This means computer use, screen recordings, before/after screenshots... going beyond unit test coverage. - Rollback cost: once it's shipped, how trivial is it to revert? @mattpocockuk calls this a "two-way door" vs. a "one-way door." - Error detection: are errors reported, logged, or simply swallowed? Try it out: > npx skills add warpdotdev/common-skills --skill merge-risk
40
36
7
920
93,304
The real reason for software factories? Bringing together all of your conversation data. Don't leave it locked away on your laptop.
We should stop keeping all of our agent conversations local. They should be stored in a shared place. This lets you do some powerful things: - Schedule self-improvement agents to review runs and pull out failure modes worth fixing - Replay past conversations against different models for benchmarking I met a team of devs zipping up Claude and Codex sessions from their computers and shipping them to a shared S3 bucket. That's... a start 😄 But really, all agent sessions should happen on a single source (like a cloud runner) so that conversation traces are stored together by default. It's why Warp Factories uses cloud sandboxes and makes sessions available via API. We also added an inbox view to see which conversations need your attention, which is quite convenient:
2
3
2,119
Three months since adopting software factories, we’re at 40% more PRs merged and 67% lower cost per PR. We got there by measuring agent quality, benchmarking models, and using self-improvement loops to make our factory better. Here's how you can do this too in Warp Factories:
7
7
2
69
265,146
Warp retweeted
Building Evals on Production Data: 🦄 AI That Works #79 xitter.cf/i/broadcasts/1oJMvNZWv…
7
8
3
107
18,366
At @aiDotEngineer, I presented our skill called /skill-doctor. It's a skill to improve your agent skills. It aggregates your past agent sessions, grades their quality, and propose improvements to fix common failures.
12
10
117
6,210
Great introduction to software factories if you've been wondering how they work
Here's my take on what a software factory really is This is my talk from @aiDotEngineer NYC, and I decided to record a version for everyone to watch on X. tl;dr - software engineering is becoming factory engineering, and there's new work (especially factory benchmarking) that engineers should start focusing on.
1
8
2,501
Getting software factories working is going to be one of the more interesting software engineering challenges in the next few years. Software engineering is becoming factory engineering. Organizations that can make their factories robust, dependable, and self-improving will be able to ship more at better cost and have a competitive advantage.
16
7
1
150
6,652
If you're interested in software factories, but you're not sure where to start, this is a good read:
Software factories can be a real culture shift. The biggest change for us was moving from individual agents on everyone's machines to one central agent setup (a factory) that we can build as a team. I wrote some learnings on how to navigate that.
1
2
40
4,834
Until next time @aiDotEngineer!
We benchmarked some agent orchestration pipelines at @aiDotEngineer. Which would you pick?
1
5
2,198
An interesting engineering challenge this next year will be closing the development loop we've started. We have small loops already (code review, automatic CI fixes, computer use verification), but we need to close the larger loop: let agents suggest what to build next.
I've been thinking about what a true, closed-loop software factory will look like. One where agents know what to fix, what costs to optimize, and the big one: what features to actually build. We're not there yet. But maybe we're getting close...
5
2
40
4,399
Software factories can be a real culture shift. The biggest change for us was moving from individual agents on everyone's machines to one central agent setup (a factory) that we can build as a team. I wrote some learnings on how to navigate that.
40
18
4
261
20,417
If you're building a software factory, this may be your north star:
I've been thinking about what a true, closed-loop software factory will look like. One where agents know what to fix, what costs to optimize, and the big one: what features to actually build. We're not there yet. But maybe we're getting close...
4
11
175
22,899