Arnon Rotem-Gal-Oz retweeted
Hi everyone,
The Specula project has been renamed to Specular. The new repo is hosted at github.com/specular-org/Spec… and the old one is no longer used.
I created the original name by fusing "Spec" and "Spectacular". Unfortunately, I didn't know it was also a real word (the plural of "speculum") and carried a meaning and its associations I hadn't intended. I apologize for making anyone uncomfortable.
We hope everyone likes the new name. It captures the original portmanteau and carries a connotation of lighting. We hope Specular could shine a light on subtle bugs in your code.
The project has received overwhelming love and support. THANK YOU! We'll continue working hard to improve it and enrich it. @qiancheng9788 just rolled out a new release (v1.5.0).
Special thanks to @tlipcon for communicating the issue and coining the new name!
Arnon Rotem-Gal-Oz retweeted
Wrote about our blob storage we built over SQLite zerodha.tech/blog/high-perfo…
Have been hard at work figuring out how to make replicated state machines both simple and scalable.
The result allows a single Enroute cluster to scale up to 300,000 repos, with more to come.
Check out our write up!
enroute.sh/blog/scaling-hori…
Arnon Rotem-Gal-Oz retweeted
New blog post, with an ice-cold take on some of the last week's systems news: brooker.co.za/blog/2026/11/1…
The throughput of real-world systems can vary (a lot!) based on subtle properties of the workload they're handling.
Load tests that don't match the workload are useless.
Arnon Rotem-Gal-Oz retweeted
quick tip we learned the hard way @ rivet:
jev does not reason
...which is by design
...and we should have rtfm first
~
we started by prompt-engineer to ask something like:
```
𝚜𝚎𝚟𝚎𝚛𝚒𝚝𝚢 = 𝚊𝚜𝚔(
"𝚐𝚒𝚟𝚎 𝚖𝚎 𝚝𝚑𝚎 𝚜𝚎𝚟𝚎𝚛𝚒𝚝𝚢 𝚘𝚏 𝚝𝚑𝚒𝚜 𝚒𝚜𝚜𝚞𝚎. 𝚏𝚘𝚛 𝚎𝚡𝚊𝚖𝚙𝚕𝚎, 𝚜𝚎𝚟-𝟷 𝚒𝚜 𝚇, 𝚜𝚎𝚟-𝟸 𝚒𝚜 𝚈, 𝚜𝚎𝚟-𝟹 𝚒𝚜 𝚉, 𝚎𝚝𝚌",
𝚘𝚞𝚝𝚙𝚞𝚝𝚜=[𝚂𝙴𝚅𝟷, 𝚂𝙴𝚅𝟸, 𝚂𝙴𝚅𝟹, 𝚂𝙴𝚅𝟺]
)
𝚛𝚎𝚝𝚞𝚛𝚗 𝚜𝚎𝚟𝚎𝚛𝚒𝚝𝚢
```
normally, an llm would spend reasoning tokens to come to a descent conclusion
but doing this with jev gives meh results, and defeats what makes jev awesome
~
after rtfm, we split it up in to tiny prompts like this:
```
𝚒𝚜_𝚍𝚘𝚠𝚗 = 𝚊𝚜𝚔("𝙳𝚒𝚍 𝚝𝚑𝚎𝚒𝚛 𝚙𝚛𝚘𝚍𝚞𝚌𝚝𝚒𝚘𝚗 𝚜𝚎𝚛𝚟𝚒𝚌𝚎 𝚐𝚘 𝚍𝚘𝚠𝚗 𝚘𝚛 𝚋𝚎𝚌𝚘𝚖𝚎 𝚞𝚗𝚞𝚜𝚊𝚋𝚕𝚎?")
𝚒𝚜_𝚋𝚕𝚘𝚌𝚔𝚎𝚍 = 𝚊𝚜𝚔("𝙰𝚛𝚎 𝚝𝚑𝚎𝚢 𝚋𝚕𝚘𝚌𝚔𝚎𝚍 𝚠𝚒𝚝𝚑 𝚗𝚘 𝚠𝚘𝚛𝚔𝚊𝚛𝚘𝚞𝚗𝚍?")
𝚒𝚜_𝚋𝚛𝚘𝚔𝚎𝚗 = 𝚊𝚜𝚔("𝙸𝚜 𝚜𝚘𝚖𝚎𝚝𝚑𝚒𝚗𝚐 𝚒𝚗 𝚁𝚒𝚟𝚎𝚝 𝚋𝚛𝚘𝚔𝚎𝚗 𝚘𝚛 𝚏𝚊𝚒𝚕𝚒𝚗𝚐?")
𝚒𝚏 (𝚒𝚜_𝚋𝚕𝚘𝚌𝚔𝚎𝚍) 𝚛𝚎𝚝𝚞𝚛𝚗 𝚂𝙴𝚅𝟷
𝚒𝚏 (𝚒𝚜_𝚍𝚘𝚠𝚗) 𝚛𝚎𝚝𝚞𝚛𝚗 𝚂𝙴𝚅𝟸
𝚒𝚏 (𝚒𝚜_𝚋𝚛𝚘𝚔𝚎𝚗) 𝚛𝚎𝚝𝚞𝚛𝚗 𝚂𝙴𝚅𝟹
𝚛𝚎𝚝𝚞𝚛𝚗 𝚂𝙴𝚅𝟺
```
which works like a charm
(example screenshot of real-world classifier below)
~
any time we find a ticket that isn't weighted well:
- we give the ticket to an llm with what we expected
- llm adds it to a test suite
- llm adjusts decision logic
- backtests new decision logic on old issues
within a few tickets the classifier logic was really solid
~
this is a feature of jev, not a limitation
in practice, it gives more control over how each decision is weighed than a generic llm
which makes it easier to reason about, monitor, and back test how agents will behave
~
for reference, this is how many jev questions we ask for a few of our classifiers in our internal support tool:
support classification and customer status: 13
severity: 6
team replies and problem identification: 6
follow-up timing: 6
linked dependencies and later conversations: 4
addressed teammate: 2
conversation grouping: 2
~
all that to say, jev is awesome
Arnon Rotem-Gal-Oz retweeted
Autoscaling research is hot, but everyone's solving the *easy* problem of stateless scaling.
Result: Very few production deployments!
Sorry State of Stateful Scaling.
Our vision paper in ACM Symp. #Cloud Computing.
Paper: tinyurl.com/StateScale-paper
Talk: youtu.be/8mlpC4bWJqY
Arnon Rotem-Gal-Oz retweeted
1/ את harmony.ai התחלנו לבנות בשיחות מכירה.
היום זו הדרך היחידה למכור לאנטרפרייז.
יותר מ-50 שיחות עם לקוחות אנטרפרייז, בחלק מהן כבר עם גרסאות שבנינו ב-vibe coding.
מה שלמדתי שם שינה לי את הדרך שבה אני חושב על פיתוח מוצר
Arnon Rotem-Gal-Oz retweeted
🧵 O Kubernetes foi desenhado para pods que ficam ligados o tempo todo (long-lived). E agentes de IA fazem o contrário... trabalham alguns segundos e ficam 90~95% do tempo ociosos esperando o LLM / uma tool ou uma aprovação humana. E na prática isso significa cluster pagando por CPU e memória ociosas quase o tempo inteiro...
Arnon Rotem-Gal-Oz retweeted
I just finished talks about my software factories at two conferences. Afterwards a bunch of people came up and told me what they were surprised about. Here is a distillation of what people thought were hot takes from my two talks.
First hot take: I tell everyone to slow down. Most speakers go on about how fast they're going, but I strongly believe in pacing AI outputs to human speeds. I regularly tell AI-pilled CIOs and CTOs to chill: almost nobody is _actually_ going fast with AI. Engineering still moves at engineering speeds, and business moves even slower. The ones pushing too hard are starting to see downtime for it.
If you don't pace your AI to human speeds, everything starts breaking around you, even at small operations like my online video game. I only have about a dozen humans working with my factory, and only a few hundred active old-timer players in the community who get along well, and still, introducing AI has still been very rocky.
My agents have, at times, overwhelmed my players with features, my accountant with emails, my merge queue with tickets, and my machines with load. Agents have pissed off my admins, resulting in name-calling, they've made messes in the game, and they've launched broken OSS without permission. The list goes on.
I can get away with all this chaos because everyone involved in my operation is very forgiving. They're excited by the potential. But even so, I had to learn to slow down, real fast. And so that's what I preach.
Hot take #2: Should you buy a software factory? Do you even need one? No. You probably don't need one. Factories take an extraordinary amount of upkeep. They turn into garbage mountains unless you put a frightening amount of time and tokens into curating them. I had to burn my previous five factories down since it eventually became cheaper to rewrite them than to fix them. So I'm not out there advocating for software factories or swarms just yet. If you do want one, be aware that it is a significant commitment of both money and human leadership to keep one running.
Several people told me that I was the only one who stood up onstage, at either conference, and said Build, don't Buy. But my experience is that software factories grow to be highly bespoke to your domain. Mine is super tangled up with my game, and I think that's inevitable. Because the factory is not just building my product, it's supporting it too. It's a company in a box. And they're easy for Claude to build for you.
Third and arguably hottest take: Next year will have a lot of human/agent and agent/agent office drama, resulting from bringing in AI employees, and it will be gloriously funny, but also frustrating.
Many of you are already on this bandwagon, but everyone next year will have a personal chief-of-staff agent. Busy people, even hobbyists, will have whole entourages of helper agents. The hot take is that people will become so dependent on these little private entourages, that they'll bring them to work. Your people will talk to my people, etc. Whole protocols will emerge for private entourages negotiating work between humans, inside companies big and small.
Nearly all humans will wind up having to start working with AI employees next year, and it'll be the beginning of the biggest cultural change we've seen in a hundred years.
So that's the hot takes. Slow down. Only build a factory if you think you really need one. And focus on empowering yourself with your own agentic chief of staff (or team). Your entourage will define how you work with others, so start by perfecting how you work with them yourself.
Arnon Rotem-Gal-Oz retweeted
Revolut shared the postmortem of their global cards outage of 5 hours. Some interesting (alarming?) notes:
1. They had no monitoring on database nodes going down (!!)
2. Across 3 regions, they ran 5 nodes, with #1. So a single node going down meant no zonal resillience (cont'd)
Arnon Rotem-Gal-Oz retweeted
A Trillion transactions
tigerbeetle.com/blog/2026-03…
Arnon Rotem-Gal-Oz retweeted
Uber built Shard Manager to solve one of distributed systems’ classic problems: shard balancing at scale.
uber.com/us/en/blog/introduc…
Arnon Rotem-Gal-Oz retweeted
We’ve published an update on our recent security incident. See a detailed overview of what happened and our remediation steps here: cursor.com/blog/security-inc…
Arnon Rotem-Gal-Oz retweeted
OpenOrange vs celld
34.8x lower S3 costs for agent workloads
This is made possible by building OpenOrange on Rivet's storage engine. We've spent years scaling the actor model, and Rivet's storage engine was built for this.
celld's architecture balloons costs for:
- Agents
- Workflows
- Background jobs
- Session stores
~
Why?
First, a primer on running databases on S3.
S3 is cheap for storing data. It's about 10x cheaper than AWS EBS replicated across 3 availability zones:
$0.023 vs $0.24 per GB per month
But it gets expensive once you actually operate on that data. Writing one object every second for a month costs:
AWS EBS ~$0
S3 $13
This is because S3 charges per operation, while AWS EBS charges for how much you store and for how long:
PUT costs $5 per million
LIST costs $5 per million
COPY costs $5 per million
GET costs $0.40 per million
On AWS EBS, these operations are effectively free.
The catch is that the number of operations scales with how many objects you have. Every object has to be written with its own PUT, so 1,000 objects cost 1,000 PUTs. Pack the same data into one object and it costs one.
~
So why is celld so expensive?
celld stores each Durable Object as its own S3 objects. So the more Durable Objects you create, the more objects you have, and the more you pay.
Every new Durable Object costs PUTs to its own objects: claiming it, then saving it when it goes to sleep. That's ~4 PUTs and 1 GET per Durable Object in our benchmark.
If you have a fixed number of Durable Objects, this cost is reasonable. But if you're constantly creating new ones, your cost grows with every one.
And agents tend to be high churn. Some agents work for days, but most do a quick, discrete task and never get used again.
These types of workloads suffer, since they frequently create new Durable Objects:
- Agents
- Workflows
- Background jobs
- Multiplayer game rooms
- Chat rooms
- Per-user sessions
- Rate limiting
~
So how does OpenOrange stay so cheap?
Rivet's storage engine is built on an LSM tree (log-structured merge tree). It's the same idea behind mature S3-backed databases like ClickHouse, used to keep performance high and cost low.
Instead of giving each Durable Object (called Actors on Rivet) its own S3 objects, Rivet's LSM packs many of them into shared S3 objects.
Here's how PUTs compare per 1,000 new Durable Objects:
- OpenOrange ~90
- celld ~4,000
When you create a Durable Object or write to SQLite on OpenOrange:
1. The write is replicated to SSDs across 3 availability zones (single-digit ms)
2. After about a minute, writes from many Durable Objects are merged into one batch
3. That batch is written to S3 as one object
~
These benchmarks are modeled on real-world agent workloads, measured at 8 to 25,000 concurrent agents. Each agent lives 60 seconds and writes a turn every 10 seconds. We only measured S3 cost, and both ran on identical hardware.
~
OpenOrange is open source today.
We've attached the benchmarks below.
Arnon Rotem-Gal-Oz retweeted
How Top AI Teams Scale to 1,000+ Agents blog.dailydoseofds.com/p/how…
Arnon Rotem-Gal-Oz retweeted
so I pissed off temporal's ceo a few months ago by making the argument that old-school durable execution is much less important than it used to be, or more specifically, is very architecturally lacking.
this is really a conversation we've been having internally for the past 6 months, because we've been trying to reconcile how to model the two different aspects of durable state that agents need: filesystem state and session state. these are distinct and if you're building an agentic application I think you need to understand the difference (which I talk about in the article).
it turns out that the biggest innovations in filesystem durability are coming out of the sandbox providers like @archildata @e2b @modal @freestyle_dev , to name a few. it's easy to have sandbox-provider-fatigue if you view them as compute resellers, but if you cut through the noise, some of them are doing super interesting work on the filesystem durability layer (and the isolation layer of course).
I go into more detail in the article, but there's a correct way to use these providers in conjunction with durable execution. that's why we released the @hatchet_dev sandbox package earlier this week and why we're integrating with really great sandbox providers. full article:
hatchet.run/blog/agent-state…
Arnon Rotem-Gal-Oz retweeted
Why Nvidia gave OpenShell to the CNCF thenewstack.io/nvidia-donate…
Arnon Rotem-Gal-Oz retweeted
Choosing QuePaxa, an unusual consensus algorithm blog.cloudflare.com/choosing…
Arnon Rotem-Gal-Oz retweeted
AWS 刚刚开源了一个叫 UAEF 的 agent 评测系统。
这篇文章探讨的核心其实是一个很实际的工程问题就是当团队在不停重构 agent 时,怎么建立一套能够万用的一套 agent 的评估框架体系。
我认为这篇文章最好的地方就是他的思路非常不错的,不论是对 agent 开发还是对于团队中 agent 的维护都很有帮助。
他们的思路是先在最前面加一层转换器,这个转换器不管是单步问答、多轮对话还是多智能体协作,先把消息、工具调用和交接记录全部转成统一的标准轨迹。
这样后面的打分评测引擎就彻底不用管 Agent 是用什么框架写的了。
拿到统一轨迹后,文章专门强调不能上来就把所有指标全跑一遍,比如订机票的 Agent 和查公司制度的问答 Agent 的评测的方式是完全不同的,应该按场景去选择评测。
另外他们强调在打分时,能用代码直接比对的绝不用大模型。
必须用大模型当裁判时,为了防止被测 Agent 在输出里夹带指令去骗高分,系统把评分标准和被测内容分开发送并对内容做转义隔离,而且强制要求裁判打分时必须原封不动地摘抄出原文依据,如果在原文里核对不到这句话,这个打分直接作废。
最后一步是怎么把跑出来的分数变成下一步修改的依据。现在很多的评测跑完只给一个平均分,平均分涨了可能掩盖了某个关键功能的倒退。
UAEF 要求每次跑测都绑定当时的配置和 Git 提交记录,跟基线版本逐项对比,只要某项指标跌了超过 5% 就会报告出来。
这样其实很好的防止了前面说到的这个平均分涨了可能掩盖了某个关键功能的倒退的问题。这一向是目前 agent 改进优化的通病。
更有意思的是它在出报告时做了两层拆分,第一层统计报告完全不用大模型,纯靠数学计算生成分布,保证同样的数据每次跑出来绝对一致。
第二层是利用大模型,让它去推演为什么这次改完出现了哪些哪些的问题,然后让 ai 给出这个下一轮怎么改的具体假设。
github:github.com/awslabs/uaef
原文:aws.amazon.com/cn/blogs/mach…
Arnon Rotem-Gal-Oz retweeted
ניסיתי לקצר, באמת שניסיתי, אבל לא הצלחתי. הקלטתי את כל הסיפור שיישאר כאן כעדות, מאלף עד תו, כי אירוע כזה עוד לא היה בישראל:
ראש שירות הבטחון הכללי, דוד זיני, מוביל בכבודו ובעצמו קונספירציה מטורפת שכל מטרתה היא להוכיח שבחירות 2026 זויפו!
תקראו את זה שוב: האיש שאחראי לשמור על הדמוקרטיה הישראלית יוצא למסע קונספירטיבי כדי לרסק את האמון בה!
סיכום של שבועיים שלקוחים מהמשטרים הכי אפלים בספר. דויד זיני לא יכול להישאר ראש השב״כ לעוד יום אפילו






















