⚡🛡️ Evan Pappas retweeted
We are cooking some pretty interesting products around post training and inference @tplr_ai .
Would love to talk to engineers and researchers to understand where you bottlenecks are .
The rewards are my eternal gratitude and some credits.
DM if you're up for it.
Crucible, Templar's pre-training platform, has completed its first production end-to-end training runs. The latest trained an 8B model on 50.53B tokens across 48 distributed A100s, at an estimated $0.1202 per million tokens of GPU rental.
The run reached 48.3% effective MFU. At AWS p4de Capacity Blocks pricing, a 48-A100 cluster operating at the literature-derived 65% compute ceiling comes to an estimated $0.1686 per million tokens. Crucible's measured $0.1202 was about 29% lower after its low-bandwidth overhead.
The comparison excludes R2 storage and operations. The full writeup shows the method and accounting.
Blog: tplr.ai/publications/blog/in…
⚡🛡️ Evan Pappas retweeted
Introduction to LLM Design & Covenant 72B: This week at the Ethereum & AI Engineering Group meetup I provided a 15 minute introduction to LLM design and then dived into the design and training of Covenant 72B; the largest model trained over the Internet.
youtube.com/watch?v=TlVQbiTd…
⚡🛡️ Evan Pappas retweeted
[1/9] Edinburgh Decentralised Finance Summit 2026.
Tuesday 1 September, Royal College of Physicians of Edinburgh.
A full day on institutional DeFi, tokenisation, risk management, regulation and autonomous finance.
Around 25 speakers. 300 attendees.
luma.com/scotland-kgav
⚡🛡️ Evan Pappas retweeted
One directive, one afternoon, and global access to Anthropic's most capable models was gone. Sovereignty does not end at nations. The tools people think with should not sit behind one party's switch. This is why Covenant builds open, distributed AI infrastructure.
The US government, citing national security authorities, has issued an export control directive to suspend all access to Fable 5 and Mythos 5 by any foreign national, whether inside or outside the United States, including foreign national Anthropic employees.
The net effect of this order is that we must abruptly disable Fable 5 and Mythos 5 for all our customers to ensure compliance.
Access to all other Claude models is not affected.
We apologize for this disruption to our customers. We believe this is a misunderstanding and are working to restore access as soon as possible.
Read our full statement: anthropic.com/news/fable-myt…
⚡🛡️ Evan Pappas retweeted
The @cursor_ai team post-trained Composer 2 on an open-weight base model using @FireworksAI_HQ's distributed RL rollout infrastructure.
Fireworks links PULSE as the theory behind the BF16 sparsity that makes compact weight updates practical.
⚡🛡️ Evan Pappas retweeted
The HF science team just made async RL weight sync ~100x cheaper on bandwidth, and you don't need a shared cluster anymore.
The problem: every RL step, the trainer typically has to sync fresh weights to the inference engine. for a 7B in bf16 that's ~14GB. for a frontier 1T fp8 checkpoint, that's ~1TB; in bf16 it would be ~2TB. per sync.
The insight: between two RL steps, ~99% of bf16 weights are bit-identical. at RL learning rates, the optimizer is whispering and bf16 literally cannot hear most of it. the stored bf16 bits don't change.
What they shipped in TRL: only the changed elements get encoded as a sparse safetensors file, dropped into a Hugging Face Bucket, and fetched by vLLM. on Qwen3-0.6B, per-step payload goes from 1.2 GB to 20 to 35 MB. This is exactly what we built Buckets for: S3-like object storage on the Hub, Xet-backed (so even full snapshots only transfer the changed chunks).
The cherry on top: we ran a FULL disaggregated training where:
- the trainer lived on one box
- vLLM ran inside a Hugging Face Space
- the Wordle environment ran in another Space
- weights flowed through one Hub bucket
no shared cluster. no RDMA. no VPN. no NCCL across clouds. just HTTPS and a bucket.
one GPU + a Hugging Face account is now enough to do real disaggregated RL. multi-replica inference fleets across regions become a small devops exercise, not a research project.
Full write-up: huggingface.co/blog/delta-we…
Open source RL keeps eating the moat!
⚡🛡️ Evan Pappas retweeted
Phil Windley (@windley), from his OpenClaw integration:
"If the action is denied, the loop does not terminate. The denial is returned to the agent as a structured result. Denial becomes a productive signal that feeds back into planning."
The deny is API now, not error.
⚡🛡️ Evan Pappas retweeted
An LLM request runs two phases. Prefill reads the whole prompt and scales with its length squared. Decode emits one token at a time.
Prefill is where the money goes, and most of it is recompute.
⚡🛡️ Evan Pappas retweeted
Probably nothing @tplr_ai @covenant_ai
⚡🛡️ Evan Pappas retweeted
Three production deployments in four months, each one strengthening the next. @FireworksAI_HQ cited PULSE for Cursor's Composer 2 training in March. Now @huggingface and @vllm_project are shipping it.
Open research compounds when adopters do the work of proving it at scale.
The HF science team just made async RL weight sync ~100x cheaper on bandwidth, and you don't need a shared cluster anymore.
The problem: every RL step, the trainer typically has to sync fresh weights to the inference engine. for a 7B in bf16 that's ~14GB. for a frontier 1T fp8 checkpoint, that's ~1TB; in bf16 it would be ~2TB. per sync.
The insight: between two RL steps, ~99% of bf16 weights are bit-identical. at RL learning rates, the optimizer is whispering and bf16 literally cannot hear most of it. the stored bf16 bits don't change.
What they shipped in TRL: only the changed elements get encoded as a sparse safetensors file, dropped into a Hugging Face Bucket, and fetched by vLLM. on Qwen3-0.6B, per-step payload goes from 1.2 GB to 20 to 35 MB. This is exactly what we built Buckets for: S3-like object storage on the Hub, Xet-backed (so even full snapshots only transfer the changed chunks).
The cherry on top: we ran a FULL disaggregated training where:
- the trainer lived on one box
- vLLM ran inside a Hugging Face Space
- the Wordle environment ran in another Space
- weights flowed through one Hub bucket
no shared cluster. no RDMA. no VPN. no NCCL across clouds. just HTTPS and a bucket.
one GPU + a Hugging Face account is now enough to do real disaggregated RL. multi-replica inference fleets across regions become a small devops exercise, not a research project.
Full write-up: huggingface.co/blog/delta-we…
Open source RL keeps eating the moat!
⚡🛡️ Evan Pappas retweeted
Today we're releasing PULSELoCo: over 100x lower trainer-to-trainer communication for distributed RL post-training.
Paired with PULSESync, every node can sit anywhere in the world. Geo-distributed RL post-training over commodity links, no datacenter interconnect needed.
1/n
⚡🛡️ Evan Pappas retweeted
The DiLoCo team at Google DeepMind and Google Research is proud to release Decoupled DiLoCo, the next frontier for resilient AI pre-training.
Decoupled DiLoCo enables training with datacenters across the world, using heterogeneous hardware, and never halting the system despite hardware failures.
This didn't receive the attention it deserved. They pre-trained this model completely peer 2 peer, no data-centers.
Everything was done over a permissionless network, I have tried the model, it's honestly not a good LLM but that's beyond the point.
We NEED this, we NEED an alternative.
- Download OpenCode
- Download Pi
- Pay for OpenSource
- Share your AI sessions
- Learn to do RL
We can't be at the mercy of ANY lab.
arxiv.org/abs/2603.08163
⚡🛡️ Evan Pappas retweeted
TGIF #29 recap is up. @Hevalon talks about Basilica becoming agentic-native compute infrastructure and @DistStateAndMe gets into the economics of running a compute network without relying on token arbitrage.
youtu.be/Etz9KLqRjHc
TGIF #29 recap is up. @DistStateAndMe explains the importance of Heterogeneous SparseLoCo, why no one has matched permissionless training at this scale even a year later, and the full value chain model that's about to come through the @covenant_ai pipeline.
youtu.be/Etz9KLqRjHc
⚡🛡️ Evan Pappas retweeted
All Covenant support and discussion now lives in one place. The Covenant Discord.
@tplr_ai, @basilic_ai, @grail_ai. The whole team, one server.
Come build with us.
discord.gg/QvXEtjNcah






















