One covenant, many orders. @tplr_ai + @basilic_ai + @grail_ai
Joined August 2025
- Tweets216
- Following4
- Followers5.1K
- Likes129
Pinned Tweet
Three production deployments in four months, each one strengthening the next. @FireworksAI_HQ cited PULSE for Cursor's Composer 2 training in March. Now @huggingface and @vllm_project are shipping it.
Open research compounds when adopters do the work of proving it at scale.
The HF science team just made async RL weight sync ~100x cheaper on bandwidth, and you don't need a shared cluster anymore.
The problem: every RL step, the trainer typically has to sync fresh weights to the inference engine. for a 7B in bf16 that's ~14GB. for a frontier 1T fp8 checkpoint, that's ~1TB; in bf16 it would be ~2TB. per sync.
The insight: between two RL steps, ~99% of bf16 weights are bit-identical. at RL learning rates, the optimizer is whispering and bf16 literally cannot hear most of it. the stored bf16 bits don't change.
What they shipped in TRL: only the changed elements get encoded as a sparse safetensors file, dropped into a Hugging Face Bucket, and fetched by vLLM. on Qwen3-0.6B, per-step payload goes from 1.2 GB to 20 to 35 MB. This is exactly what we built Buckets for: S3-like object storage on the Hub, Xet-backed (so even full snapshots only transfer the changed chunks).
The cherry on top: we ran a FULL disaggregated training where:
- the trainer lived on one box
- vLLM ran inside a Hugging Face Space
- the Wordle environment ran in another Space
- weights flowed through one Hub bucket
no shared cluster. no RDMA. no VPN. no NCCL across clouds. just HTTPS and a bucket.
one GPU + a Hugging Face account is now enough to do real disaggregated RL. multi-replica inference fleets across regions become a small devops exercise, not a research project.
Full write-up: huggingface.co/blog/delta-we…
Open source RL keeps eating the moat!
covenant retweeted
We’ve been researching fault tolerance in Crucible, Templar’s pre-training platform.
The goal: keep training through node failures and make better use of unreliable workers and spot instances, without idling an entire model replica when one stage goes down.
1/n
covenant retweeted
Crucible, Templar's pre-training platform, has completed its first production end-to-end training runs. The latest trained an 8B model on 50.53B tokens across 48 distributed A100s, at an estimated $0.1202 per million tokens of GPU rental.
The run reached 48.3% effective MFU. At AWS p4de Capacity Blocks pricing, a 48-A100 cluster operating at the literature-derived 65% compute ceiling comes to an estimated $0.1686 per million tokens. Crucible's measured $0.1202 was about 29% lower after its low-bandwidth overhead.
The comparison excludes R2 storage and operations. The full writeup shows the method and accounting.
Blog: tplr.ai/publications/blog/in…
covenant retweeted
Templar's work reduces to one question. How much of the machine-learning lifecycle can run across ordinary networks instead of a single datacentre?
Pre-training answered first, with Covenant-72B as the proof at scale. Post-training followed through our communication-efficiency research. Serving open models on distributed hardware is the piece we are working on now, and it is the one that puts the whole arc in front of users.
The internet is the datacentre.
covenant retweeted
ByteDance and Tencent each received 10,000 Nvidia H200 chips, the first big delivery after China eased import limits.
Watch what this does to the map. New compute lands in new regions. Supply chains stretch across borders. One export policy shift decides who can train what.
None of that touches how a run schedules across nodes.
Templar treats heterogeneous, cross-geography compute as the normal case. Runs that adapt to whichever nodes are open still finish when the supply picture moves.
China eases limits on Nvidia H200 chips as AI race escalates ft.trib.al/B7WRmPI
covenant retweeted
Nvidia is wrapping a $500B financing plan around GPU collateral. Compute is turning into a financialized asset you can borrow against which says plenty about how far infrastructure has come.
GPUs now sit beside real estate as a form of collateral banks accept.
When hardware secures loans, the edge goes to whoever can deploy and operate capacity fastest. The compute substrate keeps fragmenting into markets and Templar builds a distributed stack to run across it.
Interesting choice of title from The Wall Street Journal.
Nvidia and 6 financial firms have signed memorandums to mobilize more than $500B of third-party capital through independent compute-financing platforms.
- Once Wall Street accepts GPUs as recoverable collateral, Nvidia can gain an advantage that has nothing to do with FLOPS or benchmark scores.
- The new structures are explicitly aimed at buyers whose balance sheets cannot fund the buildout alone, while institutional capital can place debt against Nvidia equipment. Hardware shipments can therefore stay strong even while the underlying operators remain financially weak.
- Goldman Sachs explicitly described an opportunity to create credit backed by Nvidia compute, while Nvidia argues its systems are transferable across customers and workloads.
- If lenders assign those systems higher recovery values, Nvidia customers could borrow more cheaply against them. Rival accelerators might then need lower hardware prices just to offset a financing disadvantage.
- The “circular financing” criticism is too broad unless Nvidia ends up carrying most of the credit risk itself.
But the proposed platforms are meant to mobilize more than $500B of third-party capital, with independent financiers underwriting projects, although Nvidia can backstop up to 25% of potential deals. That makes the better comparison equipment finance rather than vendor buying its own output.
The boundary to watch is how much loss ultimately returns to Nvidia when borrowers fail.
covenant retweeted
Vercel now serves GLM 5.3 at a third of the rival's cost. Open weights are running on the same production hosts that carry the closed frontier.
That cost gap decides which models get deployed. Adoption follows the price per token. Simple math.
Open models have stopped being a research curiosity. They've turned into an infrastructure choice, with quality closing the distance too. How you serve becomes the defense, and heterogeneous compute is exactly the lever there.
covenant retweeted
Alibaba pushed a 27B open-weight model down to smartphone-sized hardware. Qwen 3.8 ships with day-zero edge support, and 3 billion downloads already sit behind the family.
Frontier weights running on consumer devices keeps eroding the story that capable models require a single datacenter. Edge silicon, spare desktops, idle clusters - the training substrate keeps spreading across heterogeneous hardware. That is the terrain our coordination layer is built for.
The model gets the headlines. The hardware spread does the actual work.
Qwen3.8-27B is now part of your everyday life — from smartphones to vehicles. ⚡️🚀Appreciate your work! @MediaTek
covenant retweeted
DeepSeek V4 Pro hit 88.5% on coding benchmarks and cleared Fable 5 at a fraction of the cost.
The open side keeps matching closed models on the hardest engineering tasks. Price tells the rest of the story.
Cheaper coding changes who gets to build. The distance between frontier and open keeps narrowing.
covenant retweeted
CoreWeave signed A100 contracts running through 2029. The older chips everyone wrote off still have a paying life ahead of them.
That defies the refresh-every-two-years loop. Hardware stays valuable well beyond its launch date when the workload fits it.
Distributed training is one way to keep that older compute earning.
The mighty A100 fleet are mission-capable from 2020 through 2029. NVIDIA computing is more than chips. CUDA gives developers and NVIDIA engineers a common platform to continually upgrade Ampere, Hopper and Blackwell throughout their useful lives.
CUDA makes NVIDIA computing versatile. Versatility makes it fungible. Fungibility drives utilization and extends durability, making NVIDIA compute a productive asset: rentable, durable and financeable.
covenant retweeted
Megawatt-scale capex keeps concentrating. The counterweight is compute that already exists in smaller pools, waiting on a coordination layer to make it usable for training.
The answers to demand do not have to all be new buildings.
Goldman Sachs is in talks with potential investors about participating in Nvidia's $500 billion AI financing initiative, after leveraging its long-standing relationship with the chipmaker to secure a coveted role in the deal, people familiar with the matter said reut.rs/4gjm19o
covenant retweeted
Hyperscalers committed $2.6 trillion to AI infrastructure, the largest spending push on record.
That capital concentrates in a few vertically integrated clusters. A bet that large leaves little margin if demand softens or a cheaper training route wins.
Distributed training spreads the same capacity across hardware many owners already run.
We are witnessing the largest CapEx spending spree in history.
Hyperscalers have committed a record $2.6 trillion in future spending across data center leases and equipment purchases.
This includes leases for data centers and power infrastructure that have not yet commenced, and existing purchase contracts for equipment and services.
These obligations do not appear on company balance sheets and are disclosed only in limited accounting footnotes, despite their enormous size and duration.
Alphabet, $GOOGL, has the largest total, with $811 billion in purchase commitments and $85 billion in leases not yet commenced, bringing its combined obligations to ~$896 billion.
By contrast, Oracle, $ORCL, has committed $32 billion to purchases, but $260 billion to leases that have not yet commenced, the largest lease obligation among the group, bringing its total to ~$292 billion.
The AI buildout has become one of the largest long-term capital commitments in corporate history.
covenant retweeted
GPU capacity getting booked through 2029 says a lot about the shape of the market. The biggest clusters are already spoken for, and everyone else fits into what is left over. When most capacity sits inside long contracts, the unclaimed cycle is the scarce resource.
That is the compute that open distributed training is built to capture.
CoreWeave $CRWV: We recently signed an A-100 contract that extends into 2029. This SKU was introduced in 2020.
covenant retweeted
Hidden chain-of-thought used to be a black box you either trusted or ignored. Once the traces are readable, the behavior turns into something reproducible rather than a leap of faith.
Open research gets more interesting when reasoning is inspectable.
covenant retweeted
Peter Robinson gave the Ethereum & AI Engineering Group a 15-minute introduction to LLM design this week, then spent the rest of the session on Covenant-72B, Templar's 72-billion-parameter model trained across the internet. The talk covers the model architecture, the SparseLoCo communication protocol, the Gauntlet evaluation system, and how the training run unfolded.
His closing assessment matches what the run was designed to test: Covenant-72B performs competitively with centrally trained models at comparable scale, and the distributed training research behind it holds up in practice.
His growth analysis is the other takeaway. He tracks centralized models expanding training capacity at roughly 5x per year against 20x for distributed models, and puts the crossover around 2031. The full talk is linked in his post.
Introduction to LLM Design & Covenant 72B: This week at the Ethereum & AI Engineering Group meetup I provided a 15 minute introduction to LLM design and then dived into the design and training of Covenant 72B; the largest model trained over the Internet.
youtube.com/watch?v=TlVQbiTd…
covenant retweeted
DeepSeek V4 Flash cuts model depth to 43 layers, bucking the trend of deeper-is-better. Architecture innovation over scale brute force.
When you can reduce layer count while maintaining capability, you open up training to hardware that can't handle 100+ layer stacks. Fewer layers means less synchronization overhead, more tolerance for heterogeneous setups, and easier distribution across loosely coupled nodes.
The industry assumption has been that frontier models need deeper architectures. That assumption is cracking.
covenant retweeted
Texas just froze new data center approvals, putting 20% of the US pipeline at risk. This is what compute centralization looks like as a single point of failure.
When one state's grid constraints can stall a fifth of national AI infrastructure, the argument for distributed training stops being theoretical. The hardware pipeline is the bottleneck. Permissionless training on heterogeneous hardware hedges against regional grid politics, export controls, and supply-chain concentration.
covenant retweeted
20% of the US project pipeline faces delays while regulators audit power infrastructure.
The centralized data center model breaks on a constraint few discuss. Grids. One state carries a fifth of planned capacity and the utility network cannot keep up. Power delivery has become the binding constraint for centralized AI clusters - more than chip supply or fabrication capacity.
Distributed training side-steps this entirely. Spread computation across smaller nodes and the grid problem disappears. Any location with available power contributes.
Hundreds of millions in grid deposits linger in uncertainty following Texas data center pause reut.rs/4whHqFN reut.rs/4whHqFN
covenant retweeted
Centralized compute at this scale demands billions in capital, years of construction, and entirely new energy infrastructure. Those economics support a handful of players. They exclude every research lab that needs a fraction of that capacity for a defined period.
Building the protocol layer that pools compute across existing hardware costs far less. It deploys faster too. The real question is whether the research community can coordinate on the protocol before capital concentration hardens the terrain.
Terafab will be built in Grimes County, Texas
In April, we broke ground on our research fab on the North Campus of Giga Texas – the precursor to Terafab.
Both Tesla & SpaceX will need far more chips than current & future global production can supply.
This is why we're building the largest chip manufacturing facility ever, with the goal of producing over 1 terawatt of compute per year
The future is built in Texas
spacex.com/updates#terafab
covenant retweeted
Training kernel optimization is an underappreciated bottleneck in distributed training.
Better kernels mean more efficient use of existing hardware, which is the same problem Templar's heterogeneous compute layer addresses at the network level.
Every efficiency gain compounds when you're training across many nodes.
Cursor has open-sourced Mixture-of-Kittens, the fused mixture-of-experts kernel that lifted its production training throughput by 41%.
A really smart technique here.
Training a big model wastes huge amounts of GPU time just moving tokens between chips, and Cursor got 41% more work out of the exact same hardware by rewriting how that movement is scheduled.
The layer it replaces had become the largest single cost in that training, swallowing more than half of total step time on some workloads.
Earlier fixes attacked only the arithmetic, since custom MXFP8 kernels sped up the expert math and left token shuffling to a separate library.
That division worked while compute dominated, but once the experts got fast enough the transfers became the thing everything else waited on.
The switch to GB300 NVL72 racks helped and hurt at the same time.
All 72 chips now sit on one high-speed link, close enough that a chip can receive its next batch of tokens while still crunching the last one.
But the small CPUs bolted onto those chips are slow, and the GPUs kept finishing early and sitting idle while the CPU decided what to send next.
So the new kernel does the whole round trip, sending tokens out, running the experts, and gathering results back, in one go without ever pausing to check with the CPU.
The biggest gain came from flipping the direction of the traffic, since letting each chip pull the tokens it needs beats having others push them, worth up to 29% more usable bandwidth.
Pulling also kills the "I'm done" messages a push requires, which were costing 5.8x more waiting time in their own tests.
Against the fastest public baseline that adds up to 2.37x faster forward passes, and across 512 production GPUs it lifted throughput from 760.9 tokens per second to 1,070.2.
That 41% came from scheduling choices rather than new hardware, which is the part rival labs should find uncomfortable.
Gains built that way compound across every step of every run, so a kernel rewrite can now beat another rack of chips.
