Austin Veselka

New blog post on context parallelism, one of the large technical feats behind ART. furtherai.github.io/blog/pos… It's quite a long read but really cool if you're interested in the engineering. I think Astra has done quite a good job with my recent prompt in writing and revising this. I did work very closely with codex for the attn and GDN versions. This was maybe gpt 5.2 so it really isn't something where it can get very far itself. So, we planned and iterated on the whole system and details together. Only thing I didn't do is write the code, the Triton kernels. But these systems handle not only CP, but prefix tree dependencies as well. No one else does that. I think the models will be able to better take over in the future (e.g. the Mamba 2 version was unsupervised by 5.6 sol, not entirely sure how well this one works cause I didn't really stress test it's implementation past what it was designed for but the design is quite simple so I suspect it's ok). Luckily we have most CP regimes so new designs don't have to come from scratch.

To give an idea of the scale of performance for ART, I ran some benchmarks against SkyRL and e2e training is 3-19x faster. Qualifier is really just that we're lora only. Though it wouldn't be too hard to do full-weight. The gains are a result of 3 large technical advantages (prefix tree packing, CP, e2e optimization and autotuner) plus optimization on actual trainer speed. 19x comes when the prompt is 5-10k tokens and completions are short (matching some internal workloads) and 3x is for decode/inference bound cause we currently don't do inference replicas (requires a bit of work to route requests and handle potentially different replica states, I'll get to it). I'm fucking crushing it. Btw I did bench against slime (the popular guy) and we released a blog post about it earlier this year (would expect roughly the same speedups), but idk if they've made any of the important technical improvements for this kind of speed in the meantime. Maybe they have some advantages for coding agent style training

Astra does really well at coming up with code-based hypotheses for errors. It's always like, "We hit this mysterious issue, could be anything. First attempt to attach diagnostics failed so we really have no idea." Then you ask for a code based hypothesis from reading the relevant code path and it says, "Yeah here's the likely problem." 10 min later it confirms and fixes it. Haven't tried this much with other models really, previously they've been working on something where I can trust them to debug quickly. Current iteration loop is too slow for that

Very cool work, check out my autotuner in ART which solves a similar problem. Async RL requires setting a lot of parameters manually (rollout workers, min and max training batch size, queue size), and these can be a big lever on the e2e throughput of your training. The autotuner tries to optimize these automatically based on the concepts of trainer and inference load relative to their peaks. e.g. trainer and inference are underfed: increase rollout workers. The full logic is a bit more complex but reasonably robust and does a great job. Main caveat is that it may not be optimal in terms of real training to, say, increase batch size. This depends on needed sample efficiency of your run and the SNR. But the tuner doesn't know this precisely and generally (could be fun research). It still is usually way better to have it on than off.

furtherai.github.io/blog/ Made a blog for technical write-ups, will discuss my ART infra work (probably cool stuff to read if you're interested in RL infrastructure), math I'm reading (guides from codex on MuP/Complete(d)P-related math) as well as research projects in the future. I hate writing so this will be leaning heavily on AI (thank God for LMs to do this for me), but I'll read and edit and guide the writing, hopefully making it make sense. Would love to see questions, feel free to reach out. First post discusses prefix tree packing in ART, check it out if you're interested in RL and how to apply prefix caching to training.

I have been really optimizing ART for a while now, this is the first of some pretty interesting things I have built and we'll share. We'll talk about some really impressive context parallelism next time and then optimizing the end-to-end training speed of ART!

I also trained two versions of the models, one with the mixed think + non-think examples and one with only non-think examples. I evaluate both versions with and without the <cot> token in the system prompt:

When building examples, I make most examples with a control token <cot> in the system prompt. These examples include the reasoning trace. This gives me a switch that affects the model's reasoning or path to the answer.

There are two exclusive branches next: - Vision: receives the sorted pages and the question only - Text: sorted, extracted evidence, the question, and context on the input (ties the answer causally to the reasoning reasoning trace). Both branches generate a final answer.

This seems to be key to getting good performance. The first version just had each page's evidence from the whole document (some pages as "irrelevant"), but during inference, the model tends to loop on "irrelevant". I believe the model learns v2's RAG-like algorithm.

Thus, ground truth pages are ~always marked relevant, with some score flexibility. I filter for scores above a threshold, sort the pages + their evidence from greatest to least and take the top K (16).

I take a document and a synthetic question. For each page, I task a VLM with extracting evidence relevant to the question, along with a relevance score in [0.0, 10.0]. If the page was used to generate the question, I prompt the model to score between [6.0, 10.0].