From my friend and guide for a long time, a timely essay: terrytao.wordpress.com/2026/…

Hey! Thanks for reading our paper! Our appendix has a brief discussion explaining why the code approximates the exact MH implementation and why this approximation is reasonable, so I'll elaborate here. Since our proposal prompt inputs the current partial trace, to calculate the acceptance probability for MH from x to x', you need to calculate the reverse transition log-probability from x' back to x, which requires a separate forward pass from the one that generates x' from x. Admittedly, our implementation for MCMC is expensive as it currently isn't batched, so instead of one LLM call per MCMC step, having two LLM calls becomes a bit costly. So as a first approximation, we used a deterministic likelihood-improving acceptance rule. To validate this inexactness, we can still estimate the KL divergence between this process that generates the boosted traces and the base model. I've attached Figure 5 from our paper here, but this shows the KL divergence gap declines with more MCMC steps, so crucially the desired effect of the trajectories becoming progressively more on-policy is maintained. A similar story is the case for the enforcement of the hard constraint. The underlying assumption is that instruction-following enforces the hard constraint in-context, but again, empirically this is inexact. A more exact implementation would have a consistency check (i.e. small LLM check on partial trace ~ privileged info consistency), but again, repeated API calls can be costly. As a way around this, we had post-generation filtering, where rerun on incorrect MCMC traces outside instead of inside the loop, which results in around 95% correctness on average. So in summary, these approximations maintain the effect we care about, which is much more on-policy trajectories with consistency well sustained. In other words, an "existence proof" that an algorithm like this is worth doing. There is most certainly room for much better practical efficiency; the approximations we made were **one** route to making the algorithm doable, but incorporating batching and partial-trace independent proposals is one step towards undoing this inexactness and making our current algorithm run a lot faster.

SFT is not dead! 🥳 We found a way to make SFT rival current prevailing posttraining methods, often generalizing better and forgetting less than RL and OPSD. 🤯 Following our prior work on reasoning with sampling, we now introduce sampling to the posttraining stack. 1/n