Josip Jukic retweeted
Sampling many solutions and taking the majority answer is the most reliable label-free way to improve LLM reasoning. Training methods reduce the consensus to a filter, a preference, or a scalar reward. What if the model could read them instead?
Self-distillation gives LLMs dense reasoning feedback without requiring a larger teacher. But ID gains can come at the cost of OOD generalization. New joint work with @iatitov : "Geometric Self-Distillation for Reasoning Generalization" (arxiv.org/abs/2607.06855)
7/8
That local collapse later appears as false consensus: multiple sampled solutions confidently agree on the same wrong answer.
Figure 7 shows that GeoSD keeps sampling useful, reduces false consensus, and makes agreement more informative.
8/8
At 8B, these gains come with only about a 10% increase in measured wall-clock time.
The broader lesson: effective privileged self-distillation requires control over teacher signals and the extent to which they can change the student.
Paper: arxiv.org/abs/2607.06855
Presenting our paper "Disentangling Latent Shifts of In-Context Learning with Weak Supervision" (with Jan Šnajder) at NeurIPS 2025, San Diego:
🗓 Fri, Dec 5 · 11:00–14:00 PST
📍 Exhibit Hall C/D/E · Poster #2615
Paper: openreview.net/pdf?id=tAq9Gx…
#NeurIPS2025