https://hello-diana.github.io/tags/self-distillation/Self-DistillationHugo Blox Builder (https://hugoblox.com)en-usThu, 07 May 2026 00:00:00 +0000https://hello-diana.github.io/media/icon_hu_3520ea6f5cfedd63.pnghttps://hello-diana.github.io/tags/self-distillation/https://hello-diana.github.io/publication/unisd/Thu, 07 May 2026 00:00:00 +0000https://hello-diana.github.io/publication/unisd/<h2 id="abstract">Abstract</h2> <p>Self-distillation offers a promising path for adapting large language models without stronger external teachers, but it remains hard to apply reliably in autoregressive LLMs. We propose <strong>UniSD</strong>, a unified framework that systematically studies self-distillation by integrating complementary mechanisms — multi-teacher agreement, EMA teacher stabilization, token-level contrastive learning, feature matching, and divergence clipping. Across six benchmarks and six models from three families, UniSD clarifies when and why self-distillation helps, and its integrated pipeline improves over the base model by +5.4 points and the strongest baseline by +2.8 points.</p>