UniSD: Towards a Unified Self-Distillation Framework for Large Language Models
May 7, 2026·
,,,,,,,,·
1 min read
Yiqiao Jin
Yiyang "Diana" Wang
Lucheng Fu
Yijia Xiao
Yinyi Luo
Haoxin Liu
B. Aditya Prakash
Josiah Hester
Jindong Wang
Srijan Kumar
Overview of the UniSD framework (adapted from Figure 1 of the paper).Abstract
Self-distillation (SD) offers a promising path for adapting large language models (LLMs) without relying on stronger external teachers. However, SD in autoregressive LLMs remains challenging because self-generated trajectories are free-form, correctness is task-dependent, and plausible rationales can still provide unstable or unreliable supervision. Existing methods mainly examine isolated design choices, leaving their effectiveness, roles, and interactions unclear. In this paper, we propose UniSD, a unified framework to systematically study self-distillation. UniSD integrates complementary mechanisms that address supervision reliability, representation alignment, and training stability, including multi-teacher agreement, EMA teacher stabilization, token-level contrastive learning, feature matching, and divergence clipping. Across six benchmarks and six models from three model families, UniSD reveals when self-distillation improves over static imitation, which components drive the gains, and how these components interact across tasks. Guided by these insights, we construct UniSD_full, an integrated pipeline that achieves the strongest overall performance, improving over the base model by +5.4 points and the strongest baseline by +2.8 points.
Type
Publication
arXiv preprint
Abstract
Self-distillation offers a promising path for adapting large language models without stronger external teachers, but it remains hard to apply reliably in autoregressive LLMs. We propose UniSD, a unified framework that systematically studies self-distillation by integrating complementary mechanisms — multi-teacher agreement, EMA teacher stabilization, token-level contrastive learning, feature matching, and divergence clipping. Across six benchmarks and six models from three families, UniSD clarifies when and why self-distillation helps, and its integrated pipeline improves over the base model by +5.4 points and the strongest baseline by +2.8 points.
