UniSD
Overview of the UniSD framework (adapted from the paper).Self-distillation lets large language models improve without a stronger external teacher, but it is hard to apply reliably: self-generated trajectories are free-form, correctness is task-dependent, and prior work studies design choices in isolation. UniSD is a unified framework that brings these choices together — multi-teacher agreement, EMA teacher stabilization, token-level contrastive learning, feature matching, and divergence clipping — to study when and why self-distillation helps.
My role. I am a co-author on this collaboration. Across six benchmarks and six models from three families, UniSD clarifies which components drive gains and how they interact, and its integrated pipeline improves over the base model by +5.4 points and over the strongest baseline by +2.8 points.
