Understanding Knowledge Distillation in Neural Sequence Generation

What is Knowledge distillation? | IBM

Understanding Knowledge Distillation in Neural Sequence Generation - Microsoft Research

Knowledge Distillation as Semiparametric Inference [Talk]

Artificial Intelligence | AI on Instagram: "Knowledge distillation is a deep learning technique where a compact “student” model learns to replicate the performance of a larger, more complex “teacher” model. Introduced in the paper “Distilling the Knowledge in a Neural Network” by Hinton, Vinyals, and Dean (2015), the process goes beyond simply training the student on labeled data, which they refer to as “hard labels”. Instead, the teacher provides “soft labels,” which are its full output probabi