What Is AI Model Distillation, and How Do Small Models Learn From Large Ones?
AI model distillation is a training technique in which a smaller, lighter student model learns part of the predictive behavior of a larger, more capable but heavier and more costly teacher model. Its formal name is usually knowledge distillation (KD). The central idea is not simply to reduce or copy the large model’s parameters. Instead, the student is trained anew to resemble the teacher’s answer distribution for the same inputs and, in some cases, its internal representations as well.arxiv.org
The reason for this technique is relatively clear. Large models can perform well on complex problems, but in production they face constraints such as response latency, memory, power, server costs, and device capability. Distillation is a choice to use the large teacher’s computational capacity during training while deploying a smaller student during operation. The best way to understand distillation, then, is not as magic that folds intelligence into a smaller form, but as a strategy for allocating training costs and operating costs to different models.arxiv.org
Why create a student model instead of using the large model directly?
The requirements of model development and model operation differ. During training, it may be possible to invest long periods of time and substantial computing resources, whereas production services often require short response times and predictable costs. In environments such as mobile devices, embedded equipment, APIs handling high request volumes, or deployments that must run inference on-device for privacy reasons, model size and latency become product requirements.arxiv.orgarxiv.org
Consider a document-classification service. For an internal review task where it is acceptable to analyze one document for several seconds, there may be room to use a large model. But if users need to see a classification result immediately whenever they press a button, or if millions of documents must be processed continuously, the memory and time required for each inference accumulate. If a student model maintains task performance close to that of the teacher, it can be an option that meets operating requirements even if some quality is sacrificed.
Still, it is not safe to conclude that a smaller model is always cheaper. Parameter count is an important indicator of storage and memory use, but actual speed also depends on input length, batch size, compute libraries, hardware, and memory-transfer costs. This is why speedup figures reported in a paper should not be applied unchanged to other hardware and workloads.arxiv.orgarxiv.org
What do teacher and student models exchange in knowledge distillation?
In ordinary supervised learning, the correct answer is given as a single label, such as “this image is a cat” or “this sentence is positive.” This can be called a hard target. A teacher model, however, does not produce only one answer; it calculates different scores or probabilities for every possible answer. For an image, for example, it may predict cat 0.70, fox 0.20, and dog 0.08.
Distillation uses this distribution as additional training material for the student. Beyond the fact that cat is the correct answer, it conveys the signal that the teacher regarded the input as somewhat similar to a fox and less similar to a dog. Hinton and coauthors explained that such soft targets can convey useful information that cannot be contained in a single hard label.arxiv.org
Here, “knowledge” does not mean a list of rules written in human language. More precisely, it means the teacher’s learned function and behavioral patterns that map particular inputs to outputs. The student may not retain the teacher’s parameters directly, and it may have a different number of layers or architecture. What matters is that, for a given input, the student’s output or representation moves closer to the teacher according to the chosen objective.arxiv.orgarxiv.org
How do soft targets and temperature work?
The final layer of a classification model commonly produces scores called logits and then converts them into a probability distribution through the softmax function. In distillation, a softmax that divides the teacher’s and student’s logits by a temperature (T) can be used to make the distribution smoother.
[ p_i^{(T)}=\frac{\exp(z_i/T)}{\sum_j\exp(z_j/T)} ]
Here, (z_i) is the logit for class (i). When (T=1), this is the ordinary softmax. Increasing the temperature flattens a distribution that is concentrated only on the highest-probability class. This can make relative differences among lower-ranked classes available as learning signals. That is why it is incomplete to describe temperature merely as “blurring the teacher’s answer.” Temperature is not a switch that automatically increases the amount of information; it is a control that changes the shape of the distribution from which the student learns.arxiv.orgproceedings.mlr.press
In practice, training usually combines two objectives. One loss encourages the student to predict the true labels, and the other, the distillation loss, encourages it to approach the teacher’s soft outputs. This can be written simply as follows.
[ L=\alpha L_{\text{ground truth}}+(1-\alpha)L_{\text{distillation}} ]
(\alpha), the temperature (T), and the type of distillation loss are all values that must be tuned. A higher temperature or a larger weight on the teacher signal is not always better. Research also suggests that the effect of temperature selection can differ according to characteristics of the teacher’s distribution, such as when the teacher was trained with label smoothing.proceedings.mlr.pressproceedings.mlr.press
Is matching outputs alone enough?
The most basic form of distillation matches only final outputs. This can be called response-based distillation. In classification models, probabilities or logits are the main targets; in language models, the distribution over the next token is the main target. It is relatively simple to implement and can be considered when the teacher’s internal architecture is unknown, as long as its outputs are accessible.arxiv.orgproceedings.mlr.press
However, some approaches hold that final answers alone cannot adequately convey the teacher’s computational process. In architectures where multi-layer representations and attention matter, such as Transformers, the student can be trained to imitate intermediate signals including the teacher’s embeddings, hidden states, and attention matrices. This is called feature-based distillation or intermediate-layer distillation.arxiv.org
TinyBERT proposed using signals from the embedding layer, Transformer layers, and prediction layer, and performing distillation during both general pretraining and task-specific fine-tuning. In ablation experiments in that paper, removing Transformer-layer distillation led to a substantial performance decline. This is not a general rule that intermediate-layer distillation is always essential; rather, it is an example showing that signals beyond final outputs can be useful depending on the architecture and task.arxiv.org
Another distinction is between offline distillation and online distillation. Offline distillation fixes an already trained teacher and trains a student. Online distillation uses multiple models that teach one another during training, or teacher signals that emerge concurrently. Whether a separate teacher can be maintained, how much training cost can be afforded, and whether models need to be trained together are criteria for choosing between them.arxiv.org
How does distillation differ from pruning and quantization?
Although all are discussed under the shared goal of model compression, they reduce different things. Distillation concerns learning signals and the behavior of the student model. Pruning, by contrast, reduces the structure by removing low-importance connections, channels, heads, or layers. Quantization reduces storage and compute costs by lowering the bit width or changing the numerical format used to represent weights and activations.arxiv.orgarxiv.orgarxiv.org
| Method | What it primarily changes | Typical strength | Condition to verify |
|---|---|---|---|
| Knowledge distillation | Student learning objectives and architecture | A small model learns the teacher’s behavior | Teacher quality, data, capacity gap |
| Pruning | Connections, channels, heads, layers | Removes redundancy from an existing structure | Whether the target hardware actually processes sparse structures faster |
| Quantization | Numerical precision | Reduces memory use and integer-operation costs | Accuracy loss, hardware and runtime support |
| Combined use | Both training and representation | Potential for greater compression | Quality degradation at each stage and operational complexity |
For example, converting 32-bit floating-point weights to 8-bit integers is quantization. Training a six-layer student Transformer with the output distribution of a large teacher as its target is distillation. They can also be applied sequentially. However, the fact that a paper achieved a high compression ratio does not mean real service latency will decrease by the same proportion. For quantization in particular, the effect depends on how well the target CPU, GPU, or NPU supports the relevant integer operations.arxiv.orgarxiv.org
What do BERT-family examples show?
Language-model distillation is a useful example for understanding the concept. BERT-base is a pretrained Transformer widely used for language-understanding tasks, but it may be burdensome to deploy in resource-constrained environments. The DistilBERT study presented a pretraining-distillation method that initializes a smaller model from some BERT layers and combines language-modeling loss, distillation loss, and cosine-distance loss.arxiv.org
The study reported retaining about 97% of BERT performance on the GLUE development set while reducing parameter count by about 40%. It also showed shorter inference time under the CPU measurement conditions defined in the paper. These figures, however, were obtained with English benchmarks, a particular BERT configuration, particular hardware, and specified input conditions. They should not be cited as universal figures guaranteeing the same results for Korean, long documents, retrieval-augmented systems, generative conversations, or other accelerators.arxiv.org
TinyBERT reported that a smaller four-layer model retained strong performance relative to BERT-base while substantially reducing size and inference time under the paper’s conditions. It also presented ablation experiments showing that general distillation, task-specific distillation, data augmentation, and intermediate-layer signals all influenced the result. The practical lesson from this example is simple: it is not enough to connect one teacher and one student; the timing of distillation, data, student architecture, and loss design jointly determine the outcome.arxiv.org
When is distillation especially effective?
Distillation is particularly worth considering not when the large model’s quality must be preserved perfectly, but when a clear performance floor and strong operating constraints exist together. Examples include the following.
- On-device inference: Environments where network connectivity is unreliable or inputs are difficult to send to external servers because of privacy requirements.
- High-volume repeated inference: Classification, ranking, and detection tasks that process very many short inputs, making cumulative cost and latency important.
- Domain-specific tasks: Cases where performance on a defined document type or label system matters more than all of a general-purpose model’s broad capabilities.
- Replacing a model ensemble: Cases where averaging predictions from several models benefits quality but is too heavy to operate, so a single student is used to approximate their behavior.arxiv.org
Conversely, when the student must never miss rare cases, or when it must faithfully reproduce a teacher’s long reasoning process, tool use, or complex safety policies, evaluation criteria need to be broader. Even if average accuracy is high, performance can deteriorate for particular groups, rare classes, or risky inputs. Research has found that the relationship between worst-class performance and average accuracy can vary with the design of the distillation objective.proceedings.mlr.press
What are the limitations of distillation?
First, the student learns not the teacher itself but the teacher’s observable behavior. If the teacher makes incorrect predictions or reflects data bias, the student can learn those signals as well. Using hard-label loss alongside distillation does not automatically resolve problems involving bias, factuality, security, or safety. Quality should still be checked after distillation using an evaluation set independent of the original teacher.arxiv.orgproceedings.mlr.press
Second, if the student is too small, it cannot imitate the teacher sufficiently. This is called the teacher–student capacity gap. Related research reports that performance can deteriorate when a small student tries to follow signals from an excessively large teacher, and that a more accurate teacher may not always be a better distillation teacher. Selecting a teacher is not merely a ranking competition; it is also a question of compatibility with the student architecture.arxiv.orgarxiv.org
Third, distillation can shift operating costs into upfront training costs. The teacher must first be trained or obtained, the teacher’s outputs for distillation data must be computed and stored, and temperature, loss weights, and layer mappings may need to be explored. Therefore, for an analysis run only once, using the teacher directly may be simpler; for a long-running, repeated service, there is greater opportunity to recover the cost of developing a student.arxiv.orgarxiv.org
Fourth, distillation does not guarantee “explainable compression.” Even if a student produces predictions similar to the teacher’s, whether it makes decisions through the same internal mechanisms and whether it preserves all of the teacher’s information are separate questions. Recent research also points out that it remains unclear whether a small student receives all of a teacher’s information through distillation.proceedings.mlr.pressproceedings.mlr.press
What are common misconceptions?
The misconception that “distillation causes no performance loss”
Distillation research shows that small models can retain high performance, but this is an observation under specified evaluation conditions. If the input distribution changes, long inputs become more common, or rare cases become important, the gap between teacher and student may grow. Therefore, rather than relying on a single figure such as “a percentage of the teacher’s performance,” evaluate the accuracy, recall, latency, memory, and safety metrics required by the service together.arxiv.orgarxiv.org
The misconception that “it copies all of the teacher model’s knowledge”
Within limited parameters and architecture, the student approximates the teacher’s outputs or some of its representations. It is difficult to regard all of the teacher’s internal information as being transferred unchanged. Distillation results depend on the data, objective function, student capacity, and optimization method.proceedings.mlr.pressproceedings.mlr.press
The misconception that “fewer parameters must mean faster inference”
Reducing parameters generally helps reduce memory requirements, but actual latency depends on the computation graph, sequence length, batch size, and hardware bottlenecks. In particular, the attention-computation burden of Transformers can vary substantially with input length, so end-to-end measurement on the target device is necessary.arxiv.orgarxiv.org
The misconception that “teacher outputs mean original data is unnecessary”
There is research on data-free or zero-shot distillation that attempts to distill using a teacher without the original training data. However, this is a special setting that requires separate synthetic-data generation or a method for approximating the input distribution. Ordinary distillation uses representative input data together with teacher signals, and approaches that also use true labels are widespread.proceedings.mlr.pressproceedings.mlr.press
What should you check before deploying distillation in practice?
It is safer to work backward from deployment goals rather than deciding based on a model name. First, quantify not “how small a model is needed,” but “which quality and operating requirements must be met.” The following is a minimum checklist.
- Separate the task criteria. In addition to average accuracy, define recall, the costs of false positives and false negatives, worst-group performance, response time, and memory limits.
- Validate the teacher. Check not only the teacher’s benchmark scores but also real domain data, error types, bias, and safety requirements. An unvalidated teacher is not guaranteed to provide good learning signals.
- Set the student budget first. Clearly define parameter count, maximum memory, allowable latency, input length, and deployment hardware. If the student is too small, the capacity gap can grow.arxiv.org
- Choose the distillation objective. For simple classification, output distillation can be a starting point; when representation quality matters, as in Transformer compression, intermediate-layer and attention distillation can be tested.arxiv.org
- Compare teacher and student with the same tests. Include not only general accuracy but also input lengths similar to real traffic, rare cases, and groups with high error rates.
- Measure on the target hardware. Distinguish model-file size, memory, initial loading, latency including tokenization and preprocessing, throughput, and power consumption.arxiv.orgarxiv.org
- Validate follow-on compression separately. If quantization or pruning is applied after distillation, record quality degradation and speed gains separately after each stage.
Conclusion: Distillation creates the right operational balance, not merely a “small model”
AI model distillation transfers predictive distributions and representations learned by a large teacher to a student, creating a model that can be used under constrained resources. The key is to add the teacher’s soft targets to a student that learns from hard labels alone and, when needed, use intermediate-layer signals as well.arxiv.orgarxiv.org
However, distillation is not a file-compression technique that preserves performance for free. Teacher errors and bias, student capacity, the representativeness of distillation data, temperature and loss design, and the characteristics of actual hardware all shape the outcome. The most sensible next step, therefore, is not to target a compression ratio from a particular paper, but to define the minimum quality and maximum latency and cost required for your own task, then compare the teacher and candidate students under the same conditions.