Memorization Inheritance in Knowledge Distillation: When Students Learn the Teacher's Mistakes

Sequence-level knowledge distillation has become standard practice in neural machine translation for creating deployment-ready models. By training compact student models on synthetic translations generated by larger teachers, practitioners achieve competitive performance with reduced inference costs. The technique underpins many production systems and enables deployment on resource-constrained devices, particularly for high-resource language pairs such as German-English or lower-resource combinations like French-German.

However, a recent study by Dankers and Raunak titled "Memorization Inheritance in Sequence-Level Knowledge Distillation for Neural Machine Translation" reveals a concerning pattern that complicates the narrative of distillation as pure model compression. Their analysis demonstrates that students inherit not merely the teacher's translation capabilities, but also its instance-level memorization patterns and failure modes. Specifically, students exhibit 3.4% higher exact match memorization and 57% higher extractive memorization compared to baseline models trained directly on the original WMT20 parallel corpus. Perhaps more troubling, students display 31% more oscillatory hallucinations than their baseline counterparts, suggesting that distillation redistributes rather than eliminates certain classes of error.

The Mechanisms of Memorization Inheritance

The paper establishes that SeqKD creates a peculiar information pathway that differs fundamentally from standard supervised learning. In conventional training, a model memorizes training pairs through direct exposure to parallel text. In SeqKD, the student never sees the original target sequences from the corpus. Instead, it trains exclusively on teacher-generated translations derived from source sequences.

One might assume this indirect training would reduce memorization by breaking the direct link between source and original target. Surprisingly, the opposite occurs. Using Transformer-large teachers trained for 300,000 steps on WMT20 data (including 48 million sentence pairs for German-English and 14 million for French-German), and Transformer-base students trained on synthetic targets generated with beam size one, the authors identify two distinct memorization phenomena.

First, primary extractive memorization increases dramatically. Students reproduce target-like sequences from the original corpus even when trained only on teacher outputs, suggesting that the teacher's parameters encode the training data in ways that leak through the generation process. Second, and more insidious, the authors identify secondary extractive memorization. Students memorize artifacts specific to the teacher's generation, such as extraneous website URLs or repetitive phrases that appear in the synthetic targets but not in the original human translations.

This secondary memorization suggests that students do not merely learn the mapping from source to target language. They learn the idiosyncratic generation patterns of the specific teacher model, including its hallucinations and copying behaviors. The teacher's beam search outputs contain traces of the original training data embedded in its parameters, and students extract these traces through the distillation process, effectively memorizing data they never directly observed.

The Denoising-Hallucination Trade-off

Perhaps the most intellectually challenging aspect of the findings concerns the relationship between denoising and hallucination. Dankers and Raunak analyze subgroups of training data stratified by quality estimation metrics and counterfactual memorization scores, the latter derived from Feldman (2020) to identify atypical examples requiring explicit memorization. They discover that students demonstrate superior denoising capabilities on low-quality source-target pairs. When faced with noisy or corrupted training examples, students filtered through SeqKD produce cleaner translations than both teachers and baselines, as measured by Comet-QE-22 scores.

However, this capacity for denoising comes with a statistical cost. The same students exhibit significantly elevated rates of oscillatory hallucinations, repetitive patterns that diverge from source meaning. The authors note increases of 31.0% in such hallucinations compared to baseline models trained directly on the original parallel data.

This pattern suggests a capacity allocation mechanism. Neural networks possess finite representational capacity. When SeqKD trains students on teacher outputs, the student learns to ignore certain types of noise present in the original parallel corpus. This frees up capacity within the smaller student model. Rather than using this capacity to improve generalization, the model appears to reallocate it toward memorizing the teacher's specific generation patterns, including hallucinatory sequences.

The error is not eliminated. It is transformed and concentrated into different failure modes. Low-frequency noise gets filtered, but systematic teacher artifacts get amplified. The student becomes expert at cleaning dirty data, yet simultaneously more prone to generating text that bears no relation to the source.

Original Insights: Compression and Iterative Degradation

These findings force us to reconsider the fundamental metaphor of knowledge distillation. We typically conceptualize distillation as a lossless compression algorithm, where we discard redundant information to preserve essential structure. The Dankers and Raunak study suggests a different analogy: distillation as photocopying, where each generation accumulates artifacts and loses fidelity to the original signal.

The implications for iterative distillation are particularly concerning. If one were to train a sequence of models, where each generation serves as teacher to the next, the compounding effects of secondary memorization could create models that bear little resemblance to the true data distribution. Instead, they would become lossy encodings of the initial teacher's hallucinations, having memorized synthetic artifacts several steps removed from any human-generated translation. Each iteration would concentrate the teacher's specific biases while filtering genuine linguistic variation.

The proposed Adaptive-SeqKD method offers a partial remedy. By finetuning the teacher briefly on high-quality data identified through intrinsic quality estimation metrics before generating training targets, practitioners can reduce both memorization and hallucinations in students. This intervention acknowledges that not all teacher outputs deserve equal weight in the student's training curriculum, effectively filtering the teacher's own outputs to remove low-quality generations that would otherwise become memorization targets.

However, Adaptive-SeqKD addresses symptoms rather than underlying causes. The fundamental issue remains that we lack principled constraints on how students allocate their capacity when learning from teachers. We cannot currently specify that a student should learn the teacher's denoising behavior without also acquiring its memorization patterns. The capacity freed by filtering noise appears to get reallocated automatically toward memorizing teacher-specific artifacts, and we lack theoretical frameworks to constrain this reallocation.

Conclusion

The work of Dankers and Raunak serves as a necessary corrective to uncritical adoption of sequence-level knowledge distillation in machine translation. Their empirical results demonstrate that students inherit fault modes alongside capabilities, memorizing both the training corpus and the teacher's synthetic artifacts with respect to the initial parallel data.

Moving forward, the field requires monitoring frameworks that track not just aggregate BLEU or COMET scores, but instance-level memorization metrics and hallucination rates. We must ask whether the inference speed gains from distillation justify the reliability risks of increased hallucination, particularly in high-stakes deployment scenarios where oscillatory hallucinations could propagate misinformation.

The central question posed by this research remains open. How do we decouple the beneficial aspects of teacher knowledge, such as denoising on low-quality subgroups, from the undesirable aspects, such as hallucination and memorization? Until we develop training objectives that explicitly penalize the memorization of teacher-specific artifacts while preserving robust translation capabilities, sequence-level knowledge distillation will remain a double-edged tool, offering efficiency at the potential cost of reliability. We need active monitoring and interventions like Adaptive-SeqKD not as optional enhancements, but as essential safeguards against the inheritance of error.