Cram Less to Fit More: Rethinking Data Efficiency for Factual Memorization in Language Models

The prevailing wisdom in large language model development has long assumed that scale is the primary solution to knowledge retention. If a model hallucinates facts or fails to recall rare information, the prescription has been straightforward: increase parameters, add more data, and extend training time. Yet a sobering prediction from prior work suggested that memorizing all facts in Wikipedia to human accuracy levels would require an impossibly large model with 10^20 parameters. This suggests that raw scaling may hit fundamental limits long before achieving reliable factual accuracy.

In their recent work, "Cram Less to Fit More: Training Data Pruning Improves Memorization of Facts," Jiayuan Ye, Vitaly Feldman, and Kunal Talwar challenge this assumption through a rigorous information-theoretic analysis. Their central thesis is both counterintuitive and empirically validated: reducing the volume of training data, when done strategically, enables smaller models to memorize significantly more facts than their larger counterparts trained on full datasets. Specifically, they demonstrate that a GPT2-Small model with 110 million parameters, when trained on a carefully pruned subset of Wikipedia, achieves 1.3 times higher factual accuracy than the same model trained on the full corpus, matching the performance of a model ten times larger trained on unfiltered data.

The Information-Theoretic Bottleneck

To understand why more data can paradoxically yield less memorization, Ye et al. formalize fact retention through the lens of information theory, connecting it to the established 2 bits per parameter memorization capacity limit observed in prior literature. This theoretical ceiling, derived from work by Allen-Zhu and Li as well as Morris et al., suggests that a model with $N$ parameters can memorize at most $2N$ bits of independent information. While this provides a hard upper bound, the authors prove a critical extension in Theorem 3.6 and Corollary 3.7: fact accuracy remains suboptimal whenever the information content of training facts exceeds this capacity, with the deficit exacerbated by skewed frequency distributions.

The mechanism behind this failure mode is subtle but consequential. Natural language datasets follow power law distributions where a small number of entities dominate the corpus while the majority appear rarely. When training proceeds on such long-tailed data, high-frequency facts consume disproportionate representational capacity through redundant repetitions, effectively crowding out rare but distinct facts in the tail. The authors validate this through controlled experiments using synthetic phonebook memorization benchmarks, demonstrating that models fail to reach capacity limits unless trained on proportionally fewer, uniformly distributed facts. Remarkably, even increasing training steps by 8x or using models 10x larger does not resolve this suboptimality when the data distribution remains skewed, indicating that the problem is not merely insufficient training time but rather a fundamental misallocation of limited model capacity.

Strategic Data Selection via Training Dynamics

The proposed solution operates on two principles: limiting the total number of facts to avoid exceeding model capacity, and flattening the frequency distribution to approximate uniformity. The challenge in implementing this strategy lies in the fact that real-world corpora do not arrive with annotated fact boundaries or frequency statistics. Ye et al. circumvent this obstacle by leveraging training loss as a proxy for fact frequency, an insight that transforms standard training dynamics into a curation mechanism.

Their LossH-Wiki selection algorithm monitors per-sample training loss to identify and retain examples that the model finds difficult to learn. High-loss examples typically correspond to rare facts or high-entropy information that the model encounters infrequently, while low-loss examples represent common knowledge that has been over-represented in the training mix. By pruning low-loss examples and maintaining a selection ratio $\alpha$ (tuned to 0.2 for 110M parameters and 0.3 for larger models), the method effectively flattens the empirical distribution without requiring explicit knowledge of entity boundaries or fact extraction.

This approach represents a significant departure from traditional data pruning strategies, which often focus on removing noisy or low-quality samples. Instead, Ye et al. remove easy samples, those containing information the model has already captured or that provide diminishing returns through repetition. The technical implementation involves training a reference model to obtain loss values, then retraining on the selected subset for approximately 8 epochs over 3 billion tokens with batch size 320 and context length 1024.

Empirical Validation and the Efficiency Frontier

The experimental results substantiate the theoretical framework across multiple scales. In semi-synthetic settings involving high-entropy title-author mappings from arXiv papers and synthetic phonebook tasks, the selection method boosts fact accuracy to the theoretical capacity limit. More impressively, when pretraining from scratch on an annotated Wikipedia corpus, the 110M parameter model trained with LossH-Wiki achieves fact accuracy equivalent to a 1.3 billion parameter model trained on the full dataset.

Crucially, this improvement in factual memorization does not come at the cost of general language understanding. The authors report that natural language understanding accuracy remains stable across model sizes, suggesting that the pruned datasets preserve the syntactic and semantic diversity necessary for general competence while eliminating redundant factual repetition. This decoupling of factual memorization from general linguistic capability challenges the assumption that these capacities necessarily compete for the same representational resources.

Broader Implications and Critical Limitations

While the results are compelling, several limitations warrant consideration. The experiments are conducted on GPT2-scale architectures (up to 1.3B parameters), leaving open the question of whether these dynamics persist in modern frontier models with 7B to 70B+ parameters and different architectural choices such as Mixture-of-Experts or advanced attention mechanisms. The scaling literature suggests that larger models exhibit different emergent capabilities and may handle long-tailed distributions more efficiently through better gradient allocation, potentially reducing the relative gains from aggressive pruning.

Furthermore, the reliance on training loss as a frequency proxy assumes a relatively clean initial dataset. Web-scale corpora contain significant noise, adversarial examples, and potentially mislabeled facts. High-loss examples in such settings might reflect label noise or grammatical errors rather than valuable rare knowledge, raising the risk that loss-based selection could amplify training on incorrect information. The method also presumes static datasets, whereas modern pretraining increasingly relies on continual or streaming data pipelines where frequency statistics evolve throughout training.

From a broader perspective, this work reframes the data quality versus quantity debate. It suggests that the "more is better" philosophy dominating recent large-scale pretraining efforts may be suboptimal for specific capabilities like factual accuracy. If smaller, curated models can match larger uncurated ones in knowledge retention, the implications for democratizing AI development are substantial. Organizations without access to industrial-scale compute could potentially train specialized models with competitive factual accuracy using careful data selection rather than massive parameter counts.

However, this raises tension with research on emergent capabilities, which often demonstrates that certain reasoning and knowledge synthesis abilities appear only at sufficient scale. If aggressive pruning reduces the diversity of exposure during training, might it inhibit the development of cross-domain reasoning or analogical thinking that arises from broad, if redundant, training? The preservation of NLU accuracy in these experiments is reassuring, but does not fully resolve whether long-horizon reasoning or creative generation might suffer from reduced data volume.

Conclusion and Open Questions

Ye, Feldman, and Talwar have provided a rigorous theoretical and empirical foundation for understanding how data distribution shapes the fundamental limits of factual memorization. By demonstrating that strategic pruning can enable a 10x reduction in effective model size for knowledge tasks, they challenge the inevitability of brute-force scaling for accurate language models.

Looking forward, several questions remain unanswered. Can automated fact boundary detection replace loss-based proxies to enable more precise capacity management? How do these principles extend to multimodal training where facts are distributed across text, image, and video modalities with different entropy characteristics? Most importantly, can we develop dynamic pruning strategies that adapt selection ratios throughout training as the model's capacity utilization evolves?

For practitioners, the immediate takeaway is clear: before investing in larger models or longer training runs, examine your data distribution. The path to better factual accuracy may not lie in cramming more information into finite parameters, but in carefully curating what the model is asked to remember.