When the Name Changes the Answer: Surface Forms and the Limits of Memorization Benchmarks

Introduction

If you ask a large language model what sport Pelé played, you will almost certainly get the right answer. If you ask the same model about Edson Arantes do Nascimento, the legal name of the very same person, you may not. The fact has not changed. Only the name has. Yet according to the paper Revisiting Non-Verbatim Memorization in Large Language Models: The Role of Entity Surface Forms by Nishida and colleagues at the Nara Institute of Science and Technology and Future Corporation, up to 23.7% of canonical-redirect question pairs yield inconsistent predictions on a model like Pythia-12B.

That single number deserves more attention than it usually gets. It quietly undermines an assumption baked into most knowledge benchmarks: that querying a fact with one canonical entity name tells you whether the model has memorized that fact. The authors argue, convincingly, that we have been conflating two different things. There is the question of whether a model has a fact, and there is the question of whether it can access that fact through a particular name. These are not the same, and treating them as the same has been quietly distorting how we measure non-verbatim memorization.

What RedirectQA Actually Measures

The standard tool for studying non-verbatim memorization is entity-based question answering. You take Wikidata triples of the form (subject, relation, object), turn them into natural language questions, and measure accuracy. Prior work in this tradition, including PopQA (Mallen et al., 2023) and the popularity analyses of Kandpal et al. (2023), established a robust finding: facts about low-frequency or low-popularity entities are less likely to be memorized. This is intuitive and well-supported.

The limitation is subtle. Each entity is instantiated with a single canonical surface form. So when a model fails, you cannot tell whether the fact is genuinely absent from its parameters or whether the model simply cannot reach the fact through the specific name you happened to use.

RedirectQA is built to break this confound. The construction is elegant in its simplicity:

• Start from Wikidata factual triples, holding the relation and gold answer fixed.
• Use Wikipedia redirect information to gather the multiple surface forms that point to the same entity.
• Vary only the surface form of the subject entity in the question.

Crucially, the redirect surface forms are categorized. The paper distinguishes among:

• Alternative names and aliases (Pelé versus Edson Arantes do Nascimento, or the pseudonyms in their David Guetta / Jack Back example)
• Abbreviations
• Spelling variants including diacritics and punctuation (J. M. Coetzee versus J M Coetzee)
• Common erroneous forms, that is, typical misspellings (Stan Coveleski versus Stan Covaleski)

This categorization is what turns a curiosity into a controlled experiment. Because the fact and the gold answer are fixed within each pair, any flip in correctness can be attributed cleanly to the surface form. The illustrative cases in the paper are instructive precisely because they go in both directions: sometimes the canonical name succeeds where the redirect fails, but sometimes the redirect succeeds where the canonical name fails. The phenomenon is not simply that obscure names are harder; it is that access is genuinely indexed by the specific token sequence.

The Asymmetry That Matters

Across 13 LLMs, the central result is that prediction outcomes frequently change when only the surface form changes, and the effect is category-dependent. Models are relatively robust to minor orthographic variation: spelling differences, diacritics, and punctuation rarely flip an answer. But they are considerably less consistent under larger lexical variation, namely aliases, alternative names, and abbreviations.

This asymmetry is the most interesting finding, and it is worth unpacking what it implies about the underlying mechanism. If memorization were stored as clean semantic representations of entities, then the surface form used to invoke the entity should be largely irrelevant once the model has resolved which entity is meant. The fact that orthographic noise is tolerated but lexical substitution is not suggests something closer to the opposite picture. Robustness to spelling noise is exactly what you would expect from subword tokenization: a slightly misspelled name still shares most of its tokens with the canonical form, so the representations remain close in embedding space. An alias, by contrast, shares no tokens at all. Edson Arantes do Nascimento and Pelé occupy entirely different regions of token space, and the model's ability to bridge them depends on whether the pretraining corpus connected the two often enough.

In other words, the asymmetry is evidence that factual access is partly indexed by lexical token identity rather than by a unified semantic entity representation. The model does not always perform full entity resolution before retrieving the fact. It sometimes retrieves the fact as a near-direct association from the surface tokens themselves.

Frequency Decomposition and Cross-Surface Coupling

The second contribution is a frequency analysis that decomposes aggregate entity frequency into surface-level frequencies. The finding here is nuanced and resists a simple story. Accuracy is associated with both the frequency of the specific surface form used and the aggregate frequency of the entity, with entity frequency often contributing beyond surface frequency.

This is the part of the paper I find most theoretically important. If memorization were purely surface-specific, only the frequency of the exact name should predict accuracy. If it were fully surface-invariant, only aggregate entity frequency should matter. The data sit in between. There is cross-surface coupling: seeing an entity often under one name partially transfers to its other names, but not completely. Memorization is neither a set of isolated per-string associations nor a clean semantic store. It is an intermediate regime.

My Take

The deeper implication, which the paper gestures at but does not fully press, is that our benchmarks have a systematic blind spot that inflates apparent knowledge for well-known entities and deflates it for everyone else. Famous entities tend to have one dominant canonical form that is overwhelmingly frequent in pretraining. Benchmarks naturally query that form. So we are evaluating models under their best-case access conditions while pretending the result generalizes to the fact itself.

This matters practically. Real users do not query in canonical forms. They use abbreviations, nicknames, legal names, transliterations, and misspellings. The paper's results suggest that the gap between benchmark accuracy and deployed reliability is not random noise but a structured, category-dependent degradation that is largest exactly where it is hardest to anticipate: lexical aliases.

There is also a connection worth drawing to the hallucination literature. If a fact is present but only reachable through certain surface forms, then querying through an unreachable form is a recipe for confabulation rather than abstention. The model does not know that it knows the fact under another name, so it fills the gap. Surface-conditioned access failures and hallucination may be two views of the same phenomenon.

Limitations and Open Questions

Several caveats are worth stating honestly. The reliance on Wikipedia redirects means the surface forms studied are those Wikipedia editors found worth redirecting, which skews toward documented entities and may under-sample the truly idiosyncratic naming a real user would produce. The frequency analysis is correlational, and pretraining frequency is estimated rather than directly observed for the closed models in the set. And accuracy as a binary measure may obscure partial knowledge or calibration effects that a probability-based metric would reveal.

The open questions are inviting. Can targeted training or alias-augmented fine-tuning close the lexical access gap without retraining? Does instruction tuning or retrieval augmentation reduce surface sensitivity, or merely paper over it? Most fundamentally, can we build representations where entity resolution happens before fact retrieval, decoupling what a model knows from the name we happen to use? Until then, RedirectQA offers a reminder that a single canonical name measures access, not knowledge, and that surface-form diversity belongs in every serious evaluation of what these models actually memorize.