Quantifying Multilingual Synergy: When Game Theory Meets Scaling Laws

Introduction

The multilingual pretraining of large language models presents a fundamental allocation problem. Given a fixed compute budget and a collection of linguistic corpora of varying sizes and qualities, how should we distribute training tokens across languages to maximize downstream performance? The answer depends critically on cross-lingual transfer, the phenomenon where representation learning on one language improves performance on another. Recent work by He et al. (2025) proposed FamilyLaw, which models transfer within language families but assumes independence across them. This assumption fails in practice. Japanese and Chinese, despite belonging to different families, exhibit substantial lexical overlap in their writing systems, creating transfer effects that family-based grouping cannot capture.

In response, Cao et al. propose ShapleyLaw: A Game-Theoretic Approach to Multilingual Scaling Laws, reframing multilingual pretraining as a cooperative game where each language acts as a player contributing to a collective payoff, defined as the reduction in test loss on a target language. By applying Shapley values from cooperative game theory, they quantify asymmetric transfer effects and derive a scaling law that predicts test loss as a function of model size, data volume, and mixture ratios. While the framework offers elegant mathematical formalism, it also exposes tensions between theoretical elegance and computational reality that merit careful examination.

The Mechanics of ShapleyLaw

The core innovation of ShapleyLaw lies in its treatment of cross-lingual transfer as a marginal contribution problem. In cooperative game theory, the Shapley value assigns to each player their average marginal contribution across all possible permutations of coalition formation. Applied to multilingual pretraining, this means evaluating how much each training language reduces the test loss on a target language when added to subsets of other languages.

Specifically, for a target language j, the transfer strength from training language i is quantified by the normalized Shapley value φ^{SV}_{i,j}. The paper demonstrates this with a 7-language setup (German, English, French, Japanese, Korean, Spanish, Chinese), revealing an asymmetric transfer matrix where Japanese contributes significantly to Chinese performance (normalized SV of 0.08) while Chinese contributes less to Japanese (0.02), reflecting the directional asymmetry in character-based lexical overlap. This stands in contrast to FamilyLaw, which would assign zero transfer between these languages due to their different family classifications.

To construct the scaling law, the authors introduce an aggregate cross-lingual transfer quantity Θ^a_j that summarizes the total transfer effect for target language j under a given mixture ratio vector p. The law takes the functional form L_j(N, D, Θ^a_j), extending standard power-law scaling to incorporate language-specific interaction terms. Empirical validation across model scales of 50M, 243M, 684M, and 1.46B parameters, with a fixed total data budget of 100B tokens, yields impressive predictive accuracy: R²_fit = 0.999 on training mixtures and R²_test = 0.990 on held-out test data for Japanese performance prediction.

Computational and Theoretical Tensions

Despite its predictive power, ShapleyLaw encounters fundamental challenges in its practical application. The computation of exact Shapley values requires evaluating model performance across all 2^K possible subsets of languages. For the seven languages in the study, this entails 128 distinct pretraining runs merely to characterize the transfer matrix. Scaling to the fifty or more languages typical of production multilingual models becomes computationally prohibitive, necessitating approximation methods such as Monte Carlo sampling or permutation truncation.

These approximations introduce systematic biases that disproportionately affect low-resource languages. The marginal contribution of a language with limited training data is inherently noisy, and sampling-based approximations may miss rare but significant interaction effects between distant language pairs. More troublingly, the application of Shapley values assumes certain axioms that multilingual transfer violates. The efficiency axiom requires that the sum of marginal contributions equals the total payoff, implying that transfer effects are conserved across the system. However, linguistic transfer is not symmetric; Japanese improves Chinese more than Chinese improves Japanese due to the directional nature of character borrowing and syntactic influence.

Similarly, the symmetry axiom states that players contributing identically to all coalitions should receive identical values. In multilingual settings, two languages might provide equivalent marginal utility when added to large English corpora, yet exhibit divergent interaction patterns when paired with typologically similar minority languages. The paper acknowledges these asymmetries in its empirical results but does not resolve the theoretical tension of applying a symmetric solution concept to an asymmetric phenomenon.

Your Take: Abstraction Versus Reality

The ShapleyLaw framework raises a deeper question about the appropriateness of cooperative game theory for modeling linguistic transfer. Cooperative games assume that players consciously coordinate to maximize collective utility, with well-defined characteristic functions mapping coalitions to payoffs. Multilingual pretraining, by contrast, involves emergent representation sharing through gradient descent rather than explicit coalition formation. The Shapley value provides a post-hoc attribution of credit, but it may not capture the mechanistic pathways through which transfer occurs, such as shared subword embeddings, overlapping syntactic patterns, or common semantic spaces.

Furthermore, the computational constraints suggest that ShapleyLaw functions more as a theoretical benchmark than a practical optimization tool. The paper demonstrates mixture ratio optimization using the derived scaling law, but this optimization relies on transfer coefficients estimated from small-scale models (50M parameters) and extrapolated to larger scales. This assumes that cross-ling