Prediction-Powered Inference and the Difference Estimator: Uncovering Survey Sampling Roots

The proliferation of large language models as measurement instruments has reignited a classical statistical problem. Researchers now routinely use models to code text, classify images, or score essays, generating predictions across massive datasets. Yet these predictions contain errors. When systematic, such errors propagate into biased treatment effect estimates, invalid policy conclusions, and unreliable scientific findings. Prediction-Powered Inference (PPI), introduced by Angelopoulos et al. (2023a), offers a statistical framework for correcting these biases by combining inexpensive model predictions with a small set of expensive, gold standard human labels. The method has gained substantial traction in machine learning circles, with extensions to clinical trials, genomics, and causal inference.

However, a careful inspection reveals that PPI is not, strictly speaking, a novel invention of the machine learning era. In the paper "PPI is the Difference Estimator: Recognizing the Survey Sampling Roots of Prediction-Powered Inference," Reagan Mozer demonstrates that the estimators underlying PPI are algebraically identical to classical tools from the survey sampling literature. Specifically, the standard PPI estimator corresponds to the difference estimator developed by Cassel et al. (1976), while the extended PPI++ estimator is equivalent to the generalized regression (GREG) estimator of Särndal et al. (2003). This recognition is not merely an academic exercise in priority. It illuminates the conditions under which these estimators achieve validity, highlights underutilized methodological extensions, and reveals opportunities for cross disciplinary collaboration that could reshape how we validate automated measurement systems.

The Algebraic Equivalence

To understand the correspondence, consider the setup common to both frameworks. A researcher wishes to estimate a population mean θ. For a large pool of N units, machine learning predictions Ŷi are available. For a small, labeled subset of n units, both the true outcome Yi and the prediction Ŷi are observed. The prediction error for labeled units is defined as ei = Yi - Ŷi.

The PPI estimator combines the average prediction across all units with a correction term derived from the labeled sample:

θ̂PPI = (1/N)ΣŶi + (1/n)Σ(Yi - Ŷi)

The second term, termed the "rectifier" in the PPI literature, adjusts for systematic bias in the model predictions. If the machine systematically overpredicts by a constant amount, this term subtracts that bias, yielding an unbiased estimate regardless of the model's accuracy.

In the survey sampling tradition, the difference estimator (Cassel et al., 1976) takes the form:

θ̂diff = (1/N)ΣŶi + (1/n)Σs(Yi - Ŷi)

where s denotes the sampled units. The structure is identical. Both estimators leverage auxiliary information (the predictions) to improve precision while using the sample error to maintain unbiasedness. This property, known as model assisted estimation, ensures that inference remains valid even when the working model is misspecified.

The correspondence extends to the optimized variant. PPI++ introduces a tuning parameter λ that controls the weight given to the predictions:

θ̂PPI++ = (1/n)ΣYi + λ[(1/N)ΣŶi - (1/n)ΣŶi]

Setting λ = 1 recovers the standard PPI estimator. The optimal λ* minimizes variance and depends on the correlation between predictions and outcomes. This is precisely the generalized regression estimator (GREG) of Särndal et al. (2003), which uses regression residuals to calibrate the auxiliary information. Both frameworks recognize that when predictions are accurate, we should weight them heavily; when they are noisy, we should shrink toward the labeled data mean.

Points of Divergence

Despite the algebraic identity, the two frameworks diverge in important conceptual and practical dimensions. These distinctions illuminate where inferential claims require additional care.

First, the mode of inference differs fundamentally. Survey