The Calibration Problem: Why AI Literacy Alone Cannot Fix Overreliance in Educational Settings

The integration of generative AI into higher education has moved beyond experimentation into routine practice. Students now regularly encounter large language models as tutors, debugging assistants, and writing collaborators. Yet this proliferation creates a fundamental pedagogical tension. While AI assistance can reduce friction in problem solving and provide timely explanations, it also risks short circuiting the cognitive processes essential for deep learning. A new study by Griffin Pitts, Neha Rani, and Weedguet Mildort, titled Trust and Reliance on AI in Education: AI Literacy and Need for Cognition as Moderators, offers crucial insights into this tension. Their work reveals that the relationship between student trust and appropriate use of AI is neither linear nor easily corrected through technical education alone.

The Mechanics of Misplaced Trust

To understand how students actually use AI assistance, Pitts and colleagues designed a carefully controlled experiment with 432 undergraduate participants completing Python output prediction problems. This methodology merits close attention because it moves beyond self reported attitudes to behavioral measures. Students interacted with an AI chatbot that provided recommendations and explanations, but with a critical twist: some suggestions were accurate while others were intentionally misleading. This design allowed the researchers to operationalize appropriate reliance as a behavioral construct; specifically, the degree to which students accepted correct recommendations while simultaneously rejecting incorrect ones.

The results challenge simplistic assumptions about trust calibration. Rather than finding that higher trust predicts better outcomes, or that trust operates independently of accuracy detection, the researchers identified a non linear relationship in which elevated trust correlated with diminished appropriate reliance. Students who reported greater trust in the AI assistant showed weaker discrimination between correct and incorrect outputs. This suggests that trust, once established, functions less as a calibrated response to accuracy and more as a heuristic that bypasses analytical evaluation.

This finding aligns with broader research on automation bias and dual process theory. When students engage with AI tools, they face an unconscious cognitive cost benefit calculation. Verification requires effortful System 2 processing; analytical, slow, and demanding. Acceptance requires only System 1 processing; intuitive, fast, and effortless. Under the time pressures and cognitive demands typical of programming problem solving, high trust appears to tip this balance toward uncritical acceptance. The AI outputs, often fluent and confident in tone, exert a persuasive influence that overwhelms the motivation to verify, even when verification would reveal errors.

The Limited Protective Power of AI Literacy

Given these risks, many educators and policymakers have proposed AI literacy as the primary intervention. The assumption follows that if students understand how large language models function, including their propensity for hallucination and their statistical nature, they will naturally exercise appropriate skepticism. Pitts and colleagues explicitly test this assumption by examining whether AI literacy and need for cognition moderate the relationship between trust and reliance behavior.

Their findings complicate the literacy narrative. While both AI literacy and need for cognition did moderate the negative relationship between trust and appropriate reliance, they did not eliminate it. Students with higher AI literacy or greater intrinsic motivation for effortful thinking showed somewhat better discrimination at high trust levels, but the fundamental curve remained intact. High trust still degraded performance. This is a crucial distinction. Moderation here implies that literacy shifts the trajectory of decline rather than flattening the curve entirely.

The study also controlled for programming self efficacy and programming literacy, confirming that these domain specific competencies did not explain away the effects. This suggests that the calibration problem transcends subject matter expertise. A student can understand Python syntax, comprehend transformer architectures, and still accept an incorrect AI suggestion because their trust in the system operates through affective and heuristic channels rather than analytical ones.

Epistemic Habits versus Tool Knowledge

These results point toward a more nuanced understanding of what constitutes effective preparation for AI augmented learning. The evidence suggests that AI literacy, conceived as technical knowledge about model capabilities and limitations, functions as a weak buffer against overreliance. What appears necessary instead is the cultivation of epistemic habits; sustained practices of verification, metacognitive monitoring, and deliberate evaluation of knowledge claims regardless of their source.

The distinction matters for instructional design. Current interventions often focus on teaching students about training data, hallucinations, and prompt engineering. While valuable, this tool knowledge does not address the automaticity with which high trust triggers acceptance. The students in this study who exhibited appropriate reliance were not merely those who knew more about AI. They were those who, despite trusting the system, maintained the cognitive disposition to interrogate its outputs. This suggests that education should prioritize reflective evaluation as a procedural skill rather than treating critical thinking as a natural byproduct of technical knowledge.

Furthermore, the non linear nature of the trust reliance relationship implies that the danger zone lies not in moderate familiarity but in confident dependence. Students who view AI as a fallible tool among many resources may maintain healthy skepticism. Those who develop strong trust, even when that trust stems from positive prior experiences, risk sliding into automation bias. The intervention target, therefore, should focus on sustaining productive cognitive friction. Systems and pedagogies that make verification easier than acceptance, or that require explicit justification for AI adoption, may prove more effective than lectures on neural network architectures.

Limitations and Open Questions

As with any experimental study, this work has boundaries that shape its interpretation. The programming context, while practically significant, may exhibit different dynamics than humanities writing or quantitative problem solving. Code has objective correctness criteria, which theoretically should make error detection easier than evaluating nuanced arguments. Yet even in this relatively structured domain, students failed to calibrate appropriately. This suggests the findings may represent a lower bound for overreliance in more ambiguous domains.

The cross sectional design also limits causal claims about the development of trust over time. The study captures trust at a single moment, yet educational relationships with AI tools develop through repeated interactions. Longitudinal research is needed to determine whether the observed relationship strengthens, weakens, or transforms as students accumulate experience with AI errors. Additionally, the intentionally misleading suggestions, while necessary for experimental control, may differ qualitatively from the subtle errors AI systems typically produce in the wild.

Conclusion

The work of Pitts, Rani, and Mildort provides empirical grounding for skepticism about quick fixes to AI overreliance. Their demonstration that higher trust degrades discrimination between correct and incorrect outputs, even among literate learners, forces a reconsideration of educational priorities. Rather than assuming that knowledge about AI systems will automatically translate into appropriate use, educators must recognize that trust operates through cognitive pathways that bypass analytical evaluation.

Moving forward, the field needs interventions that target epistemic vigilance as a habit rather than a concept. This might involve structured verification protocols, reflective prompts embedded in AI interfaces, or assessment designs that reward the process of critique rather than just the final product. The question is no longer simply how to teach students about AI, but how to preserve their capacity for independent judgment when assisted by increasingly fluent and confident systems. Until we address this calibration problem, literacy alone will remain insufficient protection against the seductive efficiency of automated assistance.