The Paradox of AI Fluency: Why Experts Fail More Often Than Novices

Introduction

The conventional wisdom surrounding expertise suggests that as users gain proficiency with a tool, their error rates should decline. A novice driver stalls the engine; an expert does not. A novice programmer produces syntax errors; an expert writes clean code on the first attempt. Yet in the domain of artificial intelligence, Christopher Potts and Moritz Sudhof present a striking counterexample in their technical report, A paradox of AI fluency. Analyzing 27,000 transcripts from the WildChat 4.8M dataset, sampled monthly from May 2023 through July 2025, the authors discover that users with high AI fluency experience significantly more failures than novices. Specifically, 64% of high-fluency user transcripts contain at least one failure indicator compared to only 24% for the lowest fluency group. This apparent contradiction dissolves upon closer inspection of what constitutes failure, revealing a nuanced relationship between expertise, task complexity, and the visibility of errors.

The Anatomy of the Paradox

The resolution to this paradox lies in the distinction between visible and invisible failures, a taxonomy developed by Potts and Sudhof in prior work. Visible failures occur when users recognize that an AI response is inadequate, incorrect, or off track, prompting them to intervene, correct, or abandon the interaction. Invisible failures, by contrast, represent a more insidious category: conversations that terminate with apparent satisfaction but in fact miss the user’s underlying goal or contain undetected errors.

The data reveals that 59% of failures among high-fluency users are visible, compared to a mere 12% among novices. This disparity stems from fundamentally different interactional modes. High-fluency users adopt what the authors term an augmentative stance, treating the AI as a collaborative thought partner rather than an oracle. These users iterate on their goals, critically assess outputs, and actively steer the conversation when the model drifts. The statistics are stark: 93% of high-fluency interactions are classified as augmentative versus under 1% for the lowest fluency users. Novices instead adopt a delegative stance, passively accepting AI outputs and plans without critical evaluation.

This behavioral difference creates a selection effect on task complexity. On a 5-point complexity scale, high-fluency users tackle tasks averaging 3.1 in complexity, while novices cluster around 1.5, a gap of 1.6 points. Experts fail more because they attempt harder problems. More importantly, their failures occur within a safety net of critical oversight. When fluent users fail, they are more likely to achieve partial recovery, adjusting course to salvage useful components of the interaction. Novices, blind to their own invisible failures, walk away with complete yet incorrect solutions.

Beyond the Interface: Expertise as Social Calibration

These findings illuminate a deeper dimension of AI literacy that extends beyond technical prompt engineering. The augmentative stance reflects an acquired understanding of AI systems as unusual social actors. Novices frequently approach conversational AI with Gricean expectations, assuming that fluent language use implies pragmatic cooperation, shared context, and mutual grounding. They treat the AI as they would a knowledgeable human colleague who implicitly understands unstated needs and conversational implicature.

Expert users, through repeated exposure, learn to abandon these assumptions. They recognize that linguistic fluency masks fragmentary world knowledge and context awareness. The augmentative behaviors identified by Potts and Sudhof, iterative refinement and explicit goal management, function as compensatory mechanisms for the AI’s lack of genuine conversational grounding. This interpretation aligns with broader research on AI agency and the importance of maintaining human oversight, even as models become more capable. The expertise gap is not merely about knowing which keywords to include in a prompt; it is about understanding the epistemic limitations of the interlocutor.

Your Take: The Pedagogy of Failure Detection

The critical question emerging from this research concerns teachability. The fluency paradox identifies failure visibility as the key mediating variable between expertise and successful outcomes, yet we do not know whether novices can be trained to recognize invisible failures without first developing full augmentative fluency. Is the ability to detect AI errors a teachable skill that can be isolated and imparted through targeted training, or is it an emergent property of deep domain expertise and extensive practice?

A targeted intervention study, exposing low-fluency users to explicit failure detection prompts and metacognitive scaffolding, would help isolate whether the augmentative stance is a modular competence or merely a byproduct of holistic expertise. If novices can learn to flag potential errors without undergoing the lengthy apprenticeship that produces high fluency, the implications for AI product design would be profound. We could engineer interfaces that make invisible failures visible through structured reflection prompts, uncertainty indicators, or forced deliberation stages.

However, if the augmentative stance requires holistic development, then current trends toward friction-free, minimalist interfaces may be actively harmful. Potts and Sudhof note that even expert users can fall into uncritical acceptance of AI responses, suggesting that the default human tendency toward delegation is strong. Product designers face a tension between immediate user satisfaction, which favors seamless, rapid interactions, and long-term success, which requires the productive friction of critical engagement. The evidence suggests that encouraging deep engagement, even at the cost of short-term convenience, leads to better outcomes. Yet achieving this without alienating users remains an unsolved design challenge.

Several limitations temper these conclusions. The WildChat dataset, while large and diverse, captures interactions with ChatGPT specifically, potentially limiting generalizability to other modalities or specialized AI tools. Additionally, the 27-month sampling period coincides with rapid model improvements, and it remains unclear whether the fluency paradox persists as base model capabilities increase. Finally, the measurement of fluency itself may be confounded with domain expertise; users who rate themselves as fluent might simply be operating in domains where they can easily spot AI errors, rather than possessing transferable AI interaction skills.

Conclusion

The work of Potts and Sudhof reframes AI literacy not as the accumulation of technical tricks or prompt templates, but as the cultivation of a critical, iterative stance toward machine-generated content. The paradox of fluency teaches us that success with AI depends less on avoiding failure than on making failure visible and recoverable. As these systems permeate professional and educational contexts, our pedagogical focus must shift from teaching users how to ask questions to teaching them how to doubt answers. The future of human-AI collaboration likely depends not on making AI more agreeable, but on making humans less credulous. Whether we can achieve this through interface design, education, or some combination remains one of the most pressing open questions in the field.