Correct Prediction, Wrong Steps? Why LLMs Need Consensus, Not Just Correct Answers

The Deception of Final Answers

Large language models have mastered the art of passing tests without understanding the material. In their paper "Correct Prediction, Wrong Steps? Consensus Reasoning Knowledge Graph for Robust Chain-of-Thought Synthesis," Ling et al. expose a troubling phenomenon: LLMs frequently arrive at correct final answers through logically flawed, hallucinated, or incoherent reasoning steps. This observation carries profound implications for how we train and evaluate reasoning systems, particularly as these models increasingly underpin high-stakes applications in clinical decision-making, financial analysis, and legal consultation.

The authors categorize reasoning failures into two distinct classes that often intermingle within single datasets. Step Internal Flaws encompass logical errors, factual hallucinations, and inconsistent conclusions within individual reasoning steps. Step-wise Flaws refer to structural pathologies in the reasoning trajectory, specifically overthinking (redundant, circular steps) and underthinking (omitted necessary intermediate deductions). Crucially, the distribution of these flaws varies by sample; some instances suffer from logical hallucinations while others suffer from missing steps, creating a complex mixture that defies single-type remediation strategies.

What makes this investigation particularly noteworthy is the authors' empirical demonstration that providing the correct final answer to the model yields no consistent improvement in reasoning quality. Across four different LLMs evaluated on PRMBench and ROSCOE benchmarks, answer-conditioning failed to produce higher quality reasoning traces. This finding strikes at the heart of current reinforcement learning approaches that optimize for outcome correctness rather than process validity, suggesting that reward signals based solely on final answers may reinforce precisely the kind of shortcutting behavior that produces "correct prediction, wrong steps" scenarios.

The Architecture of Consensus

Rather than conditioning on labels, CRAFT (Consensus Reasoning knowledge graph Aggregation for Flaw-aware Traces synthesis) leverages cross-trace structural consensus to synthesize robust reasoning paths. The framework operates through three interconnected modules that transform multiple candidate traces into a single high-quality reasoning chain.

Module I: Diverse Trace Generation and Consensus Term Extraction. CRAFT begins by rolling out K candidate traces per problem. From this ensemble, the system identifies consensus terms using TF-IRF (Term Frequency-Inverse Reasoning Frequency) weighting, a metric that identifies concepts appearing frequently within a specific problem's candidate traces but rarely across the broader reasoning corpus. This step effectively surfaces the "common knowledge" shared across multiple independent attempts at the same problem.

Module II: Consensus RKG Construction and Anomaly Filtering. Here, the innovation becomes structural. Each candidate trace is converted into a Reasoning Knowledge Graph (RKG) where reasoning steps constitute nodes and logical dependencies form directed edges. These individual graphs are aggregated into a consensus RKG, denoted as G*. The authors then apply z-score filtering to detect and remove structurally anomalous steps, identifying outliers based on their deviation from the consensus topology. This approach treats reasoning quality as a structural property of the intersection of multiple reasoning paths rather than a feature of any single trace.

Module III: Topology-Guided Trace Synthesis. Finally, CRAFT generates a novel reasoning trace by traversing the consensus RKG in topological order. Unlike selection-based methods such as the concurrent MGRS approach, which simply returns the highest-scoring original candidate trace, CRAFT synthesizes an entirely new reasoning path that represents the structural consensus of the ensemble. This synthesis capability distinguishes CRAFT from verifier-based approaches like process reward models or AgentAuditor, which rely on external judges to score individual traces.

Beyond Accuracy: Structural Fidelity and Its Limits

The empirical results demonstrate that CRAFT achieves over 10% improvement in label-prediction accuracy across logical and mathematical reasoning benchmarks, while simultaneously improving trace quality across multiple evaluation dimensions. However, the implications extend beyond these quantitative gains.

From a training perspective, CRAFT offers a pathway to generate higher quality synthetic data for model distillation. Current approaches like Vicuna, distilled from GPT outputs, potentially inherit reasoning errors embedded in their training traces. By applying consensus-based synthesis, we might create training datasets where process supervision emerges organically from structural agreement rather than expensive human annotation or brittle automated verifiers.

The method also suggests a fundamental shift in how we conceptualize reasoning verification. Traditional approaches rely on monolithic judgment, either through human annotators or trained verifiers examining single traces. CRAFT introduces a form of structural democracy, where reasoning validity emerges from the intersection of multiple cognitive paths. This parallels ensemble methods in machine learning but operates at the level of reasoning topology rather than prediction aggregation.

Nevertheless, the approach carries inherent limitations that warrant careful consideration. The consensus assumption, that structural agreement implies correctness, may fail in specific pathological scenarios. If multiple candidate traces share systematic biases or common misconceptions, the consensus RKG will amplify these errors rather than filter them. The method also assumes that valid reasoning paths share significant structural overlap; novel or creative reasoning strategies that deviate from common patterns might be erroneously filtered as anomalies. Additionally, the computational cost of generating K traces and constructing graph structures for each problem may limit scalability for real-time applications.

Forward Trajectories

CRAFT points toward a future where LLM reasoning is evaluated and trained based on process fidelity rather than outcome accuracy. The finding that correct answers provide no guidance for correct steps suggests that current RL-based training regimes may be optimizing the wrong objective function. As the field moves toward more capable reasoning systems, we may need to abandon the efficiency of outcome-based rewards in favor of process-based supervision that verifies each step's validity.

Several open questions emerge from this work. How does the optimal number of candidate traces K vary with problem difficulty? Can the consensus RKG approach be extended to multi-modal reasoning or tool-augmented environments where reasoning steps interact with external APIs? Most importantly, how can we guard against the consensus trap, where models converge on plausible but incorrect reasoning patterns that happen to agree with each other?

The paper ultimately suggests that robust reasoning requires structural redundancy. Just as biological systems rely on multiple checks and balances, artificial reasoning may require the synthesis of multiple cognitive paths to ensure fidelity. As we deploy LLMs in increasingly consequential domains, the distinction between correct predictions and correct steps may determine whether these systems serve as reliable cognitive partners or sophisticated pattern matchers prone to confident error.