Measuring the Paper Itself: Local Verification Infrastructure for the Age of AI-Assisted Science Writing

The scientific publication ecosystem currently offers researchers two inadequate options for quality assurance, each failing in distinct ways. Journal gatekeeping promises to verify both integrity and contribution through peer review, yet in practice it measures prestige rather than accuracy. Peer review remains slow, demonstrably biased, and unable to detect fabricated citations even at top tier venues. GPTZero identified 100 fabricated citations in NeurIPS 2025 papers, each of which had supposedly passed review by three to five expert reviewers. Open science provides the alternative extreme: preprint servers and open repositories offer no quality assurance whatsoever. The only filter between an AI-generated manuscript and the public scientific record is the author’s personal integrity.

This binary is collapsing under the weight of AI-assisted writing, or what Sergey V. Samsonau terms "science vibe-writing" in his paper sciwrite-lint: Verification Infrastructure for the Age of Science Vibe-Writing. Vibe-writing describes the acceptance of AI-generated content because it reads fluently, without systematic verification of its claims or references. Local open-weights models exacerbate the problem; while they protect privacy (addressing concerns about cloud services extracting training data from unpublished manuscripts), smaller models hallucinate references at rates between 14% and 95%. A researcher drafting with a local 8B parameter model produces more fabricated references and unsupported claims than one using frontier cloud models, yet major ML conferences now prohibit reviewers from uploading confidential submissions to cloud AI services. This creates a structural necessity for local verification tools that can operate on consumer hardware without transmitting sensitive manuscripts to external infrastructure.

Samsonau proposes a third option: measure the paper itself. The sciwrite-lint system represents an attempt to shift quality assurance from proxy metrics (journal rank, citation counts, h-index) to direct verification of bibliographic and evidentiary integrity.

The Architecture of Automated Verification

The technical implementation of sciwrite-lint follows a structured pipeline architecture rather than an agentic approach, maintaining reproducibility and auditability while leveraging large language models strictly as semantic engines where language understanding proves necessary. The system runs entirely on the researcher’s local machine, requiring only free public databases, a single consumer GPU, and open-weights models. This local-first design addresses both privacy constraints and the practical reality that cloud-based verification of unpublished work creates unacceptable intellectual property risks.

The verification pipeline operates through several distinct stages. First, it verifies that references actually exist, checking DOIs against canonical records. Second, it cross-references each citation against the Retraction Watch database, which contains over 60,000 entries. This addresses a critical failure mode of current LLM-assisted writing: language models have no access to retraction status at generation time, meaning retracted papers circulate through the literature with the same authority as valid research. Third, the system compares metadata against canonical records to detect subtle corruptions. Fourth, and most significantly, sciwrite-lint downloads and parses cited papers to verify that they support the claims made about them. This moves beyond syntactic verification (does the DOI resolve?) to semantic verification (does the cited text substantiate the citing claim?). Finally, the system follows one level deeper, checking the bibliographies of cited papers themselves to assess the recursive integrity of the evidence chain.

Each reference receives a per-reference reliability score aggregating these verification signals. The evaluation methodology demonstrates the system’s practical utility: tested on 30 unseen papers from arXiv and bioRxiv with injected errors, sciwrite-lint achieved 98.5% recall. False positive analysis was conducted via LLM adjudication, providing a rigorous assessment of precision in real-world conditions.

The Citation Crisis and Recursive Trust

The empirical foundations of sciwrite-lint rest upon decades of evidence that citation practices were broken long before the advent of large language models. Simkin and Roychowdhury’s analysis of misprint propagation revealed that 70% to 90% of citations are copied from other reference lists without reading the original paper. Moravcsik and Murugesan found that 25% to 54% of citations contain errors, with 38.4% citing nonexistent findings, while 25% of all citations are perfunctory. Reference lists have inflated from 29 per paper in 2003 to 45 in 2019, increasing the surface area for error without proportional increases in verification.

Philosophically, this empirical reality conflicts with Merton’s normative theory of citation as symbolic payment of intellectual debt. Competing frameworks from Gilbert, Latour, and Small characterize citations instead as tools of persuasion, enrolled allies that make claims harder to challenge, or concept symbols cited not because they were read but because they represent an idea. AI systems amplify these pathologies by automating the production of plausible-sounding but unverified citations at scale.

The recursive verification capability of sciwrite-lint addresses a particularly insidious failure mode: the machine-to-machine propagation of fabricated references. When an AI-assisted paper cites another AI-assisted paper, fabricated references propagate through the citation graph with no human verification at any node. By following citations one level deeper to verify the bibliographic integrity of cited papers themselves, sciwrite-lint attempts to break these feedback loops before they compound into systematic epistemic corruption.

Beyond Integrity: Toward Adversarial Robustness and Contribution Assessment

While the integrity verification component of sciwrite-lint is evaluated and functional, the paper also proposes an experimental extension: the SciLint Score. This metric attempts to combine integrity verification with a contribution component that operationalizes five frameworks from the philosophy of science (Popper, Lakatos, Kitcher, Laudan, and Mayo) into computable structural properties of scientific arguments. This represents an ambitious attempt to automate not merely the detection of falsehoods, but the assessment of scientific value.

The logical next step for this verification infrastructure involves adversarial red-teaming. Current evaluation tests against naive errors: hallucinated DOIs, incorrect metadata, unsupported claims. A more rigorous test would involve manuscripts where fabricated citations are semantically plausible, constructed by adversaries who understand the verification pipeline’s mechanics. If claim-support verification holds against sophisticated citation forgery (where the fabricated reference appears to support the claim but actually derives from confabulated evidence), this would demonstrate robustness that peer review genuinely cannot replicate. Peer review relies on reviewer domain expertise and attention spans; it cannot systematically download and parse every cited paper to verify argumentative support.

If adversarial testing reveals failure modes, this too provides value. Understanding exactly where LLM-adjudicated verification breaks, whether through subtle misinterpretations of statistical claims or through the elision of methodological limitations, would sharpen the reliability score into a calibrated measure of epistemic uncertainty.

Limitations remain apparent. The system currently verifies only one level deep into the citation graph, which may be insufficient for highly derivative work. The reliance on open-weights models for semantic verification introduces its own hallucination risks, creating a recursive dependency problem: we use potentially fallible AI to check potentially fallible AI. The SciLint Score’s operationalization of philosophy of science frameworks remains experimental, and the computational cost of full text parsing for extensive bibliographies may limit adoption despite the consumer GPU requirement.

Conclusion

sciwrite-lint represents a necessary evolution in scientific infrastructure, acknowledging that the volume of AI-assisted writing has overwhelmed traditional quality assurance mechanisms. By shifting verification from proxies to properties of the paper itself, and by maintaining local execution to preserve privacy, the system offers a technically viable path through the dilemma between cloud-dependent verification and unregulated open science.

The broader implication extends beyond bibliographic checking. If we can automate the verification of evidence chains and operationalize philosophical frameworks for scientific contribution, we may eventually reconstruct quality assurance around measurable properties rather than journal prestige. The adversarial testing of such systems will determine whether automated verification can achieve the reliability that peer review promises but fails to deliver. The question is no longer whether AI will transform scientific writing, but whether we can build verification tools rigorous enough to make that transformation epistemically sustainable.