The Forensics Gap: Why We Cannot Catch the Academic Fakes We Generate

Introduction

There is a growing asymmetry at the heart of generative AI: our ability to fabricate convincing images has outpaced our ability to detect those fabrications. Nowhere is this asymmetry more consequential than in academic publishing, where falsified microscopy images, manipulated Western blots, and synthetic medical scans can quietly corrupt the scientific record. A new benchmark, AEGIS: A Holistic Benchmark for Evaluating Forensic Analysis of AI-Generated Academic Images, from the BUPT Reasoning Lab, quantifies just how wide this gap has become. The headline numbers are sobering. GPT-5.1, one of the strongest multimodal models available, reaches only 48.80% overall performance on academic forgery analysis. Expert localization models barely manage 30.09% IoU when asked to pinpoint where manipulation occurred.

In this article I want to unpack what AEGIS actually measures, why academic imagery is uniquely hard to audit, and what the structure of the results tells us about where forensic research should go next.

What AEGIS Measures and Why It Is Different

Most prior image forensics benchmarks operate on generic content: faces, natural scenes, and social media photos. AEGIS targets a domain that is structurally and semantically denser. It spans seven major academic categories and 39 fine-grained subtypes, including microscopy, medical imaging, charts, diagrams, stained micrographs, and physical objects. This matters because the visual statistics of a stained cellular micrograph differ enormously from those of a celebrity face. A forensic model trained to spot the telltale artifacts of generated faces has little reason to generalize to the texture of fluorescence microscopy or the layout regularities of a scientific diagram.

The benchmark is also distinctive in how broadly it samples the generative side. AEGIS instantiates forgeries using 25 state-of-the-art generative models across four forgery strategies:

• Text Constraint Fabrication: generating an image from a textual specification
• Image Inference Forgery: synthesizing plausible content by inference from existing context
• Targeted Region Restoration: inpainting or restoring a region to remove evidence
• Targeted Region Editing: altering a specific region to introduce false content

The last two are particularly relevant to real academic misconduct, where researchers often do not fabricate a whole figure but selectively duplicate, remove, or edit a band, a cell cluster, or a region of interest. Capturing this partial, localized character of academic forgery is something earlier benchmarks largely ignored.

Finally, AEGIS does not reduce forensics to a binary real-or-fake question. It defines four complementary evaluation tasks:

1. Forgery Scope Discrimination (is the image authentic, partially forged, or fully forged)
2. Textual Artifact Recognition (detecting corrupted or nonsensical text, a common generative tell)
3. Manipulation Classification (identifying which forgery strategy was used)
4. Tampering Pinpointing (localizing the manipulated region via bounding box)

This decomposition mirrors how a careful human reviewer would actually proceed: first deciding whether something is wrong, then characterizing what kind of manipulation occurred, then grounding the suspicion to a specific region. The benchmark comprises over 20,000 forensic questions and evaluates 25 leading MLLMs, nine expert detectors, and one unified understanding-and-generation model.

The Results Reveal a Capability Split, Not a Single Weakness

The most interesting finding in AEGIS is not simply that models perform poorly. It is how they fail, and the way failure distributes across model families.

Consider the two strongest signals in the data:

• MLLMs lead on Textual Artifact Recognition at 84.74% accuracy. This is intuitive. Large multimodal models have strong language priors, so they readily notice when text inside a figure is garbled, misspelled, or semantically incoherent, a classic artifact of diffusion-based generation.
• Expert detectors peak at 79.54% accuracy on binary authenticity detection. These specialized models, often trained on frequency-domain or patch-level statistical cues, are better at catching the low-level fingerprints of synthesis that MLLMs do not reason about explicitly.

The implication is that no single model sees the whole picture. The strengths are complementary rather than overlapping. MLLMs reason about semantics; expert detectors model statistical artifacts. A forensic system that combines both is likely to outperform either alone, which is precisely why hybrid approaches that pair MLLMs with expert models have started to appear in the literature.

But complementarity does not rescue the overall picture. The aggregate numbers remain weak. Across the 25 generative models, 11 yield average forensic accuracy below 50%, meaning a coin flip would do about as well against nearly half of the generators tested. And localization, the task that matters most for actionable peer review, is where everything collapses. An IoU of 30.09% means that even when a model suspects manipulation, it cannot reliably tell a reviewer where to look. For a peer reviewer or journal integrity officer, a vague suspicion without spatial grounding is of limited practical value.

My Take: The Gap Is Structural, Not Just a Matter of Scale

It is tempting to read these numbers and assume the solution is more data and bigger models. I think that conclusion is too easy, and AEGIS gives reasons to doubt it.

First, the adversarial dynamics are fundamentally asymmetric. Generative models are trained directly to minimize perceptual and distributional differences from real images. Forensic detectors are trying to exploit residual differences that the generators are actively optimizing away. Every improvement in generation quality erodes the very signal that detection depends on. This is not a gap that closes with scale; if anything, scaling generative models widens it. The AEGIS finding that forensics lag behind generative advances is a structural prediction, not a transient observation.

Second, academic forensics demands knowledge-intensive reasoning that pure artifact detection cannot provide. A duplicated cell region or an implausibly clean Western blot band may be statistically unremarkable yet biologically suspicious. Catching this requires domain knowledge about what a real micrograph or gel should look like. This is where MLLMs in principle have an advantage over expert detectors, since they can reason about plausibility, not just pixels. But the AEGIS results suggest current MLLMs do not yet wield this knowledge with enough precision, especially for spatial grounding.

Third, the localization failure points to a deeper limitation in multimodal grounding. MLLMs are notoriously weak at precise bounding-box prediction even on natural images. Asking them to localize subtle, partial forgeries in dense scientific figures stresses exactly the capability they are worst at. I suspect localization, not detection, will be the binding constraint on practical forensic systems for the next several model generations.

Limitations and Honest Caveats

AEGIS is a diagnostic benchmark, and it should be read as such. A few caveats are worth noting:

• Generated forgeries are not identical to real misconduct. Real fraudulent figures often involve crude manual edits, copy-paste duplication, and contrast manipulation, not just generative synthesis. AEGIS captures the generative threat well but may underrepresent older, simpler manipulation tactics.
• Benchmark performance may not transfer to deployment. Models evaluated with structured prompts and well-defined tasks may behave differently when embedded in a real review pipeline with noisy, heterogeneous submissions.
• The 25 generative models are a moving target. By the time detectors catch up to this generation, the next will have arrived, which is precisely the gap the paper documents.

Conclusion and Open Questions

AEGIS makes a convincing case that academic image forensics is not a solved problem and may not be solvable by any single model family. The headline numbers, 48.80% for GPT-5.1 and 30.09% IoU for expert localization, are a warning that our detection infrastructure is unprepared for the synthetic content already circulating.

The most productive path forward, in my view, is not a single better detector but orchestrated systems that combine the semantic reasoning of MLLMs with the statistical sensitivity of expert detectors, while explicitly addressing the localization bottleneck. Beyond that, the field needs to grapple with harder questions: Can forensic models maintain reliability against generators they have never seen? Can spatial grounding be made trustworthy enough to support real editorial decisions? And given the structural asymmetry between generation and detection, should scientific publishing also invest in provenance and watermarking rather than relying on post-hoc forensics alone?

The uncomfortable truth remains: we can generate fakes far better than we can catch them. Benchmarks like AEGIS are valuable precisely because they refuse to let us forget it.