LaP-Forensics: Latent-Pixel Consistency Guided Multimodal Reasoning for Deepfake Detection
Authors & Institutions
Can Wang
The Hong Kong Polytechnic University, Hong Kong, China
Yuhao Wang
University College London, United Kingdom
Yushe Cao
Tsinghua University, China
Canran Xiao
Sun Yat-sen University, China
Fei Shen
National University of Singapore, Singapore
What Problem It Solves
LaP-Forensics addresses cross-generator image detection and pixel-level artifact localization with an explicit compatibility reference. It aims to make structured where-what-why reasoning consult that evidence, while acknowledging that reference use does not by itself prove every sentence in a free-form explanation.
Key Result
On UniversalFakeDetect, the image head reaches 97.23% accuracy on GANs, 71.30% on Deepfakes, 92.18% on Diffusion, and 98.98% on CRN. Under the official SynthScars protocol, human-artifact localization reaches 61.40 mIoU and 66.00 F1. In the controlled split, removing the residual lowers mIoU from 72.19 to 61.83 and F1 from 63.62 to 41.71; replacing the residual with zero or a donor map also materially shifts the output.
Abstract
Recent generative models can produce images with few obvious visual artifacts, weakening detectors and explanations that rely only on surface appearance. We present LaP-Forensics, a multimodal framework that augments RGB semantics with reconstruction-based forensic evidence. A frozen Stable Diffusion DDIM inversion-reconstruction model provides a fixed reconstruction reference, and its residual map measures local compatibility with that reference. Independent projectors encode the RGB image and residual map before a structured Where-What-Why model predicts a textual analysis and an artifact mask.Supervised fine-tuning is followed by Group Relative Policy Optimization (GRPO), whose reward combines mask overlap with output-structure and evidence-reference terms. These text-side terms encourage the model to refer to the consistency map but do not constitute a verifier of free-form textual truth. A separate image-level head fuses RGB and DDIM-residual class features. Experiments show cross-generator detection on UniversalFakeDetect and competitive artifact localization on the official SynthScars benchmark. Controlled cue-construction, inversion-horizon, component, reward-term, and counterfactual analyses support the utility of the residual stream under the evaluated settings, while free-form textual faithfulness and reliability under post-processing remain open limitations.
Research Starting Point
As generated images lose obvious surface defects, an RGB-only detector or vision-language explanation can sound persuasive without pointing to stable forensic evidence. For enterprise review, a useful system should separate the detection score, spatial localization, and textual account while showing what additional signal actually changes the result.
Method
A frozen Stable Diffusion DDIM inversion-reconstruction process produces a residual map between the input and reconstruction. Separate projectors encode RGB semantics and residual evidence; a multimodal branch generates structured text and a mask, while a separate image head fuses class features for detection. Supervised fine-tuning is followed by GRPO rewards for mask overlap, output format, and evidence references, with official benchmark tests kept separate from a smaller controlled ablation split.
Paper Summary
The reconstruction residual is a useful second forensic channel and improves localization measurably, but it is not a calibrated manipulation map or a source-attribution signal. Teams should value the clean separation of detection, localization, and explanation, then independently test post-processing robustness and textual faithfulness before using the generated rationale in review workflows.