Retrieval-Augmented Soundness Verification Against External Literature
Augment automated paper reviewers with scholarly literature retrieval so they can verify novelty, baseline coverage, and prior-work claims against published research.
Source paper
Hyun Ryu, Ji Yong Cho, Eunho Yang, Moontae Lee, June Yong Yang, Sean Welleck, Carolyn Rose
Generated by AI. The paper’s authors did not write these.
Augment automated paper reviewers with scholarly literature retrieval so they can verify novelty, baseline coverage, and prior-work claims against published research.
Add phrase-diversity regularization and symbolic math verification to reinforcement learning so automated reviewers stop collapsing into repetitive template questions and hallucinated notation errors.
Build a tool-augmented claim verifier that checks equations and table statistics programmatically, replacing single-pass language model reward models that weakly correlate with human judgments.
Reward automated reviewers for stress-testing every core paper claim rather than matching a static reference list, preventing valid novel critiques from being penalized.
Extend automated paper reviewing to vision-language models with full-document processing so reviewers can inspect plots, diagrams, and appendices instead of truncated text.
This paper introduces paper soundness assessment as a standalone AI reviewing task that decouples objective flaw detection from subjective criteria like novelty or clarity. The authors construct PaperSound-23K, a dataset of 23,215 quality-labeled review points across 989 ICLR submissions, and train PaperCritic via reinforcement learning to jointly maximize precision (soundness and substantiveness) and recall (coverage of high-quality reference concerns).
The framework builds PaperSound-23K from OpenReview ICLR 2020-2023 papers and synthetic LLM reviews, using GPT-5.2 and GPT-5.4 to extract main claims, segment review points, and label soundness, claim linkage, and severity. PaperCritic (Qwen3-4B-Instruct-2507 and Qwen3-8B) is first supervised fine-tuned on sound and substantive reference points arranged in a claim-diversity-aware order, then optimized via GRPO online reinforcement learning using a review-level F1 reward computed from a distilled Qwen3-4B soundness reward model, a Qwen3.6-35B-A3B substantiveness reward model, and a Qwen3.6-35B-A3B reference matcher.