Skip to content

Source paper

PaperCritic: Training AI Reviewers to Assess the Soundness of Papers

Hyun Ryu, Ji Yong Cho, Eunho Yang and 4 more

Hyun Ryu, Ji Yong Cho, Eunho Yang, Moontae Lee, June Yong Yang, Sean Welleck, Carolyn Rose

arXiv 2611.17834 · 202631 upvotes on Hugging Face

Research ideas from this paper (5)

Generated by AI. The paper’s authors did not write these.

Retrieval-Augmented Soundness Verification Against External Literature

Augment automated paper reviewers with scholarly literature retrieval so they can verify novelty, baseline coverage, and prior-work claims against published research.

PaperCritic: Training AI Reviewers to Assess the Soundness of Papers
🔴Hard🔬Significant ResearchMultiple GPUsNo close match found#LLMs#Retrieval & memory

Mitigating RL-Induced Style Narrowing and Hallucinated Notation Critiques

Add phrase-diversity regularization and symbolic math verification to reinforcement learning so automated reviewers stop collapsing into repetitive template questions and hallucinated notation errors.

PaperCritic: Training AI Reviewers to Assess the Soundness of Papers
🟡Fairly easy💡Moderate InnovationMultiple GPUsNo close match found#LLMs#Reinforcement learning

Programmatic and Decomposed Soundness Verification Without Frontier Distillation Bottlenecks

Build a tool-augmented claim verifier that checks equations and table statistics programmatically, replacing single-pass language model reward models that weakly correlate with human judgments.

PaperCritic: Training AI Reviewers to Assess the Soundness of Papers
🟠Medium🔬Significant ResearchMultiple GPUsNo close match found#LLMs#Reinforcement learning

Claim-Coverage and Iterative Reference Expansion to Overcome Incomplete, LLM-Skewed Recall

Reward automated reviewers for stress-testing every core paper claim rather than matching a static reference list, preventing valid novel critiques from being penalized.

PaperCritic: Training AI Reviewers to Assess the Soundness of Papers
🟠Medium💡Moderate InnovationMultiple GPUsNo close match found#LLMs#Reinforcement learning

Multimodal and Full-Document Soundness Assessment Across Figures and Appendices

Extend automated paper reviewing to vision-language models with full-document processing so reviewers can inspect plots, diagrams, and appendices instead of truncated text.

PaperCritic: Training AI Reviewers to Assess the Soundness of Papers
🔴Hard🔬Significant ResearchMultiple GPUsNo close match found#Multimodal#Retrieval & memory

AI summary

This paper introduces paper soundness assessment as a standalone AI reviewing task that decouples objective flaw detection from subjective criteria like novelty or clarity. The authors construct PaperSound-23K, a dataset of 23,215 quality-labeled review points across 989 ICLR submissions, and train PaperCritic via reinforcement learning to jointly maximize precision (soundness and substantiveness) and recall (coverage of high-quality reference concerns).

How it works (AI summary)

The framework builds PaperSound-23K from OpenReview ICLR 2020-2023 papers and synthetic LLM reviews, using GPT-5.2 and GPT-5.4 to extract main claims, segment review points, and label soundness, claim linkage, and severity. PaperCritic (Qwen3-4B-Instruct-2507 and Qwen3-8B) is first supervised fine-tuned on sound and substantive reference points arranged in a claim-diversity-aware order, then optimized via GRPO online reinforcement learning using a review-level F1 reward computed from a distilled Qwen3-4B soundness reward model, a Qwen3.6-35B-A3B substantiveness reward model, and a Qwen3.6-35B-A3B reference matcher.