Multimodal and Full-Document Soundness Assessment Across Figures and Appendices
Extend automated paper reviewing to vision-language models with full-document processing so reviewers can inspect plots, diagrams, and appendices instead of truncated text.
Time
Compute level
Resources
What the paper used
Prerequisites
Scope
The Gap
In Section 4.1 (Step 1) and Appendices D.1-D.2, PDFs are converted purely to text via DeepSeek-OCR, training/evaluation inputs are prefix-truncated at 14,336 tokens (truncating four test papers), and PeerReview Bench supplies only main-paper text and figure captions without image pixels or supplementary files.
“Models receive main-paper text and available figure captions, without image pixels or supplementary files, and the same paper soundness instruction as in-domain generation.”
Limitation
Omitting figure pixels and appendix/supplementary text prevents the reviewer from detecting visual plot inconsistencies, mislabeled architectures, or flawed proofs and ablations in appendices, even when ground-truth reference concerns explicitly depend on appendix evidence.
Proposed Solution
Replace text-only inputs with a multimodal, long-context reviewing pipeline built on an open vision-language model (e.g., Qwen3-VL) that ingests rendered page images of figures/tables alongside full main-text and appendix chunks via hierarchical section-level routing or context compression beyond 64K tokens, updating both the policy and the soundness/substantiveness RMs to cross-reference visual figures and supplementary sections.
Why This Matters
Giving AI reviewers access to visual data and full appendices should eliminate blind spots on experimental plots and supplementary proofs, closing a major gap between automated and expert human reviewing.
How to Get Started
How to test it: Hypothesis: providing rendered figure/table images and full appendix context to a VLM-based PaperCritic significantly improves recall on appendix-dependent and figure-dependent reference points in PeerReview Bench without hurting precision. Baseline: text-only, main-paper-only PaperCritic-4B evaluated on PeerReview Bench IDs 1-50. Metric: recall stratified by figure/appendix dependency and overall macro F1; stop if context overhead halves rollout throughput without improving figure/appendix-dependent recall by at least 5 percentage points.
- Audit PeerReview Bench (IDs 1-50) and the 100-paper ICLR test set to tag reference review points that depend on figure visuals or appendix content.
- Extract figure/table crops and full appendix text from the OpenReview PDFs and PeerReview Bench supplementary materials.
- Fine-tune an open 4B-8B vision-language model (e.g., Qwen3-VL) on the multimodal SFT targets and adapt the soundness RM to verify claims against attached figure crops and appendix spans.
- Run GRPO with F1 reward and evaluate stratified recall and precision on figure-dependent, appendix-dependent, and main-text-only review subsets.
Suggested by AI. Adapt it to your setup.
Has This Been Done?
Existing literature includes multimodal datasets like Multimodal ArXiv and tools for scientific reasoning like Innovator-VL. However, none of the listed studies explore a long-context multimodal reviewing pipeline that ingests rendered page images alongside appendices with hierarchical section routing to assess soundness across figures and supplementary sections.
- Multimodal ArXiv: A Dataset for Improving Scientific Comprehension of Large Vision-Language Models · 2024 · 318 citations
- Innovator-VL: A Multimodal Large Language Model for Scientific Discovery · 2026 · 22 citations
We searched OpenAlex and arXiv for “multimodal vision language model scientific peer review full document appendix” and “Multimodal and Full-Document Soundness Assessment Across Figures and Appendices” and an AI compared the idea with the top 10 results. A small search can miss relevant work.
Source Paper
PaperCritic: Training AI Reviewers to Assess the Soundness of PapersHyun Ryu, Ji Yong Cho, Eunho Yang +4 more · 2026
This paper introduces paper soundness assessment as a standalone AI reviewing task that decouples objective flaw detection from subjective criteria like novelty or clarity. The authors construct PaperSound-23K, a dataset of 23,215 quality-labeled review points across 989 ICLR submissions, and train PaperCritic via reinforcement learning to jointly maximize precision (soundness and substantiveness) and recall (coverage of high-quality reference concerns).
Related Ideas
- Cross-Modal Verification and Layout Optimization for Autonomous Scientific DiagrammingPaperOrchestra: A Multi-Agent Framework for Automated AI Research Paper Writing🟠Medium
- Developing an Open-Source, Fine-Grained Automated Evaluator for Instruction-Based Image EditingFireRed-Image-Edit-1.0 Techinical Report🟡Fairly easy
- Structured Item-by-Item Multimodal Verification for Iterative 4D Scene CodingVibePrevisBench: Evaluating Agentic Coding for 4D Scene Generation in Blender🟡Fairly easy
- Domain-Expert Validation Framework for Automated Scientific IllustrationsAutoFigure-Edit: Generating Editable Scientific Illustration🔴Hard
- Optical KV-Cache for Long-Horizon Dialogue AgentsDeepSeek-OCR: Contexts Optical Compression🟡Fairly easy