Beyond the Abstract: Full-Context Scientific Judgement via Hierarchical LLM Evaluation
A method that fine-tunes long-context language models to evaluate full-text scientific PDFs, weighing methodology and related work sections against abstracts for expert-level judgment.
Time
Compute level
Resources
What the paper used
Prerequisites
Scope
The Gap
Scientific Judge is currently trained exclusively on paper titles and abstracts. This forces the model to judge scientific potential without access to crucial methodological details, experimental validation, or the related work section, which are essential for true expert-level evaluation.
“Incorporating richer paper context, such as related work sections, may improve scientific judgement.”
Limitation
The model evaluates scientific ideas using only surface-level metadata (titles and abstracts), ignoring the depth and rigorous context of the full manuscript.
Proposed Solution
Extend the RLCF pipeline to process full-text PDFs by employing a long-context LLM. Fine-tune the reward model to extract and weigh specific sections such as 'Methodology' and 'Related Work' against the proposed 'Abstract' to identify logical gaps and methodological soundness.
The paper’s authors list this direction as future work, so they may already be pursuing it.
Why This Matters
Incorporating full text will drastically improve the accuracy of the judgement capability by grounding the AI's taste in methodological rigor rather than just catchy abstracts.
How to Get Started
How to test it: Test the hypothesis that conditioning Scientific Judge on extracted Methodology and Related Work sections alongside titles and abstracts improves impact prediction over the title-and-abstract-only SciJudge-Qwen3-4B/30B baseline. Measure position-swap pairwise accuracy on SciJudgeBench (1,000 main pairs), ICLR Metric OOD (611 pairs), and embedding-based topic-control (245 pairs), stopping if full-text context yields less than a 1 percentage point gain given the added long-context cost.
- Compile a corpus of full-text LaTeX sources and PDFs from the arXiv dump paired with expert review scores.
- Build a long-context LLM pipeline capable of parsing complete research papers including methodology and related work sections.
- Define an extractive fine-tuning objective to identify logical gaps and methodological soundness from full-text contents.
- Evaluate the fine-tuned reward model against an abstract-only baseline using correlation with human expert evaluations.
Code linked to this paper on Hugging Face: github.com/tongjingqi/AI-Can-Learn-Scientific-Taste
Suggested by AI. Adapt it to your setup.
Has This Been Done?
Papers like 'An auditable pipeline for fuzzy full-text screening in systematic reviews' and 'A survey on LLM-as-a-judge' touch upon processing full-text documents and LLM judgment pipelines. However, none of them specifically implement a hierarchical RLCF pipeline that contrasts full-text sections with abstracts for scientific preference fine-tuning.
- An auditable pipeline for fuzzy full-text screening in systematic reviews: integrating contrastive semantic highlighting and LLM judgment · 2026 · 1 citations
- A survey on LLM-as-a-judge · 2026 · 2135 citations
We searched OpenAlex for “A method that fine-tunes long-context language models to evaluate full-text scientific PDFs, weighing methodology and related work sections against abstracts for expert-level judgment.” and “Beyond the Abstract: Full-Context Scientific Judgement via Hierarchical LLM Evaluation” and an AI compared the idea with the top 10 results. A small search can miss relevant work.
Source Paper
AI Can Learn Scientific TasteJingqi Tong, Mingzhe Li, Hangcheng Li +20 more · 2026
This paper explores whether AI models can develop 'scientific taste' to evaluate and propose high-impact research ideas. The authors propose Reinforcement Learning from Community Feedback (RLCF) to train models using large-scale citation data, successfully creating an AI that generates research proposals with higher potential impact than baseline models.
Related Ideas
- Calibrating Automated Scientific Literature Review Evaluators Against Structural LLM BiasPaperOrchestra: A Multi-Agent Framework for Automated AI Research Paper Writing🟡Fairly easy
- Meta-Analytic Agents with Verification-Aware AggregationProbing Scientific General Intelligence of LLMs with Scientist-Aligned Workflows🟡Fairly easy
- Retrieval-Augmented Soundness Verification Against External LiteraturePaperCritic: Training AI Reviewers to Assess the Soundness of Papers🔴Hard
- Domain-Expert Validation Framework for Automated Scientific IllustrationsAutoFigure-Edit: Generating Editable Scientific Illustration🔴Hard
- Automated Reproducibility Auditing for AI-Generated Scientific BaselinesEvoScientist: Towards Multi-Agent Evolving AI Scientists for End-to-End Scientific Discovery🟡Fairly easy