End-to-End Benchmarking of Native Scientific Perception Within Interactive Agentic Workflows
Build a benchmark suite pairing raw molecular, sequence, and spectral inputs with multi-step software tools to evaluate whether multimodal agents actually use scientific perception during discovery.
Time
Compute level
Resources
What the paper used
Prerequisites
Scope
The Gap
In Sections 2.2 and 3.2, the paper trains modality-specific projectors (Sage Sci-Perception) for molecular graphs, biological sequences, and spectra using SFT and GRPO on standalone scientific reasoning and auxiliary evidence-querying tasks. However, the formal 33-benchmark evaluation in Section 4 (Table 2) tests only text, code, and GUI/visual tasks (e.g., OfficeQA Pro, MLE-bench Lite, OSWorld-Verified), while Section 5 presents only qualitative case studies (CT reconstruction, 3D modeling, Fiji/EEGLAB/QGIS, and HTML physics demos) that do not invoke the scientific encoders.
“A next step is to test these capabilities together in complete scientific workflows, checking whether the model selects relevant evidence, adapts its actions to feedback, and produces scientifically correct results.”
Limitation
Because scientific perception adaptation and agentic tool use are evaluated separately, there is no quantitative evidence showing whether projected scientific embeddings remain usable, accurate, and grounded when combined with long-horizon tool calling, software manipulation, and visual feedback in real scientific workflows.
Proposed Solution
Construct a containerized benchmark suite in the Harbor format used by the authors that supplies both raw structured inputs (molecular graphs, protein/DNA/RNA sequences, spectra) and domain software tools (e.g., RDKit, Biopython, 3D Slicer, EEGLAB, Fiji) in the same episode. Design tasks requiring the agent to inspect native scientific embeddings to formulate hypotheses, execute software transformations or simulations, compare experimental outputs against source measurements, and pass deterministic artifact and scientific-correctness checks, evaluating both full-interface and text-only ablated baselines.
The paper’s authors list this direction as future work, so they may already be pursuing it.
Why This Matters
If successful, this benchmark and evaluation protocol would verify whether native scientific encoders genuinely improve multi-step scientific execution over text-only or image-only representations and expose cross-modal interference during agentic rollouts.
How to Get Started
How to test it: Hypothesis: Native scientific embeddings outperform text/SMILES-only or rendered-plot baselines when agents must localize subtle structural features and invoke domain software tools. Baseline: Sage S2.1 Flash with Sage Sci-Perception disabled (text/image inputs only) on a 100-task Harbor scientific workflow benchmark measuring task pass rate and scientific-correctness F1; stop if native scientific embeddings fail to beat the ablated baseline by at least 5 percentage points across modalities.
- Inspect the released InternLM/Sage-S2.1-Flash and InternLM/Sage-SciPerception repositories to confirm how scientific tokens are formatted alongside tool-use prompts.
- Curate 50-100 multi-step tasks from public cheminformatics, protein-engineering, and spectroscopy datasets requiring both raw scientific inputs and executable Python/CLI tools packaged in Harbor format.
- Implement deterministic verifiers checking both final scientific accuracy and intermediate tool outputs.
- Run zero-shot evaluations comparing Sage S2.1 Flash with and without Sage Sci-Perception inputs.
Suggested by AI. Adapt it to your setup.
Has This Been Done?
Papers such as SciAgentGym and MatToolBench focus on multi-step scientific tool-use and benchmarking, but none of the listed works examine containerized end-to-end evaluation combining native structured inputs and software tools using the Harbor format with full-interface vs text-only ablated baselines.
- SciAgentGym: Benchmarking Multi-Step Scientific Tool-use in LLM Agents · 2026 · 22 citations
- MatToolBench: Benchmarking Multimodal Agents in Real-World Materials Science Workflows · 2026 · 0 citations
We searched OpenAlex and arXiv for “multimodal scientific agent benchmark molecular spectral tool use” and “End-to-End Benchmarking of Native Scientific Perception Within Interactive Agentic Workflows” and an AI compared the idea with the top 10 results. A small search can miss relevant work.
Source Paper
Sage S2.1 Flash Technical ReportKun Cai, Haodong Chen, Guangran Cheng +203 more · 2026
Sage S2.1 Flash is a 320-billion-parameter mixture-of-experts (MoE) model designed to integrate native scientific perception with general-purpose multimodal reasoning and long-horizon agentic execution. It combines dedicated scientific encoders and modality-specific projectors (Sage Sci-Perception) with post-training across large-scale verifiable environments spanning tool use, search, coding, ML engineering, workspace deliverables, and computer use, achieving top-two rankings on many of 33 evaluated benchmarks.
Related Ideas
- Domain-Expert Validation Framework for Automated Scientific IllustrationsAutoFigure-Edit: Generating Editable Scientific Illustration🔴Hard
- Multimodal and Full-Document Soundness Assessment Across Figures and AppendicesPaperCritic: Training AI Reviewers to Assess the Soundness of Papers🔴Hard
- Decoupled 4D Perception-to-Code Pipeline Using Geometric and Body Reconstruction PriorsVibePrevisBench: Evaluating Agentic Coding for 4D Scene Generation in Blender🔴Hard
- Meta-Analytic Agents with Verification-Aware AggregationProbing Scientific General Intelligence of LLMs with Scientist-Aligned Workflows🟡Fairly easy
- Democratizing Scientific Illustration: Fine-Tuning Open-Weight Models for Spatial Reasoning and SVG GenerationAutoFigure-Edit: Generating Editable Scientific Illustration🔴Hard