Enhancing Abstract Visual Logic via Synthetically Augmented Multi-Step Math SFT
Enhance abstract multi-step visual reasoning by synthetically generating logic-heavy visual puzzles and fine-tuning models with process reward models.
Time
Compute level
Resources
What the paper used
Prerequisites
Scope
The Gap
During the benchmark evaluations in Section 4.3.1, both the Penguin-VL 8B and 2B models underperformed relative to Qwen3-VL and InternVL-3.5 on abstract multi-step reasoning tasks such as MathVerse and LogicVista.
“On MathVerse and LogicVista, Penguin-8B is outperformed by Qwen3-VL and InternVL-3.5, which suggests that while the model has excellent perceptual grounding for math, it may still be refining the deeper logical chains required for advanced visual puzzles.”
Limitation
The model's reasoning capabilities are mildly constrained by a limited proportion of math-focused Supervised Fine-Tuning (SFT) data, causing it to struggle with forming the deep logical chains required for complex, visually grounded mathematical puzzles.
Proposed Solution
Implement a targeted data synthesis pipeline using stronger teacher models to generate multi-step, logic-heavy visual puzzles and procedural math charts. Fine-tune the model using Process Reward Models (PRMs) or visual Chain-of-Thought (CoT) distillation specifically specialized for multi-hop abstract visual logic.
Why This Matters
Closing this gap will allow the model to excel in highly rigorous academic and engineering tasks, making it a fully well-rounded open-source model capable of advanced structural logic.
How to Get Started
How to test it: We hypothesize that augmenting SFT data with synthetically generated multi-step visual logic problems will improve performance on MathVerse and LogicVista compared to the baseline. We measure accuracy on MathVerse and LogicVista; if accuracy does not improve after fine-tuning, we stop.
- Set up a data synthesis pipeline using a strong teacher model to generate multi-step visual logic puzzles and procedural math charts.
- Construct a fine-tuning dataset incorporating visual Chain-of-Thought reasoning steps.
- Train or adapt a Process Reward Model to evaluate intermediate reasoning steps in visual contexts.
- Fine-tune Penguin-VL on the synthesized dataset and evaluate performance on MathVerse and LogicVista benchmarks.
Code linked to this paper on Hugging Face: github.com/tencent-ailab/Penguin-VL
Suggested by AI. Adapt it to your setup.
Has This Been Done?
SD-MAR: Multi-image Analytical Reasoning via Synthetic Data and Reinforcement Learning and Unlocking Multimodal Mathematical Reasoning via Process Reward Model explore synthetic data and process supervision for multimodal and mathematical reasoning, but they do not specifically target abstract multi-step visual logic puzzles with chart-based SFT and visual CoT distillation.
- Unlocking Multimodal Mathematical Reasoning via Process Reward Model · 2025 · 96 citations
- SD-MAR: Multi-image Analytical Reasoning via Synthetic Data and Reinforcement Learning · 2026 · 0 citations
- DESIGNER: Design-Logic-Guided Multidisciplinary Data Synthesis for LLM Reasoning · 2025 · 6 citations
We searched OpenAlex and arXiv for “Enhance abstract multi-step visual reasoning by synthetically generating logic-heavy visual puzzles and fine-tuning models with process reward models.” and “Enhancing Abstract Visual Logic via Synthetically Augmented Multi-Step Math SFT” and an AI compared the idea with the top 10 results. A small search can miss relevant work.
Source Paper
Penguin-VL: Exploring the Efficiency Limits of VLM with LLM-based Vision EncodersBoqiang Zhang, Lei Ke, Ruihan Yang +5 more · 2026
This paper introduces Penguin-VL, a compact vision-centric multimodal foundation model that challenges the reliance on contrastively pretrained vision encoders like CLIP. By initializing its vision encoder directly from a text-only large language model, Penguin-VL preserves fine-grained visual cues and achieves state-of-the-art performance in dense perception and reasoning among parameter-efficient models.
Related Ideas
- Process-Supervised Reasoning-Targeted Distillation for Routing-Replaced MoEsThe Other Half of the Memory Wall: Serving 35B MoEs from SSD with Trained Routing Prediction🟠Medium
- BitDistill for Reasoning-Intensive DomainsBitNet Distillation🟠Medium
- Process-Supervised Reasoning for STEM DomainsStep-DeepResearch Technical Report🟠Medium
- Recursive Associative-Memory Graph Traversal for Multi-Step Scientific ReasoningWeight-Space Programming Doesn't Need Transformers🟡Fairly easy
- Distilling Open-Ended Generation from RandOpt EnsemblesNeural Thickets: Diverse Task Experts Are Dense Around Pretrained Weights🟡Fairly easy