Instruction-Controllable VLMs for Strict Formatting via Preference Optimization
Fine-tune open-source vision-language models using Direct Preference Optimization on a format-compliance dataset to strictly eliminate verbose explanations and enforce exact prompt output formats.
Time
Compute level
Resources
What the paper used
Prerequisites
Scope
The Gap
Proprietary models like Gemini-3-Pro and Claude-Sonnet-4.5 struggled to output answers in the exact format required by the benchmark prompts, often producing verbose explanations that required a secondary LLM for parsing.
“Although the models are required to produce final answers in a specified format, they frequently generate uncontrolled intermediate reasoning or extended explanations, violating the output constraints.”
Limitation
High-capacity VLMs suffer from an over-helpfulness alignment tax, rendering them brittle for automated metric evaluation frameworks that require strict syntax.
Proposed Solution
Apply Direct Preference Optimization (DPO) targeting format adherence. Create a lightweight dataset of format-compliant versus overly-verbose responses and fine-tune an open-source model to strictly penalize extraneous tokens while preserving spatial reasoning accuracy.
Why This Matters
Improves the deployment efficiency and API cost of VLMs in programmatic pipelines, removing the latency and unreliability of secondary parser LLMs.
How to Get Started
How to test it: Hypothesis: DPO targeting format adherence reduces extraneous tokens while maintaining spatial accuracy. Baseline: SFT-adapted Qwen3-VL-8B. Metric: Exact format compliance rate and spatial reasoning accuracy. Stop if format compliance does not reach 95%.
- Collect or synthesize a small training set of format-compliant versus verbose responses using an open-weights VLM baseline like LLaVA or Qwen-VL.
- Set up the Direct Preference Optimization training pipeline using the TRL library on a standard GPU.
- Establish an automated evaluation script to measure format adherence and spatial reasoning accuracy.
- Fine-tune the model and evaluate against proprietary baselines on targeted strict-formatting benchmarks.
Code linked to this paper on Hugging Face: github.com/Visionary-Laboratory/CourtSI
Suggested by AI. Adapt it to your setup.
Has This Been Done?
Studies like Uni-DPO and VIVA explore preference optimization and reward mechanisms for large language models and multimodal tasks. However, the specific application of direct preference optimization for strict format adherence and penalty of extraneous tokens in spatial VLMs is not directly evaluated by these works.
- Uni-DPO: A Unified Paradigm for Dynamic Preference Optimization of LLMs · 2025 · 13 citations
- VIVA: VLM-Guided Instruction-Based Video Editing with Reward Optimization · 2025 · 14 citations
- Systematic Reward Gap Optimization for Mitigating VLM Hallucinations · 2025 · 16 citations
We searched OpenAlex and arXiv for “Fine-tune open-source vision-language models using Direct Preference Optimization on a format-compliance dataset to strictly eliminate verbose explanations and enforce exact prompt output formats.” and “Instruction-Controllable VLMs for Strict Formatting via Preference Optimization” and an AI compared the idea with the top 10 results. A small search can miss relevant work.
Source Paper
Stepping VLMs onto the Court: Benchmarking Spatial Intelligence in SportsYuchen Yang, Yuqing Shao, Duxiu Huang +11 more · 2026
This paper introduces CourtSI, the first large-scale spatial intelligence dataset and benchmark tailored for sports scenarios to evaluate Vision-Language Models (VLMs). By leveraging well-defined court geometries, the authors reconstruct 3D sports scenes to generate over 1 million metric-aware QA pairs, revealing significant gaps in current VLMs' spatial reasoning capabilities.
Related Ideas
- Grammar-Constrained Command GenerationAutoDev: Automated AI-Driven Development🟡Fairly easy
- Robust Control via Constrained DecodingRecursive Language Models🟡Fairly easy
- Reasoning-Integrated Promptable Segmentation via Native instruction-TuningSAM 3: Segment Anything with Concepts🔴Hard
- Grammar-Constrained Decoding for Structured DocTagsSmolDocling: An ultra-compact vision-language model for end-to-end multi-modal document conversion🟡Fairly easy
- Interpretable Reasoning via Decodable Think TokensDeepGen 1.0: A Lightweight Unified Multimodal Model for Advancing Image Generation and Editing🟠Medium