Vision Encoder Adaptation via LoRA
Apply Parameter-Efficient Fine-Tuning adapters to the vision encoder layers during instruction tuning to learn robust degradation-invariant visual features without destroying pre-trained capabilities.
Source paper
Jiaqi Tang, Jianmin Chen, Wei Wei, Xiaogang Xu, Runtao Liu, Xiangyu Wu, Qipeng Xie, Jiafei Wu, Lei Zhang, Qifeng Chen
Generated by AI. The paper’s authors did not write these.
Apply Parameter-Efficient Fine-Tuning adapters to the vision encoder layers during instruction tuning to learn robust degradation-invariant visual features without destroying pre-trained capabilities.
Replace explicit ground-truth parameter penalties with a self-supervised consistency reward derived from a frozen off-the-shelf image restoration teacher model.
Compile explicit reasoning token sequences into latent model weights using knowledge distillation to eliminate intermediate token generation overhead at inference time.
This paper presents Robust-R1, a multimodal large language model framework designed to handle visual degradations (blur, noise, etc.) through explicit reasoning rather than just implicit adaptation. The authors construct a dataset with synthetic degradations and structured reasoning annotations, then use Supervised Fine-Tuning and Reinforcement Learning (with specific rewards for parameter accuracy and reasoning length) to teach the model to diagnose degradations, analyze their impact, and reconstruct pristine semantic meaning.
The model uses Qwen2.5-VL as a base. It employs a three-stage approach: (1) Constructing a dataset (based on A-OKVQA) with synthetic degradations and GPT-4o generated reasoning chains; (2) Supervised Fine-Tuning (SFT) to teach the reasoning format; (3) Reinforcement Learning with Group Relative Preference Optimization (GRPO) using two reward functions: one to align predicted degradation parameters with ground truth, and another to dynamically scale the reasoning chain length based on degradation intensity.