Parameter-Efficient Adaptation of Pre-trained Vision Encoders
Reduce high-resolution vision encoder training costs by applying parameter-efficient fine-tuning to pre-trained vision models instead of training from scratch.
Time
Compute level
Resources
What the paper used
Prerequisites
Scope
The Gap
Section 3.2 ('Vision Encoder') states that the 1.2B parameter encoder was 'trained entirely from scratch' to handle 11MP inputs, explicitly departing from fine-tuning pre-trained models.
“Our approach, dots.ocr, is designed to perform end-to-end document parsing by unifying three core tasks within a single Vision-Language Model: layout detection, text recognition, and relational understanding”
Limitation
Training a 1.2B parameter vision encoder from scratch is computationally prohibitive for most research labs and ignores the rich semantic priors available in modern pre-trained backbones (like SigLIP or CLIP). It creates a high barrier to entry and huge energy cost.
Proposed Solution
Investigate replacing the from-scratch training with Parameter-Efficient Fine-Tuning (PEFT) techniques (e.g., LoRA or Adapters) applied to high-quality open-source pre-trained encoders (e.g., SigLIP-SO400M). Implement patch interpolation or sliding window attention to handle the 11MP resolution requirement without retraining weights from random initialization.
Why This Matters
Drastically reduces training cost and carbon footprint while potentially achieving comparable performance, making the architecture accessible to the broader research community.
How to Get Started
How to test it: We hypothesize that employing PEFT on a pre-trained vision encoder will maintain parsing accuracy while reducing training compute. We will compare against the fully-trained baseline using OverallEdit distance on OmniDocBench, stopping if performance drops by more than 2%.
- Select a pre-trained open-source vision encoder such as SigLIP-SO400M as the base backbone.
- Incorporate patch interpolation or sliding-window attention mechanisms to support high-resolution eleven-megapixel inputs.
- Apply parameter-efficient fine-tuning techniques like LoRA to the backbone and benchmark performance against scratch-trained counterparts.
Code linked to this paper on Hugging Face: github.com/rednote-hilab/dots.ocr
Suggested by AI. Adapt it to your setup.
Has This Been Done?
Papers like Parameter-Efficient Model Adaptation for Vision Transformers and Parameter-Efficient Fine-Tuning for Pre-Trained Vision Models: A Survey and Benchmark explore applying PEFT to vision models broadly. However, they do not specifically address the combination of high 11MP resolution handling via patch interpolation and SigLIP encoders as proposed here.
- Parameter-Efficient Model Adaptation for Vision Transformers · 2023 · 195 citations
- Parameter-Efficient Fine-Tuning for Pre-Trained Vision Models: A Survey and Benchmark · 2026 · 201 citations
We searched OpenAlex for “Reduce high-resolution vision encoder training costs by applying parameter-efficient fine-tuning to pre-trained vision models instead of training from scratch.” and “Parameter-Efficient Adaptation of Pre-trained Vision Encoders” and an AI compared the idea with the top 6 results. A small search can miss relevant work.
Source Paper
dots.ocr: Multilingual Document Layout Parsing in a Single Vision-Language ModelYumeng Li, Guang Yang, Hao Liu +2 more · 2025
This paper introduces dots.ocr, a unified Vision-Language Model designed for end-to-end document layout parsing, text recognition, and relational understanding across 126 languages. The authors achieve this by training a 1.2B parameter vision encoder from scratch on a massive, synthetically generated multilingual dataset produced by a novel three-stage data engine, demonstrating state-of-the-art performance on OmniDocBench and the newly introduced XDocParse benchmark.
Related Ideas
- Vision Encoder Adaptation via LoRARobust-R1: Degradation-Aware Reasoning for Robust Visual Understanding🟡Fairly easy
- Hierarchical Active Perception EncodersOmniGAIA: Towards Native Omni-Modal AI Agents🔴Hard
- Parameter-Efficient Action Adaptation and Multi-Layer Feature Readout for Frontier VLMsScaleVLA: Scaling Vision-Language-Action Models to 552 Billion Parameters🔴Hard
- Adaptive Strided Sampling and Pruning for Real-Time Edge InferenceLoGeR: Long-Context Geometric Reconstruction with Hybrid Memory🟡Fairly easy
- Native Video-Based Instructive Editing PipelineOmniVCus: Feedforward Subject-driven Video Customization with Multimodal Control Conditions🔴Hard