Skip to content
AI-generated research idea🟡Fairly easy💡Moderate InnovationRelated work found

Parameter-Efficient Adaptation of Pre-trained Vision Encoders

Reduce high-resolution vision encoder training costs by applying parameter-efficient fine-tuning to pre-trained vision models instead of training from scratch.

Time

3 days

Compute level

Multiple GPUs

Resources

1x A100 GPU (80GB) for fine-tuning with PEFT (LoRA) on pre-trained backbones.

What the paper used

1.2B parameter Vision Encoder and 1.7B parameter Language Model Decoder trained on approximately 300k diverse samples using AdamW optimizer with peak learning rate 5e-5.

Prerequisites

transformer architectures

Scope

modify existing code

The Gap

Section 3.2 ('Vision Encoder') states that the 1.2B parameter encoder was 'trained entirely from scratch' to handle 11MP inputs, explicitly departing from fine-tuning pre-trained models.

“Our approach, dots.ocr, is designed to perform end-to-end document parsing by unifying three core tasks within a single Vision-Language Model: layout detection, text recognition, and relational understanding”
From the paper, Section 3, page 3. Matched against the PDF text.

Limitation

Training a 1.2B parameter vision encoder from scratch is computationally prohibitive for most research labs and ignores the rich semantic priors available in modern pre-trained backbones (like SigLIP or CLIP). It creates a high barrier to entry and huge energy cost.

Proposed Solution

Investigate replacing the from-scratch training with Parameter-Efficient Fine-Tuning (PEFT) techniques (e.g., LoRA or Adapters) applied to high-quality open-source pre-trained encoders (e.g., SigLIP-SO400M). Implement patch interpolation or sliding window attention to handle the 11MP resolution requirement without retraining weights from random initialization.

Why This Matters

Drastically reduces training cost and carbon footprint while potentially achieving comparable performance, making the architecture accessible to the broader research community.

How to Get Started

How to test it: We hypothesize that employing PEFT on a pre-trained vision encoder will maintain parsing accuracy while reducing training compute. We will compare against the fully-trained baseline using OverallEdit distance on OmniDocBench, stopping if performance drops by more than 2%.

  1. Select a pre-trained open-source vision encoder such as SigLIP-SO400M as the base backbone.
  2. Incorporate patch interpolation or sliding-window attention mechanisms to support high-resolution eleven-megapixel inputs.
  3. Apply parameter-efficient fine-tuning techniques like LoRA to the backbone and benchmark performance against scratch-trained counterparts.

Code linked to this paper on Hugging Face: github.com/rednote-hilab/dots.ocr

Suggested by AI. Adapt it to your setup.

Has This Been Done?

Related work found

Papers like Parameter-Efficient Model Adaptation for Vision Transformers and Parameter-Efficient Fine-Tuning for Pre-Trained Vision Models: A Survey and Benchmark explore applying PEFT to vision models broadly. However, they do not specifically address the combination of high 11MP resolution handling via patch interpolation and SigLIP encoders as proposed here.

We searched OpenAlex for “Reduce high-resolution vision encoder training costs by applying parameter-efficient fine-tuning to pre-trained vision models instead of training from scratch.” and “Parameter-Efficient Adaptation of Pre-trained Vision Encoders” and an AI compared the idea with the top 6 results. A small search can miss relevant work.

Source Paper

dots.ocr: Multilingual Document Layout Parsing in a Single Vision-Language Model

Yumeng Li, Guang Yang, Hao Liu +2 more · 2025

This paper introduces dots.ocr, a unified Vision-Language Model designed for end-to-end document layout parsing, text recognition, and relational understanding across 126 languages. The authors achieve this by training a 1.2B parameter vision encoder from scratch on a massive, synthetically generated multilingual dataset produced by a novel three-stage data engine, demonstrating state-of-the-art performance on OmniDocBench and the newly introduced XDocParse benchmark.

Related Ideas

Is this a good idea?

Share onXLinkedIn