Query-Driven Dynamic Layout Parsing
Enable flexible information extraction by conditioning document parsing models on natural language queries rather than fixed reading progressions.
Time
Compute level
Resources
What the paper used
Prerequisites
Scope
The Gap
Section 3.1 ('Unified Task Formulation') defines the output as a fixed sequence S aligned with 'human reading progression' (Eq. 1).
“Crucially, the sequence S is generated in an order that aligns with human reading progression. This task formulation compels the model to not only parse content but also to comprehend the document's logical flow”
Limitation
Enforcing a single, deterministic 'reading order' is restrictive for non-linear documents (e.g., newspapers, websites, forms) where multiple valid traversal paths exist. The current model rigidly serializes content, which may not match user intent when specific information is needed.
Proposed Solution
Reformulate the training objective to support Query-Driven Parsing. Condition the generation on a natural language prompt (e.g., 'Extract the table in the bottom left', 'Read the sidebar headers'). This requires modifying the data engine to generate prompt-response pairs rather than just full-document sequences.
Why This Matters
Transforms the model from a static serializer into a dynamic agent capable of targeted information extraction, significantly increasing utility for complex downstream tasks.
How to Get Started
How to test it: We hypothesize that conditioning the sequence generation on natural language prompts will enable flexible query-driven extraction. We will compare against the standard reading order model using Reading Order Edit and extraction precision metrics, stopping if general parsing performance degrades.
- Modify the data generation engine to pair document images with specific user queries and localized target answers.
- Reformulate the model's training objective to take natural language prompt tokens as conditional inputs during sequence generation.
- Fine-tune the model on the query-driven parsing dataset and evaluate extraction accuracy against targeted prompts.
Code linked to this paper on Hugging Face: github.com/rednote-hilab/dots.ocr
Suggested by AI. Adapt it to your setup.
Has This Been Done?
While Intelligent Document Parsing and other works touch on document parsing or graph attention for specific parsing tasks, none of the listed papers discuss query-driven dynamic layout parsing with natural language prompts as proposed.
We searched OpenAlex for “Enable flexible information extraction by conditioning document parsing models on natural language queries rather than fixed reading progressions.” and “Query-Driven Dynamic Layout Parsing” and an AI compared the idea with the top 6 results. A small search can miss relevant work.
Source Paper
dots.ocr: Multilingual Document Layout Parsing in a Single Vision-Language ModelYumeng Li, Guang Yang, Hao Liu +2 more · 2025
This paper introduces dots.ocr, a unified Vision-Language Model designed for end-to-end document layout parsing, text recognition, and relational understanding across 126 languages. The authors achieve this by training a 1.2B parameter vision encoder from scratch on a massive, synthetically generated multilingual dataset produced by a novel three-stage data engine, demonstrating state-of-the-art performance on OmniDocBench and the newly introduced XDocParse benchmark.
Related Ideas
- Reasoning-Integrated Promptable Segmentation via Native instruction-TuningSAM 3: Segment Anything with Concepts🔴Hard
- Efficient Input-Dependent Queries for Attention ResidualsAttention Residuals🟠Medium
- Task-Conditioned Query-Based Decoding for Fine-Grained SemanticsUtonia: Toward One Encoder for All Point Clouds🟠Medium
- Iterative Semantic Correction for Layout AnalysisPaddleOCR-VL: Boosting Multilingual Document Parsing via a 0.9B Ultra-Compact Vision-Language Model🟠Medium
- Grammar-Constrained Command GenerationAutoDev: Automated AI-Driven Development🟡Fairly easy