Generative Vision-Language Model for Table Recognition
Replace specialized table recognition modules with a fine-tuned Vision-Language Model that directly translates table image crops into clean HTML or LaTeX code.
Time
Compute level
Resources
Prerequisites
Scope
The Gap
Section 2.2.4 relies on 'TableMaster' and 'StructEqTable' for table recognition. These are specialized, task-specific models that classify table structures.
Limitation
Task-specific table recognizers often struggle with 'borderless' tables or tables with complex merged cells spanning multiple rows/columns, which are common in financial reports. They require rigid post-processing to convert detection results into HTML/LaTeX.
Proposed Solution
Replace the specialized table recognition module with a small, fine-tuned Vision-Language Model (VLM) (e.g., a fine-tuned version of Donut or LLaVA-Next). Instead of predicting box coordinates, train the model to directly generate the HTML or LaTeX code corresponding to the image crop of the table detected by the layout analysis.
Why This Matters
Tables contain high-density information critical for financial and scientific analysis. Direct generation of structural code (HTML) via VLMs is currently state-of-the-art and would significantly reduce parsing errors compared to coordinate-based reconstruction.
How to Get Started
- Preprocess benchmark table datasets such as PubTabNet and FinTabNet to pair image crops with ground truth markup.
- Fine-tune a lightweight Vision-Language Model like Donut or LLaVA-Next on the table markup generation task.
- Implement syntax validity checks as a post-processing step for generated HTML and LaTeX strings.
- Evaluate structure recognition performance using exact match and TED metrics compared to specialized baselines.
Suggested by AI. Adapt it to your setup.
Has This Been Done?
Works like MinerU2.5 and Qwen3-VL Technical Report cover unified vision-language models and decoupled models for high-resolution document parsing, which inherently includes direct generation of markup formats like HTML or LaTeX from table image crops.
- MinerU2.5: A Decoupled Vision-Language Model for Efficient High-Resolution Document Parsing · 2026 · 189 citations
- Qwen3-VL Technical Report · 2025 · 5762 citations
We searched OpenAlex for “Replace specialized table recognition modules with a fine-tuned Vision-Language Model that directly translates table image crops into clean HTML or LaTeX code.” and “Generative Vision-Language Model for Table Recognition” and an AI compared the idea with the top 10 results. A small search can miss relevant work.
Source Paper
MinerU: An Open-Source Solution for Precise Document Content ExtractionBin Wang, Chao Xu, Xiaomeng Zhao +15 more · 2024
This paper introduces MinerU, an open-source document extraction tool designed to convert diverse and complex PDF documents—such as textbooks, academic papers, and exams—into high-quality machine-readable formats like Markdown and JSON. The authors address the limitations of existing tools by creating a pipeline that integrates specialized deep learning models for layout analysis, formula detection, and content recognition, validated against a newly constructed diverse evaluation dataset.
Related Ideas
- 2D-Attentional Adapters for Complex Table TopologyPaddleOCR-VL: Boosting Multilingual Document Parsing via a 0.9B Ultra-Compact Vision-Language Model🔴Hard
- Direct Visual-to-Graph Generation for Irregular LayoutsRAG-Anything: All-in-One RAG Framework🟠Medium
- Multi-Granularity Visual Encoding for Small-Scale Document ElementsSmolDocling: An ultra-compact vision-language model for end-to-end multi-modal document conversion🟠Medium
- Hierarchical Cell-Level Parallelism for Large TablesDolphin: Document Image Parsing via Heterogeneous Anchor Prompting🟠Medium
- Iterative Semantic Correction for Layout AnalysisPaddleOCR-VL: Boosting Multilingual Document Parsing via a 0.9B Ultra-Compact Vision-Language Model🟠Medium