Skip to content
AI-generated research idea🟡Fairly easy💡Moderate InnovationNo close match found

Dual-Stream Vision Fusion for Concurrent Motion Dynamics and Text/OCR Recognition

Combine frozen V-JEPA motion representations with text-aligned SigLIP features via dual-stream spectral projection to preserve dynamic reasoning while restoring static visual knowledge.

Time

several weeks

Compute level

Multiple GPUs

Resources

1-2 standard high-memory GPUs (e.g., A100 or H100), public video and OCR datasets.

What the paper used

Trained on 14 to 30 NVIDIA H100 GPUs (10 to 21 hours run time, 229 to 648 H100-hours per model run) using 4.53M samples with global batch size 150-300.

Prerequisites

multimodal architectures, vision transformers, feature fusion

Scope

modify existing code to add second vision encoder stream and combined projector

The Gap

In Section I (Scope and limitations), the paper highlights that V-JEPA is trained solely via latent feature prediction without text alignment, causing V-JEPA-LM to lag behind text-aligned encoders like SigLIP on static image QA, document reading, and knowledge-heavy benchmarks (e.g., NExT-QA).

“V-JEPA is trained only to predict masked regions of video in latent space, never to align with or read text, so text recognition lies outside what the encoder was built for”
From the paper, Section I, page 29. Matched against the PDF text.

Limitation

V-JEPA-LM suffers from poor single-frame image QA and optical character recognition (OCR) capabilities due to V-JEPA's lack of text-aligned pretraining data.

Proposed Solution

Build a dual-stream spectral bridge that fuses visual tokens from frozen V-JEPA (predictive encoder for dynamics) and frozen SigLIP 2 (text-aligned encoder for fine-grained semantic/text OCR). Channel V-JEPA's change subspace tokens alongside SigLIP's frame-level semantic tokens into a joint projector MLP, fine-tuning the VLM with the combined counterfactual and standard multimodal QA mixture.

Why This Matters

Eliminates the trade-off between dynamic motion understanding and static visual knowledge, allowing a single lightweight video LLM to excel at both dynamic motion QA and static/text-heavy image QA.

How to Get Started

How to test it: Test if dual-stream fusion maintains V-JEPA's high TOMATO dynamics score while closing the gap on NExT-QA and PerceptionTest. Baseline: V-JEPA-LM single encoder. Metric: Average accuracy across TOMATO and NExT-QA. Stopping result: Stop if dual-stream fusion decreases TOMATO dynamics score by >3 percentage points.

  1. Set up the V-JEPA-LM repository and reproduce the 2B Qwen3 baseline model.
  2. Add SigLIP 2 ViT-L/14 as a parallel vision backbone alongside frozen V-JEPA 2.1.
  3. Concatenate/project the output representations through a dual-stream spectral bridge.
  4. Train on the 4.53M dataset mixture and evaluate on TOMATO (dynamics) and NExT-QA (knowledge/text).

Suggested by AI. Adapt it to your setup.

Has This Been Done?

No close match in 10 results

Papers such as 'Can World Models Benefit VLMs for World Dynamics?' and 'PROSPECT' study the fusion of predictive latent representations with semantic vision-language models. None of the listed papers combine a frozen V-JEPA predictive encoder for motion dynamics with a frozen SigLIP 2 encoder via a joint change-subspace projector MLP.

We searched OpenAlex and arXiv for “dual stream vision encoder V-JEPA SigLIP video VLM” and “Dual-Stream Vision Fusion for Concurrent Motion Dynamics and Text/OCR Recognition” and an AI compared the idea with the top 10 results. A small search can miss relevant work.

Source Paper

V-JEPA-LM: Video-Language Models That Read Dynamics Beyond Static, Text-Aligned Representations

Keuntae Kim, Dongjun Kim, Yong Suk Choi · 2026

This paper investigates why Video Large Language Models (video LLMs) fail to read visual motion dynamics even when powered by predictive world-model encoders like V-JEPA. The authors demonstrate that standard text captioning and QA supervision allow language models to rely on single-frame static shortcuts rather than dynamic features. To address this, they propose counterfactual supervision using fourteen pixel-level video transformations that isolate change components, enabling state-of-the-art motion and temporal reasoning across eleven video benchmarks with significantly lower compute and data requirements.

Related Ideas

Is this a good idea?

Share onXLinkedIn