Skip to content

Source paper

V-JEPA-LM: Video-Language Models That Read Dynamics Beyond Static, Text-Aligned Representations

Keuntae Kim, Dongjun Kim, Yong Suk Choi
arXiv 2611.17831 · 202625 upvotes on Hugging Face

Research ideas from this paper (3)

Generated by AI. The paper’s authors did not write these.

AI summary

This paper investigates why Video Large Language Models (video LLMs) fail to read visual motion dynamics even when powered by predictive world-model encoders like V-JEPA. The authors demonstrate that standard text captioning and QA supervision allow language models to rely on single-frame static shortcuts rather than dynamic features. To address this, they propose counterfactual supervision using fourteen pixel-level video transformations that isolate change components, enabling state-of-the-art motion and temporal reasoning across eleven video benchmarks with significantly lower compute and data requirements.

How it works (AI summary)

The model attaches a frozen V-JEPA 2.1 predictive video encoder (ViT-L/ViT-g) to an open language model (Qwen3 or Llama-3) using a spectral bridge that compresses clips into 1,024 tokens. Training is performed in a single stage on 4.53 million public samples, augmented by counterfactual video pairs generated programmatically via six pixel-transformation families (sign flip, order, position, magnitude, count, and stop/cut) without language model annotations.