Skip to content
AI-generated research idea🟠Medium🔬Significant ResearchRelated work found

Action-Conditioned Counterfactual Synthesis for Embodied Physical Planning

Synthesize action-conditioned counterfactual video outcomes in robotic physics environments to train video LLMs on physical causality and trajectory feasibility.

Time

several weeks

Compute level

Multiple GPUs

Resources

Standard GPU setup, open-source physics simulators (ManiSkill, RoboSuite).

What the paper used

Trained on 14 to 30 NVIDIA H100 GPUs (10 to 21 hours run time, 229 to 648 H100-hours per model run) using 4.53M samples with global batch size 150-300.

Prerequisites

robotics, physics simulation, video VLMs

Scope

new data generator pipeline and fine-tuning experiments

The Gap

In Section B (Counterfactual data) and Section 4, counterfactual generation relies on deterministic pixel transformations (e.g., reversal, speed changes, or heuristic synthetic simulator edits in CLEVRER) without modeling interactive, action-conditioned motor dynamics or physical feedback.

Limitation

The offline pixel-level counterfactual generator cannot produce action-conditioned physical counterfactuals needed for embodied robot manipulation and causal action planning.

Proposed Solution

Develop an extensible action-conditioned counterfactual generator using a physics simulator (e.g., RoboSuite or ManiSkill). For a given robot intervention, execute paired action sequences—such as apply target force vs. opposite force, or execute trajectory A vs. perturbed trajectory B—and render paired dynamic videos with auto-generated counterfactual QA labels regarding trajectory success, contact dynamics, and obstacle collision.

Why This Matters

Extends non-annotated counterfactual supervision from simple video frame transformations to complex physical causation and robotics planning tasks.

How to Get Started

How to test it: Evaluate whether action-conditioned counterfactual training improves embodied planning accuracy on standard manipulation benchmarks over baseline V-JEPA-LM. Baseline: Standard V-JEPA-LM 2B. Metric: Pair accuracy on physical action outcome predictions. Stopping result: Stop if synthesized physical counterfactuals fail to outperform standard counterfactual video training by at least 3 points on action QA.

  1. Script a simulator pipeline in ManiSkill/RoboSuite to render paired robot manipulation clips with perturbed control actions.
  2. Format the generated paired trajectories into counterfactual multiple-choice QA items balancing option lengths.
  3. Train V-JEPA-LM on the synthetic physical action dataset mixture.
  4. Evaluate model performance on physical dynamics and robot manipulation benchmarks (e.g., RoboVQA or EgoDex).

Suggested by AI. Adapt it to your setup.

Has This Been Done?

Related work found

Papers like 'DreamTrue' and 'Learning Actionable Manipulation Recovery via Counterfactual Failure Synthesis' investigate action-conditioned counterfactual generation and failure synthesis for robotics. However, they leave open the specific formulation of rendering paired simulator trajectories with counterfactual QA labels for contact dynamics and force perturbations.

We searched OpenAlex and arXiv for “action conditioned counterfactual video generation robotic planning VLM” and “Action-Conditioned Counterfactual Synthesis for Embodied Physical Planning” and an AI compared the idea with the top 10 results. A small search can miss relevant work.

Source Paper

V-JEPA-LM: Video-Language Models That Read Dynamics Beyond Static, Text-Aligned Representations

Keuntae Kim, Dongjun Kim, Yong Suk Choi · 2026

This paper investigates why Video Large Language Models (video LLMs) fail to read visual motion dynamics even when powered by predictive world-model encoders like V-JEPA. The authors demonstrate that standard text captioning and QA supervision allow language models to rely on single-frame static shortcuts rather than dynamic features. To address this, they propose counterfactual supervision using fourteen pixel-level video transformations that isolate change components, enabling state-of-the-art motion and temporal reasoning across eleven video benchmarks with significantly lower compute and data requirements.

Related Ideas

Is this a good idea?

Share onXLinkedIn