Decoupled 4D Perception-to-Code Pipeline Using Geometric and Body Reconstruction Priors
Equip scene-coding agents with explicit 3D depth, camera-pose, and human-motion estimators so they reconstruct geometry from reference media instead of guessing from raw pixels.
Time
Compute level
Resources
What the paper used
Prerequisites
Scope
The Gap
On media-conditioned tasks (photos, continuations, and videos), the performance gap between the best and worst agents widens from 12.0 points on scripts to 19.9 points on photos, with picture structure and camera accounting for 63% of the gap between the top four and remaining seven agents. In an oracle-caption ablation on five video tasks (Table 5), replacing the video with a text description of ground-truth events, cuts, and camera moves boosts DeepSeek V4.1 Flash by +14.0 Rule-based Scene Check points and +17.2 Screen Check points at a quarter of the API cost.
“Progress on this step thus depends on visual understanding of space and camera, whereas more knowledge of the Blender API saves rounds, not points.”
Limitation
End-to-end multimodal coding agents struggle to infer metric 3D layout, depth ordering, camera intrinsics, and continuous camera/body trajectories directly from raw pixels in photos and multi-shot videos, making visual perception rather than Blender coding the primary bottleneck.
Proposed Solution
Augment the coding agent's sandbox with a lightweight perception toolset mirroring the benchmark's 4D lifting tools—specifically monocular/multi-view depth and camera estimation (e.g., VGGT or MapAnything), shot boundary detection, instance tracking (e.g., SAM 3), and 3D human mesh recovery (e.g., GVHMR)—returning structured JSON scene graphs (per-shot camera extrinsics, focal lengths, 3D bounding boxes, and root trajectories) to the LLM. The coding agent then synthesizes modular Blender Python scripts directly from these calibrated geometric primitives while refining semantics and missing occlusions visually.
Why This Matters
Providing explicit geometric and kinematic priors is expected to close a substantial portion of the 14-point perception gap observed in the oracle-caption ablation while reducing token and turn overhead on visual tasks.
How to Get Started
How to test it: Test whether giving coding agents callable 3D/4D perception tools improves media-task Rule-based Scene Check (P1 Picture structure, A2 Layout, B1 Routes, C1 Camera) over raw-vision agent baselines (DeepSeek V4.1 Flash and Gemini 4 Argon) on the 148 photo, continuation, and video tasks of VibePrevisBench. Stop if noisy depth/pose predictions degrade physical validity or fail to beat the raw-vision baseline by at least 5 points on video and photo tasks.
- Set up a multi-GPU inference environment running shot-boundary detection, SAM 3 segmentation, VGGT or MapAnything camera/depth estimation, and GVHMR body recovery, and verify their outputs on the 5 video tasks from Table 5.
- Design a clean JSON schema exposing per-shot camera focal lengths, 6-DoF camera paths, 3D object boxes, and 3D human root trajectories as sandbox tools callable by Claude Code or Codex CLI.
- Evaluate the perception-augmented agent on the 39 photo tasks (where ScanNet++/ARKitScenes ground-truth boxes exist) and 86 video tasks, measuring P1 Picture structure, A2 Layout, and C1 Camera scores.
- Compare overall Rule-based Scene Check scores and API cost against the unmodified raw-vision agent baselines and the oracle-caption upper bound.
Suggested by AI. Adapt it to your setup.
Has This Been Done?
Papers such as SceneAgent and Crowd4D explore 4D scene reconstruction and simulation environments, but they do not specifically propose augmenting a coding agent's sandbox with a structured monocular and multi-view perception-to-code pipeline using tools like VGGT or GVHMR.
- Crowd4D: Scene-Aware Monocular 4D Crowd Reconstruction · 2026 · 0 citations
- ARROW: Arbitrary Reconstruction and Tracking of 4D Observations in the Wild · 2026 · 2 citations
- SceneAgent: An Agentic Pipeline from 3D and 4D Captures to Simulation Environments · 2026 · 1 citations
We searched OpenAlex and arXiv for “4D scene reconstruction coding agent VGGT GVHMR Blender tool use” and “Decoupled 4D Perception-to-Code Pipeline Using Geometric and Body Reconstruction Priors” and an AI compared the idea with the top 10 results. A small search can miss relevant work.
Source Paper
VibePrevisBench: Evaluating Agentic Coding for 4D Scene Generation in BlenderYuzhe Chen, Muxi Diao, Lele Yang +19 more · 2026
The paper introduces VibePrevisBench, a benchmark of 242 multi-shot, multi-person tasks for evaluating how well coding agents write executable Blender scripts to generate 4D scenes and graybox proxy videos from scripts, photos, frame continuations, and reference videos. Deliveries are scored via a deterministic Rule-based Scene Check across nine spatial, temporal, and physical dimensions and a VLM-based Screen Check assessing on-screen recognizability and story readability. Evaluating 11 frontier coding agents reveals that visual-spatial understanding separates models, all agents fall back on common camera defaults for challenging requests, and agents fail to fix errors when inspecting their own renders.
Related Ideas
- Extending Spatial Curation to Dynamic Scenes via 4D RepresentationHoli-Spatial: Evolving Video Streams into Holistic 3D Spatial Intelligence🔴Hard
- 4D Spatiotemporal Pretraining for Dynamic ScenesUtonia: Toward One Encoder for All Point Clouds🔴Hard
- Extending Decoupled Semantic-Detail Representations to Video ModalitiesCheers: Decoupling Patch Details from Semantic Representations Enables Unified Multimodal Comprehension and Generation⛔Very hard
- Overcoming Perspective Ambiguity in 2D-to-3D VLM ReasoningStepping VLMs onto the Court: Benchmarking Spatial Intelligence in Sports🟠Medium
- Hierarchical Active Perception EncodersOmniGAIA: Towards Native Omni-Modal AI Agents🔴Hard