End-to-End Empirical Validation of Proxy Scores Against Downstream Video Generation Quality
Benchmark how strongly 4D scene-accuracy scores predict final synthesized video quality across diverse agents and generative renderers to validate proxy-level evaluation.
Time
Compute level
Resources
What the paper used
Prerequisites
Scope
The Gap
VibePrevisBench evaluates graybox proxy videos and the underlying 4D Blender scenes under the motivating premise that real-world video generation models inherit the proxy's layout, motion, and camera errors. However, in Section C.5 and Figure 2, the authors only inspect two qualitative video pairs rendered with Seedance 2.5 and explicitly note that systematic correlation across all agents remains unmeasured.
“Correlating scores with real-world video quality across all agents is left to future work.”
Limitation
It is unknown to what extent gains in Rule-based Scene Check and VLM-based Screen Check actually translate into higher-fidelity, prompt-faithful final videos across different proxy-conditioned video generators, or which 4D scene errors downstream generators tolerate versus amplify.
Proposed Solution
Conduct a large-scale downstream rendering study by feeding the rendered graybox proxy videos from all 11 agents (or a stratified multi-tier subset) through multiple proxy-conditioned real-world video generation models (such as Seedance 2.5, Kling 4.0, Cosmos-transfer1, and VACE). Evaluate the resulting real-world videos with human preference rankings, video-level action/camera consistency benchmarks, and reference-video similarity metrics, then compute per-dimension sensitivity regressions showing how each of the nine 4D scene dimensions impacts final video quality.
The paper’s authors list this direction as future work, so they may already be pursuing it.
Why This Matters
Quantifying which 4D proxy errors propagate into synthesized real-world videos is expected to establish predictive validity for proxy-level benchmarks and guide which error types coding agents must prioritize fixing.
How to Get Started
How to test it: Test the hypothesis that VibePrevisBench overall scores positively correlate (Spearman rho >= 0.75) with human-judged and VLM-judged fidelity of downstream synthesized real-world videos across representative strong, mid-tier, and weak coding agents on at least two proxy-conditioned video generators. Stop if video generation models override or ignore proxy geometry so heavily that proxy score differences fail to predict downstream quality.
- Select a stratified subset of 50 tasks across all four input kinds and render proxy videos from 5 representative agents spanning the score spectrum (e.g., GPT-6 Astra, Gemini 4 Argon, DeepSeek V4.1 Flash, GLM 5.3 Flash, Muse Spark 1.3).
- Generate real-world videos from each proxy video using at least two proxy-conditioned generators (e.g., open-weights VACE or Cosmos-transfer1 and commercial Seedance 2.5 or Kling 4.0) with matched text prompts.
- Collect pairwise human preference judgments and automated camera/action alignment metrics comparing the generated real-world videos against the task briefs and reference clips.
- Run regression analyses mapping the nine Rule-based Scene Check dimensions and four Screen Check dimensions to downstream real-world video error rates.
Suggested by AI. Adapt it to your setup.
Has This Been Done?
Works like WorldModelBench and PhySimCode evaluate video generation models and physics code generation, but they do not conduct the specific downstream regression study evaluating graybox proxy videos against multiple real-world video generation models across nine 4D scene dimensions.
- PhySimCode: A Benchmark and Evaluation Method for Physics Video to Code Generation · 2026 · 1 citations
- WorldModelBench: Judging Video Generation Models As World Models · 2025 · 175 citations
We searched OpenAlex and arXiv for “proxy video conditioned video generation benchmark evaluation correlation” and “End-to-End Empirical Validation of Proxy Scores Against Downstream Video Generation Quality” and an AI compared the idea with the top 10 results. A small search can miss relevant work.
Source Paper
VibePrevisBench: Evaluating Agentic Coding for 4D Scene Generation in BlenderYuzhe Chen, Muxi Diao, Lele Yang +19 more · 2026
The paper introduces VibePrevisBench, a benchmark of 242 multi-shot, multi-person tasks for evaluating how well coding agents write executable Blender scripts to generate 4D scenes and graybox proxy videos from scripts, photos, frame continuations, and reference videos. Deliveries are scored via a deterministic Rule-based Scene Check across nine spatial, temporal, and physical dimensions and a VLM-based Screen Check assessing on-screen recognizability and story readability. Evaluating 11 frontier coding agents reveals that visual-spatial understanding separates models, all agents fall back on common camera defaults for challenging requests, and agents fail to fix errors when inspecting their own renders.
Related Ideas
- Real-Time Causal Streaming ArchitectureStable Video Infinity: Infinite-Length Video Generation with Error Recycling🟠Medium
- Direct Causal-Mode Distribution Matching for Autoregressive Video DistillationHiAR: Efficient Autoregressive Long Video Generation via Hierarchical Denoising🔴Hard
- Reference-Conditioned VAE for High-Fidelity StreamingPersonaLive! Expressive Portrait Image Animation for Live Streaming🟠Medium
- Real-Time Interactive World Generation via Diffusion DistillationWorldCam: Interactive Autoregressive 3D Gaming Worlds with Camera Pose as a Unifying Geometric Representation🟠Medium
- Query-Adaptive Video Context Compression and Frame-Level Visual Dependence for Long-Video Speculative DraftingMMFlash: Faster Speculative Decoding for Vision-Language Models🟠Medium