Distilling Generative Video Depth Priors into Lightweight Linear-Complexity Models
Transfer generative geometric priors from heavy video diffusion transformers into an efficient linear-complexity sequence model using distillation matching gradient fields.
Time
Compute level
Resources
What the paper used
Prerequisites
Scope
The Gap
DVD's deterministic adaptation currently utilizes massive pre-trained Video Diffusion Transformer backbones (e.g., Wan2.1-1.3B) and relies on computationally heavy attention mechanisms over spatio-temporal sequences.
Limitation
Despite bypassing iterative stochastic sampling, the sheer parameter count and quadratic complexity of the DiT backbone prevent the model from achieving true real-time inference (>= 10Hz), limiting its use in latency-critical on-device applications.
Proposed Solution
Implement architectural distillation to transfer the generative geometric priors from the heavy Wan2.1 DiT into an efficient linear-complexity sequence model (such as RWKV or a CNN-Transformer hybrid). Use teacher-student knowledge distillation specifically matching the Latent Manifold Rectification (LMR) gradient fields to preserve edge sharpness.
The paper’s authors list this direction as future work, so they may already be pursuing it.
Why This Matters
This will unlock the high-fidelity structural priors of foundation models for edge deployment, enabling advanced spatial awareness on mobile robots, AR/VR headsets, and other computationally constrained devices.
How to Get Started
How to test it: We hypothesize that distilling the Wan2.1 DiT into an efficient linear-complexity sequence model will achieve real-time inference (>= 10Hz) without severe accuracy drop. We will measure inference FPS and AbsRel on ScanNet, stopping if FPS < 10 or AbsRel degrades by more than 10%.
- Download pre-trained Wan2.1-1.3B video diffusion transformer weights to serve as the fixed teacher model.
- Initialize an efficient student model using an open-source RWKV or linear-complexity architecture.
- Construct a distillation loss function targeting the Latent Manifold Rectification gradient fields to preserve structural edge sharpness.
- Evaluate the distilled model on standard video depth datasets and measure speedup, memory footprint, and accuracy.
Code linked to this paper on Hugging Face: github.com/EnVision-Research/DVD
Suggested by AI. Adapt it to your setup.
Has This Been Done?
Papers such as "Wan: Open and Advanced Large-Scale Video Generative Models" and "SQuad: Sub-Quadratic Attention Distillation for Efficient Video Generation" explore video generation models and attention distillation techniques. However, they do not specifically implement the proposed distillation of Wan2.1 into linear-complexity models matching Latent Manifold Rectification gradient fields for depth.
- Wan: Open and Advanced Large-Scale Video Generative Models · 2025 · 3714 citations
- SQuad: Sub-Quadratic Attention Distillation for Efficient Video Generation · 2026 · 0 citations
We searched OpenAlex and arXiv for “Transfer generative geometric priors from heavy video diffusion transformers into an efficient linear-complexity sequence model using distillation matching gradient fields.” and “Distilling Generative Video Depth Priors into Lightweight Linear-Complexity Models” and an AI compared the idea with the top 10 results. A small search can miss relevant work.
Source Paper
DVD: Deterministic Video Depth Estimation with Generative PriorsHongfei Zhang, Harold Haodong Chen, Chenfei Liao +12 more · 2026
The paper introduces DVD, a novel framework that deterministically adapts pre-trained video diffusion models into single-pass depth regressors, overcoming both the stochastic hallucinations of generative models and the semantic ambiguity of discriminative models. By bypassing iterative sampling, DVD achieves state-of-the-art zero-shot video depth estimation with high temporal coherence, utilizing significantly less task-specific training data than leading baselines.
Related Ideas
- Hybrid Diffusion-Regression for Large-Parallax SynthesisSharp Monocular View Synthesis in Less Than a Second🔴Hard
- Modality-Adaptive Flow MatchingDreamID-Omni: Unified Framework for Controllable Human-Centric Audio-Video Generation🔴Hard
- Self-Supervised Geometric Distillation for Long-Horizon Data WallsLoGeR: Long-Context Geometric Reconstruction with Hybrid Memory🔴Hard
- Accelerating Video VAE Decoding via Distilled QuantizationTurboDiffusion: Accelerating Video Diffusion Models by 100-200 Times🟡Fairly easy
- Uncertainty-Aware Robust DistillationLiveTalk: Real-Time Multimodal Interactive Video Diffusion via Improved On-Policy Distillation🟡Fairly easy