Non-Linear Latent Tracking for Robust Depth Estimation in Unconstrained Long Videos
Replace the linear overlap solver with a non-linear temporal context memory bank to track global metric anchors across non-overlapping keyframes in unconstrained long videos.
Time
Compute level
Resources
What the paper used
Prerequisites
Scope
The Gap
The model's ability to process long videos relies on 'Global Affine Coherence,' which uses a closed-form least-squares solver to align sliding windows based on overlapping frames. This assumes inter-window discrepancies are predominantly linear scale-shift transformations.
“In unconstrained long videos, extreme dynamics, such as prolonged occlusions, rapid illumination shifts, or erratic camera motions, can introduce local non-linear distortions that temporarily overpower our global affine assumption, leading to scale inconsistencies.”
Limitation
Under extreme unconstrained dynamics, such as prolonged occlusions, rapid illumination shifts, or erratic camera motions transitioning across completely disjointed scenes with zero visual overlap, the linear affine assumption fails, causing severe global scale drift.
Proposed Solution
Replace the linear overlap solver with a non-linear temporal context memory bank or an integrated recurrent state-space model (e.g., Mamba) to track global metric anchors across non-overlapping keyframes. This would learn to dynamically adjust metric scale based on semantic environmental cues rather than relying strictly on direct pixel-wise overlap.
The paper’s authors list this direction as future work, so they may already be pursuing it.
Why This Matters
Resolving long-horizon scale drift is the missing link for infinite-length metric depth anchoring, which is critical for continuous robotic navigation, autonomous driving, and large-scale 3D scene reconstruction.
How to Get Started
How to test it: We hypothesize that incorporating a non-linear temporal context memory bank will maintain scale consistency under erratic motions. We will compare against standard DVD on long unconstrained videos, measuring AbsRel and delta1, and stop if temporal scale drift does not decrease by at least 15%.
- Set up the baseline evaluation pipeline using TartanAir and KITTI datasets to measure depth estimation error over long video sequences.
- Implement a state-space model or temporal memory bank module to replace the closed-form least-squares sliding window alignment.
- Train the modified depth network end-to-end on long video clips using standard scale-invariant depth loss functions.
- Benchmark the non-linear tracking approach against the original global affine coherence method on long-horizon trajectory metrics.
Code linked to this paper on Hugging Face: github.com/EnVision-Research/DVD
Suggested by AI. Adapt it to your setup.
Has This Been Done?
None of the provided papers discuss non-linear temporal context memory banks or state-space models like Mamba for tracking global metric anchors in depth estimation. The listed literature primarily touches upon video scene parsing, talking head generation, and general slam or tracking without addressing this specific proposal.
We searched OpenAlex and arXiv for “Replace the linear overlap solver with a non-linear temporal context memory bank to track global metric anchors across non-overlapping keyframes in unconstrained long videos.” and “Non-Linear Latent Tracking for Robust Depth Estimation in Unconstrained Long Videos” and an AI compared the idea with the top 10 results. A small search can miss relevant work.
Source Paper
DVD: Deterministic Video Depth Estimation with Generative PriorsHongfei Zhang, Harold Haodong Chen, Chenfei Liao +12 more · 2026
The paper introduces DVD, a novel framework that deterministically adapts pre-trained video diffusion models into single-pass depth regressors, overcoming both the stochastic hallucinations of generative models and the semantic ambiguity of discriminative models. By bypassing iterative sampling, DVD achieves state-of-the-art zero-shot video depth estimation with high temporal coherence, utilizing significantly less task-specific training data than leading baselines.
Related Ideas
- Trajectory-Guided Motion Encoders for Video InputOmniLottie: Generating Vector Animations via Parameterized Lottie Tokens🟠Medium
- 4D Spatiotemporal Pretraining for Dynamic ScenesUtonia: Toward One Encoder for All Point Clouds🔴Hard
- Coordinate-Independent Relative Pose and Scale Representationπ^4: A Foundation Model for 4D Scene Geometry and Motion🔴Hard
- Latency-Aware Scalable Video Tracking for Crowded EnvironmentsSAM 3: Segment Anything with Concepts🟠Medium
- Improving Temporal Consistency in World Models via Physics-Informed Dynamic MaskingGrounding World Simulation Models in a Real-World Metropolis🔴Hard