Density-Aware Token Retention for Mesh Completeness
Augment token pruning with a spatial density score to ensure rare 3D regions retain enough points for complete mesh reconstruction.
Time
Compute level
Resources
Prerequisites
Scope
The Gap
In Table 2 and the Discussion (Section 6), the authors note that InfiniteVGGT underperforms baselines (CUT3R) on the 'Completion' metric, despite better accuracy.
Limitation
The current pruning metric relies solely on 'diversity' (cosine similarity of keys) to save memory. This discriminates against homogeneous textures (e.g., large walls, roads) which look similar in feature space but are spatially required to form a complete dense surface. Pruning them creates holes in the reconstruction.
Proposed Solution
Augment the pruning score with a 'Spatial Density' term. Project tokens into a coarse 3D voxel grid during the forward pass. Increase the retention score for tokens that belong to low-density voxels, ensuring that even feature-redundant tokens are kept if they are the only ones covering a specific spatial region.
Why This Matters
Addressing this would fix the model's primary weakness (reconstruction completeness), making it superior to baselines in all metrics, not just accuracy.
How to Get Started
- Examine the token pruning module in the InfiniteVGGT codebase and identify where retention scores are computed.
- Implement a voxelization function that projects token coordinates into a coarse 3D grid during the forward pass.
- Add a density penalty term to the retention score calculation that boosts tokens located in sparsely populated voxels.
- Evaluate the updated model on the completion metric benchmarks and compare output mesh quality against CUT3R.
Code linked to this paper on Hugging Face: github.com/AutoLab-SAI-SJTU/InfiniteVGGT
Suggested by AI. Adapt it to your setup.
Has This Been Done?
The provided results include surveys on multimodal data and token optimization techniques like SieveServe, but none of them propose augmenting pruning scores with a 3D voxel-based spatial density term for mesh completeness.
We searched OpenAlex for “Augment token pruning with a spatial density score to ensure rare 3D regions retain enough points for complete mesh reconstruction.” and “Density-Aware Token Retention for Mesh Completeness” and an AI compared the idea with the top 10 results. A small search can miss relevant work.
Source Paper
InfiniteVGGT: Visual Geometry Grounded Transformer for Endless StreamsShuai Yuan, Yantai Yang, Xiaotian Yang +4 more · 2026
This paper introduces InfiniteVGGT, a transformer-based model for continuous 3D geometry reconstruction from infinite video streams without memory overflow. The authors propose a rolling memory mechanism that dynamically prunes the KV cache based on key-diversity (cosine similarity) rather than attention weights, enabling compatibility with FlashAttention and long-term stability.
Related Ideas
- Hybrid Representation for Open-Surface GenerationUltraShape 1.0: High-Fidelity 3D Shape Generation via Scalable Geometric Refinement🟠Medium
- Semantic-Aware Variable-Rate TokenizationNextFlow: Unified Sequential Modeling Activates Multimodal Understanding and Generation🔴Hard
- Adaptive Early-Stopping and Feed-Forward Alternatives for 3DGS OptimizationHoli-Spatial: Evolving Video Streams into Holistic 3D Spatial Intelligence🟠Medium
- Topological Loss Integration for Geometric ParsingDeepSeek-OCR: Contexts Optical Compression🟠Medium
- Bypassing VAE Compression Bottlenecks for Ultra-High-Resolution Video DepthDVD: Deterministic Video Depth Estimation with Generative Priors🟠Medium