Skip to content
AI-generated research idea🟠Medium💡Moderate InnovationNo close match found

High-Fidelity Initialization via Large Reconstruction Models

Replace monocular depth map reconstruction with a large reconstruction model outputting 3D Gaussian Splats, providing accurate geometric initialization for physics simulations.

Time

1-2 weeks

Compute level

Multiple GPUs

Resources

1 multi-GPU node (e.g. 4x A100/H100) to train and integrate heavy Large Reconstruction Models into the pipeline.

What the paper used

128 A100 GPU-days for training; single H200 GPU for inference and 3D scene reconstruction taking around 13.5s.

Prerequisites

3D reconstruction, Gaussian Splatting, transformer architectures

Scope

modify existing code

The Gap

Section 3.1 describes using monocular depth estimation (MoGE-2) and inpainting to create a point cloud representation. The paper's Limitations section explicitly admits this leads to inaccurate geometry and depth errors, which propagate into the physics simulation causing unnatural collisions or object distortions.

“Reconstructing 3D scenes can be inaccurate due to errors in depth estimation, leading to suboptimal simulation and video results.”
From the paper, Section 5, page 15. Matched against the PDF text.

Limitation

The physics simulation is limited by the quality of the initial 3D reconstruction, which currently relies on prone-to-error depth maps and heuristics.

Proposed Solution

Replace the depth-map-based reconstruction with a Large Reconstruction Model (LRM) that outputs 3D Gaussian Splats. Integrate these splats directly into the physics solver (leveraging techniques like PhysGaussian), providing a more accurate, fully textured 3D initialization that reduces geometric artifacts and improves simulation fidelity.

The paper’s authors list this direction as future work, so they may already be pursuing it.

Why This Matters

Improving the geometric prior is the most direct way to increase the physical realism of the simulation, ensuring collisions and interactions behave predictably.

How to Get Started

How to test it: We hypothesize that replacing heuristic depth reconstruction with Large Reconstruction Models producing 3D Gaussian Splats will improve geometric fidelity and simulation stability compared to the baseline. If visual artifacts are not reduced in user studies, we will halt.

  1. Acquire public weights and inference code for a Large Reconstruction Model capable of outputting 3D Gaussian Splats.
  2. Integrate the Gaussian Splat representation into the PhysGaussian or equivalent physics solver pipeline.
  3. Benchmark the geometric accuracy and collision realism of simulated objects against the original depth-map method.
  4. Evaluate end-to-end video generation quality and simulation fidelity on standard object-interaction datasets.

Code linked to this paper on Hugging Face: github.com/liuwei283/RealWonder

Suggested by AI. Adapt it to your setup.

Has This Been Done?

No close match in 10 results

The listed surveys and works such as Avat3r: Large Animatable Gaussian Reconstruction Model for High-Fidelity 3D Head Avatars discuss Gaussian splatting and reconstruction models broadly. However, none of the listed papers combine Large Reconstruction Models with physics solvers like PhysGaussian for 3D initialization in the manner proposed.

We searched OpenAlex and arXiv for “Replace monocular depth map reconstruction with a large reconstruction model outputting 3D Gaussian Splats, providing accurate geometric initialization for physics simulations.” and “High-Fidelity Initialization via Large Reconstruction Models” and an AI compared the idea with the top 10 results. A small search can miss relevant work.

Source Paper

RealWonder: Real-Time Physical Action-Conditioned Video Generation

Wei Liu, Ziyu Chen, Zizhang Li +3 more · 2026

RealWonder proposes a real-time system for generating videos conditioned on 3D physical actions (forces, gravity, collisions) from a single image. The method bridges the gap between abstract physical actions and visual generation by using a physics simulator to produce intermediate optical flow and coarse RGB previews, which then condition a distilled, fast-inference video diffusion model.

Related Ideas

Is this a good idea?

Share onXLinkedIn