Hybrid Diffusion-Regression for Large-Parallax Synthesis
Implement a cascaded architecture where Gaussian splatting outputs serve as coarse geometric conditioning for a latent diffusion model to improve hallucination in large-baseline views.
Source paper
Lars Mescheder, Wei Dong, Shiwei Li, Xuyang Bai, Marcel Santos, Peiyun Hu, Bruno Lecouat, Mingmin Zhen, Amaël Delaunoy, Tian Fang, Yanghai Tsin, Stephan R. Richter, Vladlen Koltun
Generated by AI. The paper’s authors did not write these.
Implement a cascaded architecture where Gaussian splatting outputs serve as coarse geometric conditioning for a latent diffusion model to improve hallucination in large-baseline views.
Integrate a Spherical Harmonics representation and a per-Gaussian appearance MLP into the Gaussian decoder to render view-dependent specularity.
Incorporate a segmentation-based semantic head and physics-informed loss functions to correct monodepth estimation errors on reflective and transparent surfaces.
SHARP is a feed-forward neural network that synthesizes a high-resolution 3D Gaussian representation from a single monocular image in less than one second. The method achieves state-of-the-art visual fidelity for nearby view synthesis by regressing Gaussian parameters directly, enabling real-time photorealistic rendering at over 100 frames per second on standard hardware.
The model utilizes a Depth Pro backbone for feature extraction and a two-layer depth decoder to initialize base Gaussians. A specialized Gaussian decoder then predicts refinements for position, scale, rotation, color, and opacity, while a learned depth adjustment module resolves monocular scale ambiguities through a task-specific information bottleneck during training.