Skip to content

Source paper

The Prism Hypothesis: Harmonizing Semantic and Pixel Representations via Unified Autoencoding

Weichen Fan, Haiwen Diao, Quan Wang and 2 more

Weichen Fan, Haiwen Diao, Quan Wang, Dahua Lin, Ziwei Liu

arXiv 2512.19693 · 202554 upvotes on Hugging Face

Research ideas from this paper (3)

Generated by AI. The paper’s authors did not write these.

AI summary

This paper introduces Unified Autoencoding (UAE) and the 'Prism Hypothesis', which posits that semantic information (categories, attributes) primarily resides in low-frequency components of data, while pixel-level details (texture, edges) reside in high-frequency components. By decomposing latent representations into frequency bands using FFT, UAE aligns the low-frequency band with a frozen semantic teacher (like DINOv2) while learning high-frequency residuals for reconstruction, effectively creating a single token space that is both semantically rich and capable of high-fidelity generation.

How it works (AI summary)

The model initializes a Unified Encoder from a pretrained DINOv2. It uses an FFT-based 'Band Projector' to split the latent space into concentric frequency bands. A 'semantic-wise loss' forces the lowest frequency band to match the teacher's semantic features, while a 'pixel-wise reconstruction loss' trains the higher frequency bands to capture residuals needed for decoding. A 'Frequency Band Modulator' with noise injection is used during training to ensure robustness and proper factorization.