Learnable Anisotropic Spectral Partitioning
Replace fixed radial Fourier masks with learnable 2D spectral filters conditioned on global latents to optimize semantic and detail separation dynamically.
Source paper
Weichen Fan, Haiwen Diao, Quan Wang, Dahua Lin, Ziwei Liu
Generated by AI. The paper’s authors did not write these.
Replace fixed radial Fourier masks with learnable 2D spectral filters conditioned on global latents to optimize semantic and detail separation dynamically.
Extend 2D image autoencoding to spatiotemporal video latents using a 3D FFT band projector to decompose static content and motion residuals.
Implement an attention-based gating mechanism that dynamically weights frequency bands contributing to semantic alignment loss based on input content.
This paper introduces Unified Autoencoding (UAE) and the 'Prism Hypothesis', which posits that semantic information (categories, attributes) primarily resides in low-frequency components of data, while pixel-level details (texture, edges) reside in high-frequency components. By decomposing latent representations into frequency bands using FFT, UAE aligns the low-frequency band with a frozen semantic teacher (like DINOv2) while learning high-frequency residuals for reconstruction, effectively creating a single token space that is both semantically rich and capable of high-fidelity generation.
The model initializes a Unified Encoder from a pretrained DINOv2. It uses an FFT-based 'Band Projector' to split the latent space into concentric frequency bands. A 'semantic-wise loss' forces the lowest frequency band to match the teacher's semantic features, while a 'pixel-wise reconstruction loss' trains the higher frequency bands to capture residuals needed for decoding. A 'Frequency Band Modulator' with noise injection is used during training to ensure robustness and proper factorization.