Continuous Soft-Masking for Cautious Weight Decay
Replace hard binary indicator functions in weight decay update rules with continuous soft-masking using scaled sigmoid or tanh functions to apply proportional regularization.
Source paper
Lizhang Chen, Jonathan Li, Kaizhao Liang, Baiyu Su, Cong Xie, Nuo Wang Pierse, Chen Liang, Ni Lao, Qiang Liu
Generated by AI. The paper’s authors did not write these.
Replace hard binary indicator functions in weight decay update rules with continuous soft-masking using scaled sigmoid or tanh functions to apply proportional regularization.
Integrate cautious weight decay into reinforcement learning and diffusion model training pipelines to evaluate its impact on policy stability and sample generation quality.
Formulate a momentum-smoothed cautious mask that applies weight decay based on historical alignment trends rather than instantaneous optimizer updates.
The paper introduces Cautious Weight Decay (CWD), a one-line, optimizer-agnostic modification that applies decoupled weight decay only when the parameter's sign aligns with the optimizer's update direction. By preserving the original loss objective, CWD improves convergence, lowers final validation loss, and increases zero-shot accuracy across large-scale language and vision models without requiring hyperparameter retuning.
CWD utilizes a sign-selective binary mask (an indicator function) during the parameter update step, applying weight decay selectively. Theoretically, this is supported by Lyapunov analysis and Filippov differential inclusions, proving that standard optimizers with CWD undergo sliding-mode dynamics to reach locally Pareto-optimal points on the stationary manifold of the unregularized objective.