Physics-Constrained Reward Modeling for RLHF
A reward modeling method that penalizes physically impossible motions using trajectory smoothness heuristics during reinforcement learning from human feedback.
Time
Compute level
Resources
Prerequisites
Scope
The Gap
In Section 4.2 'RLHF', the authors utilize a VLM-based reward model to evaluate 'motion dynamics.' However, general-purpose VLMs primarily assess visual alignment and aesthetics; they lack an intuitive understanding of Newtonian physics, often rewarding plausible-looking but physically impossible motions (e.g., objects sliding without friction or inconsistent gravity).
Limitation
The current reward model optimizes for aesthetic appeal but fails to penalize violations of physical laws, leading to 'dream-like' physics in generated motion.
Proposed Solution
Augment the online RL stage with a physics-consistency reward term. Integrate an optical flow estimator (like RAFT) or a monocular depth estimator to track object trajectories. Calculate a 'physics violation score' based on trajectory smoothness and conservation of momentum heuristics, and use this as a negative reward signal in the PPO/DPO loop to enforce realistic motion dynamics.
Why This Matters
Improving physical realism is the primary hurdle for using generative video in professional VFX and simulation workflows.
How to Get Started
- Integrate RAFT or a monocular depth estimator to track object trajectories during the video generation RL stage.
- Develop a physics violation scoring function based on momentum conservation and trajectory smoothness.
- Incorporate the physics penalty into the PPO reward function and measure improvements in motion realism.
Code linked to this paper on Hugging Face: github.com/Tencent-Hunyuan/HunyuanVideo-1.5
Suggested by AI. Adapt it to your setup.
Has This Been Done?
The listed works discuss reinforcement learning, curriculum learning, and RLHF alignment, but none of them cover integrating optical flow or monocular depth estimators to compute a physics violation score as a negative reward signal in the PPO/DPO loop.
We searched OpenAlex for “A reward modeling method that penalizes physically impossible motions using trajectory smoothness heuristics during reinforcement learning from human feedback.” and “Physics-Constrained Reward Modeling for RLHF” and an AI compared the idea with the top 10 results. A small search can miss relevant work.
Source Paper
HunyuanVideo 1.5 Technical ReportBing Wu, Chang Zou, Changlin Li +78 more · 2025
This paper introduces HunyuanVideo 1.5, an open-source 8.3B parameter video generation model that achieves state-of-the-art performance comparable to proprietary models like Sora and Kling. The authors propose a unified Diffusion Transformer (DiT) architecture optimized with a novel Selective and Sliding Tile Attention (SSTA) mechanism and a high-compression 3D VAE, trained via a progressive multi-stage pipeline involving text-to-image, text-to-video, and image-to-video tasks.
Related Ideas
- Optical Flow-Guided Physics RegularizationYume-1.5: A Text-Controlled Interactive World Generation Model🟡Fairly easy
- Trajectory-Guided Motion Encoders for Video InputOmniLottie: Generating Vector Animations via Parameterized Lottie Tokens🟠Medium
- Interactive Policy Refinement via World Model RLGigaBrain-0: A World Model-Powered Vision-Language-Action Model⛔Very hard
- Closed-Loop Synchronization for Long-Horizon ConsistencyRealWonder: Real-Time Physical Action-Conditioned Video Generation🟠Medium
- Physics-Informed Diffusion Guidance for Data GenerationGigaBrain-0: A World Model-Powered Vision-Language-Action Model🔴Hard