Temporal-Consistent Reinforcement Learning for Video Autoregression
A reinforcement learning framework for video generation that uses optical flow consistency and smoothness constraints to prevent flickering and semantic degradation during post-training.
Time
Compute level
Resources
What the paper used
Prerequisites
Scope
The Gap
In Section 5.4.12, the authors attempt to improve generation quality using Reinforcement Learning (Reward Forcing with VideoAlign), but report that it caused severe flickering and degradation of semantic scores. Consequently, they abandoned RL for post-training.
“Future work could explore reinforcement learning techniques that explicitly optimize smoothness-related objectives to further enhance temporal consistency.”
Limitation
The current distillation method relies heavily on adversarial losses which match distributions but do not explicitly penalize high-frequency temporal inconsistencies, and standard RL approaches fail to stabilize the autoregressive chain.
Proposed Solution
Develop a Temporal Consistency Policy Gradient (TCPG) framework. Instead of frame-level rewards, utilize a reward model specifically trained on optical flow consistency or temporal clip-level coherence. Crucially, implement a 'smoothness constraint' in the value function that penalizes rapid latent shifts between autoregressive chunks, ensuring the RL update respects the transition boundaries established by the First-Frame Anchor.
The paper’s authors list this direction as future work, so they may already be pursuing it.
Why This Matters
Solving the flickering issue in RL for video would unlock the ability to fine-tune 14B+ video models for specific human preferences (alignment) without sacrificing temporal stability, a critical step for production-grade world models.
How to Get Started
How to test it: Train a policy gradient RL framework using clip-level optical flow consistency rewards and latent smoothness constraints on the distilled 14B model. Compare against the default Helios-Distilled baseline on HeliosBench using the Motion Smoothness metric (RAFT). Stop if RL optimization destabilizes generation or degrades aesthetic quality scores below 6.0.
- Set up a baseline video autoregression pipeline and replicate the reward-forcing training failure mode to observe visual flickering.
- Implement an optical flow consistency reward model and a clip-level temporal coherence metric.
- Incorporate a smoothness constraint into the value function to penalize rapid latent shifts between chunks.
- Fine-tune a video generation model using the temporal-consistent policy gradient framework and evaluate via subjective and objective video quality metrics.
Code linked to this paper on Hugging Face: github.com/PKU-YuanGroup/Helios
Suggested by AI. Adapt it to your setup.
Has This Been Done?
The listed literature covers video generation alignment, physical/geometric consistency, and reinforcement learning surveys like 'Video Generation Models: A Survey of Post-Training and Alignment' and 'World-R1: Reinforcing 3D Constraints for Text-to-Video Generation', but none propose a Temporal Consistency Policy Gradient framework utilizing a smoothness constraint in the value function for autoregressive chunks.
- Video Generation Models: A Survey of Post-Training and Alignment · 2026 · 3 citations
- A Survey: Spatiotemporal Consistency in Video Generation · 2026 · 13 citations
- World-R1: Reinforcing 3D Constraints for Text-to-Video Generation · 2026 · 18 citations
We searched OpenAlex and arXiv for “A reinforcement learning framework for video generation that uses optical flow consistency and smoothness constraints to prevent flickering and semantic degradation during post-training.” and “Temporal-Consistent Reinforcement Learning for Video Autoregression” and an AI compared the idea with the top 10 results. A small search can miss relevant work.
Source Paper
Helios: Real Real-Time Long Video Generation ModelShenghai Yuan, Yuanyang Yin, Zongjian Li +3 more · 2026
Helios introduces a 14-billion parameter autoregressive video generation model capable of synthesizing minute-scale videos at 19.5 FPS on a single H100 GPU. The authors achieve this by adapting a pre-trained bidirectional transformer with novel token compression techniques, drift-mitigation strategies that avoid expensive training-time rollouts, and a hierarchical adversarial distillation pipeline.
Related Ideas
- Explicit Motion-Guided Control for Video EditingUniVideo: Unified Understanding, Generation, and Editing for Videos🟠Medium
- Direct Causal-Mode Distribution Matching for Autoregressive Video DistillationHiAR: Efficient Autoregressive Long Video Generation via Hierarchical Denoising🔴Hard
- Optimization of Identity-Coherence via Direct Preference LearningDreamID-V:Bridging the Image-to-Video Gap for High-Fidelity Face Swapping via Diffusion Transformer🟠Medium
- Motion-Aware Transition Modeling via Multi-Frame ConditioningStoryMem: Multi-shot Long Video Storytelling with Memory🟡Fairly easy
- Natural Predictive Pre-training on Web-Scale VideoIn Pursuit of Pixel Supervision for Visual Pre-training⛔Very hard