Hierarchical Language-Goal Conditioning
A system that conditions reactive action generation on high-level goals from a frozen language model to enable long-horizon planning.
Source paper
Loïc Magne, Anas Awadalla, Guanzhi Wang, Yinzhen Xu, Joshua Belofsky, Fengyuan Hu, Joohwan Kim, Ludwig Schmidt, Georgia Gkioxari, Jan Kautz, Yisong Yue, Yejin Choi, Yuke Zhu, Linxi "Jim" Fan
Generated by AI. The paper’s authors did not write these.
A system that conditions reactive action generation on high-level goals from a frozen language model to enable long-horizon planning.
A method that applies consistency distillation to compress multi-step diffusion generation into a single forward pass for real-time interactive applications.
A system that trains an inverse dynamics model to pseudo-label unannotated gameplay videos, expanding training data to cover underrepresented genres.
This paper introduces NitroGen, a foundation model for generalist gaming agents trained on a massive dataset of 40,000 hours of gameplay videos with actions extracted from on-screen gamepad overlays. By using a vision-action transformer architecture with flow matching, NitroGen demonstrates strong cross-game generalization and acts as a robust pre-trained backbone for downstream tasks.
The authors curate an internet-scale dataset by extracting actions from gamepad overlays in videos using template matching and segmentation. The model uses a SigLIP 2 vision encoder and a Diffusion Transformer (DiT) with flow matching to predict chunks of 16 actions conditioned on a single history frame.