Semi-Supervised Inverse Dynamics Labeling
A system that trains an inverse dynamics model to pseudo-label unannotated gameplay videos, expanding training data to cover underrepresented genres.
Time
Compute level
Resources
Prerequisites
Scope
The Gap
The dataset construction relies entirely on videos containing 'gamepad overlays' (Section 2.1). The authors note this introduces a bias towards Action-RPGs and console games, while underrepresenting genres like strategy or simulation that rely on keyboard/mouse inputs or lack overlays.
Limitation
Reliance on visual overlays discards the vast majority of available gameplay video on the internet, limiting the diversity of the training data and the model's generalization to non-gamepad genres.
Proposed Solution
Train an Inverse Dynamics Model (IDM) on the existing labeled NitroGen dataset to learn the mapping from (State_t, State_t+1) to Action_t. Use this IDM to pseudo-label a massive corpus of raw gameplay videos (without overlays), filtering by confidence. Re-train NitroGen on this expanded, semi-supervised dataset to cover underrepresented genres.
Why This Matters
Massively scales the training data availability from 'thousands of hours with overlays' to 'millions of hours of any gameplay', potentially solving the genre bias issue.
How to Get Started
- Extract state-action transition pairs from the existing overlay-labeled NitroGen dataset to build an initial training set.
- Train an Inverse Dynamics Model (IDM) using PyTorch to map state pairs (State_t, State_t+1) to discrete actions.
- Collect a large corpus of unannotated gameplay videos across diverse genres like strategy and simulation lacking overlays.
- Generate and filter pseudo-labels using the trained IDM confidence scores, then retrain the baseline model on the expanded semi-supervised dataset.
Code linked to this paper on Hugging Face: github.com/MineDojo/NitroGen
Suggested by AI. Adapt it to your setup.
Has This Been Done?
Works such as 'On the Sample Efficiency of Inverse Dynamics Models for Semi-Supervised Imitation Learning' explore the general concept of using inverse dynamics models for semi-supervised imitation learning. They do not specifically implement this with the NitroGen dataset and gameplay videos as proposed.
- On the Sample Efficiency of Inverse Dynamics Models for Semi-Supervised Imitation Learning · 2026 · 2 citations
We searched OpenAlex for “A system that trains an inverse dynamics model to pseudo-label unannotated gameplay videos, expanding training data to cover underrepresented genres.” and “Semi-Supervised Inverse Dynamics Labeling” and an AI compared the idea with the top 6 results. A small search can miss relevant work.
Source Paper
NitroGen: An Open Foundation Model for Generalist Gaming AgentsLoïc Magne, Anas Awadalla, Guanzhi Wang +11 more · 2026
This paper introduces NitroGen, a foundation model for generalist gaming agents trained on a massive dataset of 40,000 hours of gameplay videos with actions extracted from on-screen gamepad overlays. By using a vision-action transformer architecture with flow matching, NitroGen demonstrates strong cross-game generalization and acts as a robust pre-trained backbone for downstream tasks.
Related Ideas
- Unsupervised Adaptation to Human Multiplayer GameplaySolaris: Building a Multiplayer Video World Model in Minecraft🟡Fairly easy
- Multi-Agent World Modeling: Extending Camera-Conditioned DiTs to Dynamic EntitiesWorldCam: Interactive Autoregressive 3D Gaming Worlds with Camera Pose as a Unifying Geometric Representation⛔Very hard
- Hierarchical Dynamic Memory Aggregation for Long-Form Video ReasoningV-JEPA-LM: Video-Language Models That Read Dynamics Beyond Static, Text-Aligned Representations🟠Medium
- Self-Supervised Geometric Distillation for Long-Horizon Data WallsLoGeR: Long-Context Geometric Reconstruction with Hybrid Memory🔴Hard
- Natural Predictive Pre-training on Web-Scale VideoIn Pursuit of Pixel Supervision for Visual Pre-training⛔Very hard