Skip to content
AI-generated research idea🔴Hard🔬Significant ResearchRelated work found

Semi-Supervised Inverse Dynamics Labeling

A system that trains an inverse dynamics model to pseudo-label unannotated gameplay videos, expanding training data to cover underrepresented genres.

Time

half a year or more

Compute level

Large cluster

Resources

Large compute cluster

Prerequisites

computer vision, self-supervised learning

Scope

large-scale data collection

The Gap

The dataset construction relies entirely on videos containing 'gamepad overlays' (Section 2.1). The authors note this introduces a bias towards Action-RPGs and console games, while underrepresenting genres like strategy or simulation that rely on keyboard/mouse inputs or lack overlays.

Limitation

Reliance on visual overlays discards the vast majority of available gameplay video on the internet, limiting the diversity of the training data and the model's generalization to non-gamepad genres.

Proposed Solution

Train an Inverse Dynamics Model (IDM) on the existing labeled NitroGen dataset to learn the mapping from (State_t, State_t+1) to Action_t. Use this IDM to pseudo-label a massive corpus of raw gameplay videos (without overlays), filtering by confidence. Re-train NitroGen on this expanded, semi-supervised dataset to cover underrepresented genres.

Why This Matters

Massively scales the training data availability from 'thousands of hours with overlays' to 'millions of hours of any gameplay', potentially solving the genre bias issue.

How to Get Started

  1. Extract state-action transition pairs from the existing overlay-labeled NitroGen dataset to build an initial training set.
  2. Train an Inverse Dynamics Model (IDM) using PyTorch to map state pairs (State_t, State_t+1) to discrete actions.
  3. Collect a large corpus of unannotated gameplay videos across diverse genres like strategy and simulation lacking overlays.
  4. Generate and filter pseudo-labels using the trained IDM confidence scores, then retrain the baseline model on the expanded semi-supervised dataset.

Code linked to this paper on Hugging Face: github.com/MineDojo/NitroGen

Suggested by AI. Adapt it to your setup.

Has This Been Done?

Related work found

Works such as 'On the Sample Efficiency of Inverse Dynamics Models for Semi-Supervised Imitation Learning' explore the general concept of using inverse dynamics models for semi-supervised imitation learning. They do not specifically implement this with the NitroGen dataset and gameplay videos as proposed.

We searched OpenAlex for “A system that trains an inverse dynamics model to pseudo-label unannotated gameplay videos, expanding training data to cover underrepresented genres.” and “Semi-Supervised Inverse Dynamics Labeling” and an AI compared the idea with the top 6 results. A small search can miss relevant work.

Source Paper

NitroGen: An Open Foundation Model for Generalist Gaming Agents

Loïc Magne, Anas Awadalla, Guanzhi Wang +11 more · 2026

This paper introduces NitroGen, a foundation model for generalist gaming agents trained on a massive dataset of 40,000 hours of gameplay videos with actions extracted from on-screen gamepad overlays. By using a vision-action transformer architecture with flow matching, NitroGen demonstrates strong cross-game generalization and acts as a robust pre-trained backbone for downstream tasks.

Related Ideas

Is this a good idea?

Share onXLinkedIn