Distilling Agentic Reasoning into Fast Scene Diffusion
Distill the expensive iterative Designer-Critic agent loop into a fast, non-iterative Diffusion Transformer conditioned on layout graphs.
Time
Compute level
Resources
Prerequisites
Scope
The Gap
Table 16 indicates a high generation cost ($13.98/scene) and high latency (3:26 minutes/scene). The bulk of this cost (75.9%) comes from the iterative Designer agent calls.
Limitation
The high cost and latency prevent 'online' scene generation, where a robot might need to imagine a scene in real-time for planning (world modeling). The current iterative VLM approach is too slow for run-time applications.
Proposed Solution
Use the dataset of valid scenes generated by SceneSmith as supervision to train a faster, non-iterative generative model (e.g., a Diffusion Transformer conditioned on layout graphs). This 'distilled' model would bypass the slow Designer-Critic loop for rapid inference, using SceneSmith as the high-quality teacher.
Why This Matters
Reducing generation time from minutes to milliseconds would allow this technology to be used as an internal world model for model-predictive control or rapid online replanning.
How to Get Started
- Collect a training dataset of valid, high-quality scenes generated by the SceneSmith pipeline.
- Train a baseline Diffusion Transformer conditioned on layout graphs using standard image-to-layout datasets like ADE20K or structured scene graphs.
- Distill the iterative SceneSmith outputs into the student diffusion model using imitation learning.
- Benchmark inference cost and latency against the original teacher pipeline.
Code linked to this paper on Hugging Face: github.com/nepfaff/scenesmith
Suggested by AI. Adapt it to your setup.
Has This Been Done?
The listed papers cover agentic workflows, layout generation, and diffusion models separately, but none discuss distilling an agentic scene generation loop into a fast non-iterative Diffusion Transformer using SceneSmith or similar teacher datasets.
We searched OpenAlex for “Distill the expensive iterative Designer-Critic agent loop into a fast, non-iterative Diffusion Transformer conditioned on layout graphs.” and “Distilling Agentic Reasoning into Fast Scene Diffusion” and an AI compared the idea with the top 10 results. A small search can miss relevant work.
Source Paper
SceneSmith: Agentic Generation of Simulation-Ready Indoor ScenesNicholas Pfaff, Thomas Cohn, Sergey Zakharov +2 more · 2026
SceneSmith is a hierarchical, agentic framework that utilizes Large Language Models (LLMs) and Vision-Language Models (VLMs) to generate diverse, simulation-ready indoor environments from text prompts. By decomposing scene generation into sequential stages (layout, furniture, wall/ceiling objects, manipulands) managed by a Designer-Critic-Orchestrator agent trio, the system produces physically valid scenes with dense clutter and articulated objects suitable for robotic training.
Related Ideas
- Accelerating 3D-Guided RL Fine-Tuning with Trajectory DistillationGeometry-Guided Reinforcement Learning for Multi-view Consistent 3D Scene Editing🔴Hard
- Consistency Distillation for Real-Time InferenceFlashPortrait: 6x Faster Infinite Portrait Animation with Adaptive Latent Prediction🔴Hard
- Distilling Generative Video Depth Priors into Lightweight Linear-Complexity ModelsDVD: Deterministic Video Depth Estimation with Generative Priors🔴Hard
- Synthetic Trajectory Distillation for Specialized Open-Weight ModelsDeepCode: Open Agentic Coding🟠Medium
- Real-Time Interactive World Generation via Diffusion DistillationWorldCam: Interactive Autoregressive 3D Gaming Worlds with Camera Pose as a Unifying Geometric Representation🟠Medium