Skip to content
AI-generated research idea🔴Hard🔬Significant ResearchRelated work found

End-to-End Joint Optimization of MLLM and MMDiT

Fine-tune both the multimodal language model and the video diffusion generator end-to-end using parameter-efficient adapters so the understanding module aligns its hidden states with generative physics.

Time

a couple of months

Compute level

Large cluster

Resources

Large compute cluster

Prerequisites

transformer architectures

Scope

modify existing code

The Gap

In Section 3.3.1 and Section C, the authors explicitly state the MLLM is kept frozen to preserve understanding capabilities. This results in an 'assembled' system where the understanding module cannot adapt its internal representations to better suit the specific requirements of the generative diffusion process.

Limitation

Because the MLLM is frozen, the semantic features it produces are optimized for next-token prediction (text generation) rather than being optimal conditioning signals for video generation, potentially creating a semantic bottleneck.

Proposed Solution

Implement an end-to-end training strategy where the MLLM is unfrozen or fine-tuned using parameter-efficient methods (e.g., LoRA) alongside the MMDiT. This allows the MLLM to learn to output hidden states that are specifically aligned with the manifold of the video generator, bridging the gap between linguistic understanding and generative physics.

Why This Matters

Enabling the MLLM to adapt to generation tasks would create a truly 'native' multimodal model, likely improving instruction adherence and the handling of complex visual prompts.

How to Get Started

  1. Download an open-source MLLM and a compatible video diffusion model codebase.
  2. Implement a training script integrating LoRA into both the vision-language encoder and the diffusion transformer backbone.
  3. Set up a small video-text dataset for a multi-gpu cluster to test stability and monitor loss convergence.
  4. Evaluate the jointly trained model on standard video generation benchmarks to measure visual quality and alignment.

Code linked to this paper on Hugging Face: github.com/KlingTeam/UniVideo

Suggested by AI. Adapt it to your setup.

Has This Been Done?

Related work found

Works like Exploring MLLM-Diffusion Information Transfer with MetaCanvas and JoyAI-Image explore the intersection of multimodal language models and diffusion/generative architectures for understanding and generation. However, none of the listed papers specifically details the end-to-end joint fine-tuning of an MLLM with an MMDiT using parameter-efficient methods like LoRA to bridge linguistic understanding and generative physics.

We searched OpenAlex for “Fine-tune both the multimodal language model and the video diffusion generator end-to-end using parameter-efficient adapters so the understanding module aligns its hidden states with generative physics.” and “End-to-End Joint Optimization of MLLM and MMDiT” and an AI compared the idea with the top 10 results. A small search can miss relevant work.

Source Paper

UniVideo: Unified Understanding, Generation, and Editing for Videos

Cong Wei, Quande Liu, Zixuan Ye +5 more · 2025

UniVideo introduces a unified framework capable of video understanding, generation, and editing within a single system by integrating a Multimodal Large Language Model (MLLM) with a Multimodal Diffusion Transformer (MMDiT). The approach allows the model to process complex multimodal instructions (text, images, video) via the MLLM and project semantic guidance to the diffusion generator, achieving state-of-the-art results across multiple video tasks without task-specific adapters.

Related Ideas

Is this a good idea?

Share onXLinkedIn