Skip to content
AI-generated research idea🟡Fairly easy💡Moderate InnovationNo close match found

Task-Specific Reward Experts for MR-GRPO

Expand the reinforcement learning reward system into a mixture of specialized weak supervisor experts dynamically activated by prompt keywords.

Time

4-7 days

Compute level

Large cluster

Resources

Cluster-scale resources (e.g., 32-64 H100 GPUs) matching the paper's RL training setup to evaluate multiple reward model inferences during on-policy rollout.

What the paper used

RL trained for 1,500 steps with group size G=8 on 64x H200 GPUs

Prerequisites

reinforcement learning

Scope

modify existing code

The Gap

The Reinforcement Learning stage uses MR-GRPO with a mixture of three broad rewards: VLM Preference (general quality), OCR (text accuracy), and CLIP (semantic alignment). The weights are adjusted based on prompt category (text-rendering vs. general).

Limitation

The current reward set is too coarse for fine-grained editing and spatial reasoning. A general VLM preference reward may overlook subtle spatial errors (e.g., 'cup left of spoon' vs 'spoon left of cup') or counting errors, as these global metrics often prioritize texture and style over precise structural logic.

Proposed Solution

Expand the reward model into a Mixture of Experts (MoE) system. Integrate specialized, lightweight 'weak supervisor' models as reward functions, such as an Object Detector (for counting/position), a Depth Estimator (for spatial relations), and a segmentation model. Dynamically activate these rewards based on keywords in the user prompt.

Why This Matters

This directly addresses the 'hallucination' problem in complex instruction following. By providing precise, disentangled feedback signals, the model can learn specific capabilities (like counting or relative positioning) much more effectively than via a noisy global reward.

How to Get Started

How to test it: Hypothesis: specialized reward experts improve fine-grained spatial and counting accuracy. Baseline: standard global VLM preference reward and OCR reward. Metric: UniREditBench and RISE spatial/logical scores. Stop if training becomes unstable or reward hacking occurs.

  1. Collect lightweight pretrained expert models including an object detector, a depth estimator, and a segmentation model.
  2. Design a keyword-matching router to assign weights to each expert reward based on the user prompt.
  3. Integrate the multi-expert reward framework into the existing MR-GRPO training pipeline.
  4. Benchmark the reward-optimized model against the baseline on spatial reasoning and counting tasks.

Code linked to this paper on Hugging Face: github.com/DeepGenTeam/DeepGen

Suggested by AI. Adapt it to your setup.

Has This Been Done?

No close match in 6 results

The listed literature contains works applying GRPO to normative reasoning, molecular optimization, and clinical reliability, along with general surveys on reward models. However, none of these papers cover expanding reward models into an MoE system with specialized perception supervisors activated via prompt keywords.

We searched OpenAlex for “Expand the reinforcement learning reward system into a mixture of specialized weak supervisor experts dynamically activated by prompt keywords.” and “Task-Specific Reward Experts for MR-GRPO” and an AI compared the idea with the top 6 results. A small search can miss relevant work.

Source Paper

DeepGen 1.0: A Lightweight Unified Multimodal Model for Advancing Image Generation and Editing

Dianyi Wang, Ruihang Li, Feng Han +17 more · 2026

The paper introduces DeepGen 1.0, a lightweight (5B parameter) unified multimodal model capable of image generation, editing, and reasoning. By employing a novel Stacked Channel Bridging (SCB) architecture to fuse Vision-Language Model (VLM) features with a Diffusion Transformer (DiT) and utilizing a three-stage training pipeline culminating in Reinforcement Learning (MR-GRPO), the model achieves performance competitive with significantly larger closed-source and open-source models.

Related Ideas

Is this a good idea?

Share onXLinkedIn