Skip to content
AI-generated research idea🔴Hard🔬Significant ResearchClosely matching work found

Centralized Training with Decentralized Execution (CTDE) for Multi-Agent Systems

Incorporate a centralized critic using Multi-Agent PPO to stabilize training across independent language model agents operating in a shared environment.

Time

5 days

Compute level

Large cluster

Resources

A cluster of multiple nodes with 8 GPUs each to support distributed agent rollouts and centralized critic training.

What the paper used

Llama-3.2-3B-Instruct across text-to-SQL tasks with multiple agents.

Prerequisites

multi-agent reinforcement learning (MARL)

Scope

new model architecture

The Gap

Section 3.2.2 suggests extending the framework to multi-LLM settings by treating each LLM as an 'independent MDP' and optimizing them separately. This ignores the non-stationarity introduced when multiple agents (e.g., a generator and a reviewer) update their policies simultaneously, which often leads to unstable convergence or cyclic behaviors.

“When multiple distinct LLMs with their own parameters must be jointly optimized, a straightforward approach treats each LLM as an independent MDP and optimizes them separately.”
From the paper, Section 3.2.2, page 7. Matched against the PDF text.

Limitation

Treating interacting agents as independent learners fails to capture the joint dynamics required for effective multi-agent coordination.

Proposed Solution

Integrate Multi-Agent Reinforcement Learning (MARL) algorithms such as MAPPO (Multi-Agent PPO). Modify the Lightning Server to support a centralized Critic that observes the global state (inputs/outputs of all agents in the workflow) during training to stabilize the updates, while keeping execution decentralized on the Client side.

The paper’s authors list this direction as future work, so they may already be pursuing it.

Why This Matters

Essential for robustly training multi-agent systems (e.g., Red Teaming, Debate protocols) where agents must learn complementary strategies.

How to Get Started

How to test it: We hypothesize that using MAPPO with a centralized critic for multi-agent workflows will stabilize updates and yield higher trajectory rewards than independent optimization. We will measure the success rate on multi-turn SQL queries and stop if validation reward improvement plateaus for 3 consecutive epochs.

  1. Modify the Lightning Server architecture to support multi-agent rollouts with shared global state observations.
  2. Implement the Multi-Agent PPO (MAPPO) algorithm with a centralized Critic network.
  3. Set up a multi-LLM benchmark environment featuring generator-reviewer workflows, such as ChatDev or MetaGPT tasks.
  4. Compare the training stability and convergence rate of MAPPO against independent MDP optimization baselines.

Code linked to this paper on Hugging Face: github.com/microsoft/agent-lightning

Suggested by AI. Adapt it to your setup.

Has This Been Done?

Closely matching work found

The concept of Centralized Training with Decentralized Execution (CTDE) and variants like MAPPO are extensively studied in multi-agent reinforcement learning, as overviewed in 'An Introduction to Centralized Training for Decentralized Execution in Cooperative Multi-Agent Reinforcement Learning' and 'Is Centralized Training with Decentralized Execution Framework Centralized Enough for MARL?'.

We searched OpenAlex for “Incorporate a centralized critic using Multi-Agent PPO to stabilize training across independent language model agents operating in a shared environment.” and “Centralized Training with Decentralized Execution (CTDE) for Multi-Agent Systems” and an AI compared the idea with the top 10 results. A small search can miss relevant work.

Source Paper

Agent Lightning: Train ANY AI Agents with Reinforcement Learning

Xufang Luo, Yuge Zhang, Zhiyuan He +5 more · 2025

The paper introduces "Agent Lightning," a framework designed to decouple Reinforcement Learning (RL) training from agent execution, enabling the optimization of LLMs within diverse agent frameworks (e.g., LangChain, AutoGen) with minimal code changes. By formulating agent execution as a Markov Decision Process (MDP) and utilizing a hierarchical RL algorithm called "LightningRL," the system decomposes complex multi-turn trajectories into individual transitions, allowing for efficient, scalable policy updates using standard single-turn RL methods.

Related Ideas

Is this a good idea?

Share onXLinkedIn