Skip to content

Source paper

DeepPlanning: Benchmarking Long-Horizon Agentic Planning with Verifiable Constraints

Yinger Zhang, Shutong Jiang, Renhao Li and 6 more

Yinger Zhang, Shutong Jiang, Renhao Li, Jianhong Tu, Yang Su, Lianghao Deng, Xudong Guo, Chenxu Lv, Junyang Lin

arXiv 2601.18137 · 202628 upvotes on Hugging Face

Research ideas from this paper (3)

Generated by AI. The paper’s authors did not write these.

AI summary

The paper introduces DEEPPLANNING, a challenging benchmark for evaluating Long-Horizon Agentic Planning (LHAP) in realistic scenarios like multi-day travel and complex shopping. It utilizes a "Layered Task Generation" pipeline to create solvable tasks with strict global constraints (e.g., budgets, schedules) and evaluates agents using code-based verification metrics rather than LLM judges, revealing that even state-of-the-art models struggle with combinatorial optimization and implicit constraint discovery.

How it works (AI summary)

The authors employ a synthetic data generation pipeline where a solver first creates a valid plan and then reverse-engineers a user query by injecting personalized and environmental constraints. The evaluation occurs in an offline sandbox with simulated SQL-like databases and APIs, requiring agents to proactively retrieve information and generate structured JSON plans that satisfy verifiable logic and arithmetic constraints.