Skip to content

Source paper

MobilityBench: A Benchmark for Evaluating Route-Planning Agents in Real-World Mobility Scenarios

Zhiheng Song, Jingshuai Zhang, Chuan Qin and 6 more

Zhiheng Song, Jingshuai Zhang, Chuan Qin, Chao Wang, Chao Chen, Longfei Xu, Kaikui Liu, Xiangxiang Chu, Hengshu Zhu

arXiv 2602.22638 · 202691 upvotes on Hugging Face

Research ideas from this paper (3)

Generated by AI. The paper’s authors did not write these.

AI summary

This paper introduces MobilityBench, a scalable and reproducible benchmark designed to evaluate Large Language Model (LLM) based agents on real-world route-planning tasks. The authors constructed the dataset from over 100,000 anonymized real-user queries from Amap, established a deterministic API-replay sandbox to eliminate environmental variance (e.g., changing traffic), and proposed a multi-dimensional evaluation protocol to assess capabilities like instruction understanding, tool use, and efficiency.

How it works (AI summary)

The core methodology involves an 'episode-centric' formulation where agents interact with a deterministic sandbox that replays cached API responses (POIs, routing, traffic) corresponding to historical real-world states. Agents are evaluated using a taxonomy of 11 intent scenarios ranging from simple POI retrieval to complex preference-constrained planning, with metrics focusing on execution correctness (Delivery Rate, Final Pass Rate) and process quality (efficiency, tool schema compliance).