Skip to content

Source paper

DataFlow: An LLM-Driven Framework for Unified Data Preparation and Workflow Automation in the Era of Data-Centric AI

Hao Liang, Xiaochen Ma, Zhou Liu and 32 more

Hao Liang, Xiaochen Ma, Zhou Liu, Zhen Hao Wong, Zhengyang Zhao, Zimo Meng, Runming He, Chengyu Shen, Qifeng Cai, Zhaoyang Han, Meiyi Qiang, Yalin Feng, Tianyi Bai, Zewei Pan, Ziyi Guo, Yizhen Jiang, Jingwen Deng, Qijie You, Peichao Lai, Tianyu Guo, Chi Hsu Tsai, Hengyi Feng, Rui Hu, Wenkai Yu, Junbo Niu, Bohan Zeng, Ruichuan An, Lu Ma, Jihao Huang, Yaowei Zheng, Conghui He, Linpeng Tang, Bin Cui, Weinan E, Wentao Zhang

arXiv 2512.16676 · 2025188 upvotes on Hugging Face

Research ideas from this paper (3)

Generated by AI. The paper’s authors did not write these.

AI summary

This paper presents DataFlow, a unified, open-source framework for LLM-driven data preparation that replaces ad-hoc scripts with composable, Python-based operators and pipelines. It features a novel agentic orchestration layer that can synthesize executable data workflows from natural language commands, demonstrating superior performance in generating high-quality synthetic data for math, code, and text tasks compared to existing baselines.

How it works (AI summary)

The framework utilizes a PyTorch-like programming interface where data processing logic is encapsulated in modular 'Operators' (generate, evaluate, filter, refine) that interact with a global tabular storage abstraction. These operators are composed into 'Pipelines' (DAGs) which can be manually defined or automatically constructed by a multi-agent system (DataFlow-Agent) that performs intent analysis, operator retrieval, code synthesis, and iterative verification.