Native Multimodal Operator Support via Late Fusion Storage
Extend data storage abstractions to reference binary image blobs directly and introduce multimodal operators that interface with vision-language models for late-fusion pipelines.
Source paper
Hao Liang, Xiaochen Ma, Zhou Liu, Zhen Hao Wong, Zhengyang Zhao, Zimo Meng, Runming He, Chengyu Shen, Qifeng Cai, Zhaoyang Han, Meiyi Qiang, Yalin Feng, Tianyi Bai, Zewei Pan, Ziyi Guo, Yizhen Jiang, Jingwen Deng, Qijie You, Peichao Lai, Tianyu Guo, Chi Hsu Tsai, Hengyi Feng, Rui Hu, Wenkai Yu, Junbo Niu, Bohan Zeng, Ruichuan An, Lu Ma, Jihao Huang, Yaowei Zheng, Conghui He, Linpeng Tang, Bin Cui, Weinan E, Wentao Zhang
Generated by AI. The paper’s authors did not write these.
Extend data storage abstractions to reference binary image blobs directly and introduce multimodal operators that interface with vision-language models for late-fusion pipelines.
Incorporate a proxy evaluation stage into the agent verification loop to assess data training utility using validation loss from a lightweight proxy model.
Implement a self-distillation meta-operator that trains lightweight student models on expensive large-scale model outputs to cost-effectively filter massive pre-training datasets.
This paper presents DataFlow, a unified, open-source framework for LLM-driven data preparation that replaces ad-hoc scripts with composable, Python-based operators and pipelines. It features a novel agentic orchestration layer that can synthesize executable data workflows from natural language commands, demonstrating superior performance in generating high-quality synthetic data for math, code, and text tasks compared to existing baselines.
The framework utilizes a PyTorch-like programming interface where data processing logic is encapsulated in modular 'Operators' (generate, evaluate, filter, refine) that interact with a global tabular storage abstraction. These operators are composed into 'Pipelines' (DAGs) which can be manually defined or automatically constructed by a multi-agent system (DataFlow-Agent) that performs intent analysis, operator retrieval, code synthesis, and iterative verification.