Skip to content

Source paper

Mobile-O: Unified Multimodal Understanding and Generation on Mobile Device

Abdelrahman Shaker, Ahmed Heakl, Jaseel Muhammad and 8 more

Abdelrahman Shaker, Ahmed Heakl, Jaseel Muhammad, Ritesh Thawkar, Omkar Thawakar, Senmao Li, Hisham Cholakkal, Ian Reid, Eric P. Xing, Salman Khan, Fahad Shahbaz Khan

arXiv 2602.20161 · 202619 upvotes on Hugging Face

Research ideas from this paper (3)

Generated by AI. The paper’s authors did not write these.

AI summary

This paper introduces Mobile-O, a lightweight unified multimodal model designed for mobile devices that performs both visual understanding (image captioning, VQA) and text-to-image generation. The authors propose a novel 'Mobile Conditioning Projector' (MCP) to efficiently bridge a compact Vision-Language Model (FastVLM) with a diffusion generator (DiT), alongside a three-stage training pipeline utilizing a new 'quadruplet' data format to jointly optimize both tasks with minimal data.

How it works (AI summary)

The model architecture couples a FastVLM-0.5B (using FastViT and Qwen2-0.5B) with a SANA-0.6B-based diffusion transformer. The core technical contribution is the Mobile Conditioning Projector (MCP), which uses layerwise feature fusion and depthwise-separable convolutions to project VLM tokens into the diffusion conditioning space. Training involves a specific 'quadruplet' scheme (generation prompt, image, question, answer) to align understanding and generation tasks simultaneously.