Skip to content
AI-generated research idea🔴Hard🔬Significant ResearchRelated work found

Vector-Based Editable Diagram Generation via GUI Agents

Drive professional vector software via GUI agents to generate editable SVG and PDF diagrams directly from textual style plans.

Time

half a year or more

Compute level

Multiple GPUs

Resources

High-end GPUs for training GUI agents, potential access to proprietary software APIs or extensive SVG datasets.

Prerequisites

Reinforcement learning, computer vision (for GUI agents), or compiler/syntax theory (for SVG/TikZ generation).

Scope

Significant Research

The Gap

The current `Visualizer` agent generates raster images (pixels), specifically at 4K resolution to mitigate blurriness. However, as noted in the Limitations (Section 9), this fails to meet the academic standard for vector graphics (SVG/PDF), making post-generation editing of specific text or nodes impossible without regeneration.

Limitation

The inability to produce editable vector graphics limits the usability of the tool for final manuscript submission, where manual tweaking is often required.

Proposed Solution

Replace the raster generation module with a GUI Agent capable of driving professional vector software (e.g., Adobe Illustrator or Inkscape) or fine-tune a model to output robust TikZ/SVG code directly. This would involve training the agent to map the 'Stylist's' textual plan into sequences of API calls or vector primitives rather than pixel diffusion.

Why This Matters

This would transform the tool from a prototyping utility into a production-grade asset, allowing researchers to tweak labels or move nodes in the final output, a strict requirement for many top-tier conferences.

How to Get Started

  1. Set up a headless environment with vector graphics software such as Inkscape or access to a TikZ code generation pipeline.
  2. Construct a dataset of textual style plans paired with corresponding ground-truth SVG or TikZ code files.
  3. Fine-tune an open-source Vision-Language Model or GUI agent to translate stylist plans into sequences of vector API calls.
  4. Evaluate the generated vector files for editability, rendering correctness, and visual appeal against raster image baselines.

Code linked to this paper on Hugging Face: github.com/dwzhu-pku/PaperBanana

Suggested by AI. Adapt it to your setup.

Has This Been Done?

Related work found

From Pixels to Paths: A Multi-Agent Framework for Editable Scientific Illustration explores using multi-agent setups for editable vector-based illustrations. Other works like OS-Helix and Agent Banana explore general GUI interactions and image editing with agentic tooling, but do not fully solve specific vector software integration or direct TikZ/SVG fine-tuning pipelines.

We searched OpenAlex for “Drive professional vector software via GUI agents to generate editable SVG and PDF diagrams directly from textual style plans.” and “Vector-Based Editable Diagram Generation via GUI Agents” and an AI compared the idea with the top 10 results. A small search can miss relevant work.

Source Paper

PaperBanana: Automating Academic Illustration for AI Scientists

Dawei Zhu, Rui Meng, Yale Song +4 more · 2026

This paper introduces PaperBanana, an agentic framework designed to automate the generation of publication-ready academic illustrations, specifically methodology diagrams and statistical plots. The system orchestrates five specialized agents (Retriever, Planner, Stylist, Visualizer, and Critic) powered by Large Language Models and image generation models to convert textual methodology descriptions into high-fidelity, aesthetically compliant figures.

Related Ideas

Is this a good idea?

Share onXLinkedIn