Skip to content
AI-generated research idea🟠Medium🔬Significant ResearchRelated work found

Trajectory-Guided Motion Encoders for Video Input

Integrate explicit 2D point trajectories from CoTracker into the Lottie Tokenizer via a cross-attention adapter to eliminate temporal motion errors in video-to-vector graphics generation.

Time

2-4 days

Compute level

Multiple GPUs

Resources

A multi-GPU machine with optical flow/tracking pre-extractors and training infrastructure for cross-attention adapters.

Prerequisites

computer vision, optical flow, transformer architectures

Scope

new model architecture

The Gap

Section 5.1 and Appendix A.2.3 note that Video-to-Lottie generation suffers from 'temporal errors' and 'motion smoothing.' The model relies on the VLM's generic video features, which struggle to decompose complex pixel dynamics into the precise Bézier curves required by the Lottie format.

“Future work could explore constrained decoding, reinforcement learning with renderability rewards, or agentic integration with professional tools like After Effects to improve reliability and practical applicability.”
From the paper, Section 6, page 11. Matched against the PDF text.

Limitation

The visual encoder extracts high-level semantics but misses the fine-grained trajectory data needed for accurate vector motion replication.

Proposed Solution

Integrate a specialized motion tracker (e.g., CoTracker or optical flow estimation) to extract explicit 2D point trajectories from the input video. Feed these trajectories as dense 'control signals' into the Lottie Tokenizer to directly guide the prediction of spatial transformation tokens (position/anchor points) via a cross-attention adapter.

Why This Matters

This would significantly improve the temporal fidelity of Video-to-Lottie conversions, making the tool viable for rotoscoping and professional animation recovery tasks.

How to Get Started

How to test it: Hypothesis: Explicit 2D point trajectories from motion trackers improve motion alignment in Video-to-Lottie generation. Baseline: OmniLottie video-to-lottie without trajectory cross-attention. Metric: Motion Alignment score evaluated by LLM judge. Stopping condition: Statistically significant improvement in Motion Alignment over baseline.

  1. Download pre-trained CoTracker and extract dense point trajectories from a benchmark video dataset like DAVIS.
  2. Implement a baseline Lottie generation model using a standard VLM backbone without motion guidance.
  3. Construct a cross-attention adapter module to inject trajectory control signals into the Lottie Tokenizer.
  4. Evaluate generated vector outputs against input videos using trajectory error and rendering fidelity metrics.

Code linked to this paper on Hugging Face: github.com/OpenVGLab/OmniLottie

Suggested by AI. Adapt it to your setup.

Has This Been Done?

Related work found

Publications like Motion Prompting and Trajectory Attention explore using trajectories or motion signals for video generation and control. They do not specifically adapt trajectory-guided motion encoders to drive spatial transformation tokens in a Lottie tokenization framework.

We searched OpenAlex for “Integrate explicit 2D point trajectories from CoTracker into the Lottie Tokenizer via a cross-attention adapter to eliminate temporal motion errors in video-to-vector graphics generation.” and “Trajectory-Guided Motion Encoders for Video Input” and an AI compared the idea with the top 6 results. A small search can miss relevant work.

Source Paper

OmniLottie: Generating Vector Animations via Parameterized Lottie Tokens

Yiying Yang, Wei Cheng, Sijin Chen +5 more · 2026

This paper presents OmniLottie, an end-to-end framework leveraging a multimodal Large Language Model (Qwen2.5-VL) to generate Lottie vector animations from text, images, or video instructions. The authors introduce a specialized "Lottie Tokenizer" to compress complex JSON structures into efficient command sequences and release MMLottie-2M, a large-scale dataset of vector animations, to train the model for high visual fidelity and semantic alignment.

Related Ideas

Is this a good idea?

Share onXLinkedIn