Trajectory-Guided Motion Encoders for Video Input
Integrate explicit 2D point trajectories from CoTracker into the Lottie Tokenizer via a cross-attention adapter to eliminate temporal motion errors in video-to-vector graphics generation.
Time
Compute level
Resources
Prerequisites
Scope
The Gap
Section 5.1 and Appendix A.2.3 note that Video-to-Lottie generation suffers from 'temporal errors' and 'motion smoothing.' The model relies on the VLM's generic video features, which struggle to decompose complex pixel dynamics into the precise Bézier curves required by the Lottie format.
“Future work could explore constrained decoding, reinforcement learning with renderability rewards, or agentic integration with professional tools like After Effects to improve reliability and practical applicability.”
Limitation
The visual encoder extracts high-level semantics but misses the fine-grained trajectory data needed for accurate vector motion replication.
Proposed Solution
Integrate a specialized motion tracker (e.g., CoTracker or optical flow estimation) to extract explicit 2D point trajectories from the input video. Feed these trajectories as dense 'control signals' into the Lottie Tokenizer to directly guide the prediction of spatial transformation tokens (position/anchor points) via a cross-attention adapter.
Why This Matters
This would significantly improve the temporal fidelity of Video-to-Lottie conversions, making the tool viable for rotoscoping and professional animation recovery tasks.
How to Get Started
How to test it: Hypothesis: Explicit 2D point trajectories from motion trackers improve motion alignment in Video-to-Lottie generation. Baseline: OmniLottie video-to-lottie without trajectory cross-attention. Metric: Motion Alignment score evaluated by LLM judge. Stopping condition: Statistically significant improvement in Motion Alignment over baseline.
- Download pre-trained CoTracker and extract dense point trajectories from a benchmark video dataset like DAVIS.
- Implement a baseline Lottie generation model using a standard VLM backbone without motion guidance.
- Construct a cross-attention adapter module to inject trajectory control signals into the Lottie Tokenizer.
- Evaluate generated vector outputs against input videos using trajectory error and rendering fidelity metrics.
Code linked to this paper on Hugging Face: github.com/OpenVGLab/OmniLottie
Suggested by AI. Adapt it to your setup.
Has This Been Done?
Publications like Motion Prompting and Trajectory Attention explore using trajectories or motion signals for video generation and control. They do not specifically adapt trajectory-guided motion encoders to drive spatial transformation tokens in a Lottie tokenization framework.
- Motion Prompting: Controlling Video Generation with Motion Trajectories · 2025 · 212 citations
- Trajectory Attention for Fine-grained Video Motion Control · 2024 · 40 citations
We searched OpenAlex for “Integrate explicit 2D point trajectories from CoTracker into the Lottie Tokenizer via a cross-attention adapter to eliminate temporal motion errors in video-to-vector graphics generation.” and “Trajectory-Guided Motion Encoders for Video Input” and an AI compared the idea with the top 6 results. A small search can miss relevant work.
Source Paper
OmniLottie: Generating Vector Animations via Parameterized Lottie TokensYiying Yang, Wei Cheng, Sijin Chen +5 more · 2026
This paper presents OmniLottie, an end-to-end framework leveraging a multimodal Large Language Model (Qwen2.5-VL) to generate Lottie vector animations from text, images, or video instructions. The authors introduce a specialized "Lottie Tokenizer" to compress complex JSON structures into efficient command sequences and release MMLottie-2M, a large-scale dataset of vector animations, to train the model for high visual fidelity and semantic alignment.
Related Ideas
- Explicit Motion-Guided Control for Video EditingUniVideo: Unified Understanding, Generation, and Editing for Videos🟠Medium
- Native Video-Based Instructive Editing PipelineOmniVCus: Feedforward Subject-driven Video Customization with Multimodal Control Conditions🔴Hard
- Closed-Loop Visual-Feedback Agent for Complex Camera Trajectories and Multi-Event ContinuityMovieRoll: A Systematic Framework for Rolling Out Long-Form Videos🟠Medium
- Motion-Aware Transition Modeling via Multi-Frame ConditioningStoryMem: Multi-shot Long Video Storytelling with Memory🟡Fairly easy
- Temporal-Native Tools for Motion AnalysisPyVision-RL: Forging Open Agentic Vision Models via RL🟠Medium