Visual Token Compression for Extended Context History
Implement visual token merging to compress historical screenshots into sparse feature vectors, maintaining long-term visual memory without exceeding context window limits.
Time
Compute level
Resources
What the paper used
Prerequisites
Scope
The Gap
In Section 3.1, the authors state: 'keeping the full history in the context window becomes computationally intensive... we keep only the most recent N=3 observations and remove earlier ones.'
“we keep only the most recent N observations in the history and remove earlier ones”
Limitation
This truncation causes catastrophic forgetting in long-horizon tasks. For example, if a price comparison task requires recalling a price seen 10 steps ago on a different URL, the model physically cannot see it once it falls out of the N=3 window.
Proposed Solution
Implement a 'Visual Memory' module or Visual Token Merging (e.g., ToMe) to compress historical screenshots into sparse feature vectors rather than discarding them entirely. This would allow the model to retain visual cues from the start of the trajectory without blowing up the context window.
Why This Matters
Enables the agent to solve tasks requiring information retention across multiple web pages and long navigational sequences, significantly improving performance on the 'Compositional' and 'Comparison' benchmarks.
How to Get Started
How to test it: Hypothesis: Visual token compression (ToMe) prevents catastrophic forgetting in long-horizon tasks while fitting within the N=3 frame limit. Baseline: Standard Fara-7B dropping older screenshots. Metric: Accuracy on multi-step tasks requiring long-range memory (e.g., Shopping Lists). Stop if accuracy decreases or latency increases significantly.
- Analyze the context length impact of retaining full historical screenshots in multi-step visual agents.
- Implement a visual token merging module based on spatial and temporal redundancy reduction.
- Integrate the compressed visual memory buffer into the observation history pipeline.
- Benchmark task success rates on long-horizon visual tasks against baseline models that drop historical frames.
Code linked to this paper on Hugging Face: github.com/microsoft/fara
Suggested by AI. Adapt it to your setup.
Has This Been Done?
Works like Glyph: Scaling Context Windows via Visual-Text Compression and Vision-centric Token Compression in Large Language Model explore visual token compression and scaling context windows. However, they do not specifically implement the proposed visual memory module using visual token merging tailored for historical screenshots in agent trajectories.
- Compression Tells Intelligence: Visual Coding, Visual Token Technology, and the Unification · 2026 · 3 citations
- Glyph: Scaling Context Windows via Visual-Text Compression · 2026 · 68 citations
- Vision-centric Token Compression in Large Language Model · 2025 · 61 citations
- Planning-Aligned Token Compression for Long-Context Autonomous Driving · 2026 · 3 citations
We searched OpenAlex for “Implement visual token merging to compress historical screenshots into sparse feature vectors, maintaining long-term visual memory without exceeding context window limits.” and “Visual Token Compression for Extended Context History” and an AI compared the idea with the top 10 results. A small search can miss relevant work.
Source Paper
Fara-7B: An Efficient Agentic Model for Computer UseAhmed Awadallah, Yash Lara, Raghav Magazine +9 more · 2025
This paper presents Fara-7B, a 7-billion parameter vision-language model specialized for computer use agents (CUAs), trained on a novel synthetic dataset generated by the FaraGen engine. FaraGen overcomes data scarcity by using a multi-agent system to propose, solve, and verify 145k web interaction trajectories, enabling Fara-7B to achieve state-of-the-art performance among 7B models on benchmarks like WebVoyager and the newly introduced WebTailBench.
Related Ideas
- Adaptive Attention-Based Memory CompressionHelios: Real Real-Time Long Video Generation Model🟠Medium
- Recurrent Visual Memory for Infinite-Context TrajectoriesMobile-Agent-v3: Foundamental Agents for GUI Automation🔴Hard
- Query-Adaptive Video Context Compression and Frame-Level Visual Dependence for Long-Video Speculative DraftingMMFlash: Faster Speculative Decoding for Vision-Language Models🟠Medium
- Uncertainty-Triggered Visual Frame Retention for Long-Horizon GUI and Hybrid AgentsSage S2.1 Flash Technical Report🟠Medium
- Semantic Compressive Memory for Long-Horizon ContextMiroThinker: Pushing the Performance Boundaries of Open-Source Research Agents via Model, Context, and Interactive Scaling🟠Medium