Skip to content
AI-generated research idea🔴Hard🔬Significant ResearchRelated work found

Visual Token Compression for Extended Context History

Implement visual token merging to compress historical screenshots into sparse feature vectors, maintaining long-term visual memory without exceeding context window limits.

Time

1-3 days

Compute level

Multiple GPUs

Resources

A multi-GPU training setup for evaluating and fine-tuning the modified VLM architecture.

What the paper used

145K trajectories with 1 million steps, trained on 64 H100 GPUs

Prerequisites

transformer architectures, computer vision

Scope

new model architecture

The Gap

In Section 3.1, the authors state: 'keeping the full history in the context window becomes computationally intensive... we keep only the most recent N=3 observations and remove earlier ones.'

“we keep only the most recent N observations in the history and remove earlier ones”
From the paper, Page 12. Matched against the PDF text.

Limitation

This truncation causes catastrophic forgetting in long-horizon tasks. For example, if a price comparison task requires recalling a price seen 10 steps ago on a different URL, the model physically cannot see it once it falls out of the N=3 window.

Proposed Solution

Implement a 'Visual Memory' module or Visual Token Merging (e.g., ToMe) to compress historical screenshots into sparse feature vectors rather than discarding them entirely. This would allow the model to retain visual cues from the start of the trajectory without blowing up the context window.

Why This Matters

Enables the agent to solve tasks requiring information retention across multiple web pages and long navigational sequences, significantly improving performance on the 'Compositional' and 'Comparison' benchmarks.

How to Get Started

How to test it: Hypothesis: Visual token compression (ToMe) prevents catastrophic forgetting in long-horizon tasks while fitting within the N=3 frame limit. Baseline: Standard Fara-7B dropping older screenshots. Metric: Accuracy on multi-step tasks requiring long-range memory (e.g., Shopping Lists). Stop if accuracy decreases or latency increases significantly.

  1. Analyze the context length impact of retaining full historical screenshots in multi-step visual agents.
  2. Implement a visual token merging module based on spatial and temporal redundancy reduction.
  3. Integrate the compressed visual memory buffer into the observation history pipeline.
  4. Benchmark task success rates on long-horizon visual tasks against baseline models that drop historical frames.

Code linked to this paper on Hugging Face: github.com/microsoft/fara

Suggested by AI. Adapt it to your setup.

Has This Been Done?

Related work found

Works like Glyph: Scaling Context Windows via Visual-Text Compression and Vision-centric Token Compression in Large Language Model explore visual token compression and scaling context windows. However, they do not specifically implement the proposed visual memory module using visual token merging tailored for historical screenshots in agent trajectories.

We searched OpenAlex for “Implement visual token merging to compress historical screenshots into sparse feature vectors, maintaining long-term visual memory without exceeding context window limits.” and “Visual Token Compression for Extended Context History” and an AI compared the idea with the top 10 results. A small search can miss relevant work.

Source Paper

Fara-7B: An Efficient Agentic Model for Computer Use

Ahmed Awadallah, Yash Lara, Raghav Magazine +9 more · 2025

This paper presents Fara-7B, a 7-billion parameter vision-language model specialized for computer use agents (CUAs), trained on a novel synthetic dataset generated by the FaraGen engine. FaraGen overcomes data scarcity by using a multi-agent system to propose, solve, and verify 145k web interaction trajectories, enabling Fara-7B to achieve state-of-the-art performance among 7B models on benchmarks like WebVoyager and the newly introduced WebTailBench.

Related Ideas

Is this a good idea?

Share onXLinkedIn