Skip to content
AI-generated research idea🟠Medium🔬Significant ResearchRelated work found

Compressive Memory for Agentic Contexts

Replace heuristic history dropping with a compressive memory mechanism that encodes older context chunks into soft tokens retained in the attention cache.

Time

2 weeks

Compute level

Large cluster

Resources

A multi-GPU cluster for fine-tuning a 744B MoE model over long-context training sequences.

What the paper used

GLM-5 744B model evaluated with 202,752 maximum context length.

Prerequisites

transformer architectures

Scope

new model architecture

The Gap

In Section 4.2.4, the paper admits that model accuracy 'degrades substantially' beyond 100k tokens in agentic loops, despite being trained on 200k contexts. Their current solution is a heuristic 'Keep-recent-k' strategy that simply discards older history, causing the loss of early constraints or observations.

“Prior work [26] has introduced context management, where Discard-all resets the context by removing the entire history of tool calls.”
From the paper, Section 4.2.4, page 19. Matched against the PDF text.

Limitation

The reliance on heuristic context truncation (Discard-all/Keep-recent) lobotomizes the agent's long-term memory, preventing it from recalling critical information from the start of a long session.

Proposed Solution

Replace heuristic dropping with a 'Compressive Memory' mechanism. Fine-tune the model to compress older context chunks into soft 'memory tokens' that are retained in the DSA attention cache. This allows the model to maintain semantic access to the entire history without the linear computational cost of full attention.

Why This Matters

Solving the 'lost-in-the-middle' problem for agents is crucial for autonomous systems that need to operate over days or weeks without forgetting user instructions.

How to Get Started

How to test it: We hypothesize that compressive memory fine-tuning retains long-term session context better than heuristic truncation. Baseline is Keep-recent-k/Discard-all; metric is accuracy on BrowseComp and long-context retrieval tasks. We stop if performance falls below the heuristic baseline.

  1. Reproduce the context degradation phenomenon using a long-horizon agentic task benchmark like AgentBench.
  2. Implement a memory compression module that aggregates older KV cache states into fixed-size soft memory tokens.
  3. Fine-tune an open-source language model (such as Llama-3) on long-context QA datasets using the compressive memory loss.
  4. Benchmark the compressed memory approach against the standard 'Keep-recent-k' strategy in terms of accuracy and latency.

Code linked to this paper on Hugging Face: github.com/zai-org/GLM-5

Suggested by AI. Adapt it to your setup.

Has This Been Done?

Related work found

Several listed works explore memory compression and context management for agents and language models, including What to Keep, What to Forget: A Rate--Distortion View of Memory Compaction in LLMs and Agents and A Human-Inspired Reading Agent with Gist Memory of Very Long Contexts. However, none specifically propose fine-tuning a model to compress older context chunks into soft memory tokens retained in a DSA attention cache as detailed here.

We searched OpenAlex for “Replace heuristic history dropping with a compressive memory mechanism that encodes older context chunks into soft tokens retained in the attention cache.” and “Compressive Memory for Agentic Contexts” and an AI compared the idea with the top 10 results. A small search can miss relevant work.

Source Paper

GLM-5: from Vibe Coding to Agentic Engineering

GLM-5 Team, Aohan Zeng, Xin Lv +183 more · 2026

This paper introduces GLM-5, a 744B parameter Mixture-of-Experts foundation model optimized for agentic engineering tasks. It employs DeepSeek Sparse Attention (DSA) for efficient long-context processing and utilizes a novel asynchronous reinforcement learning pipeline that decouples generation from training to achieve state-of-the-art performance in reasoning and coding benchmarks.

Related Ideas

Is this a good idea?

Share onXLinkedIn