Skip to content
AI-generated research idea🔴Hard🔬Significant ResearchNo close match found

Infinite-Context Dual-Autoregressive Architecture for Unbounded Audiobook Synthesis

Integrate a memory-efficient sequence modeling mechanism and chunk-based reinforcement learning into an autoregressive audio backbone to support unbounded context synthesis.

Time

6 weeks

Compute level

Large cluster

Resources

32x NVIDIA H200 GPUs for 4 weeks of distributed model modification and chunk-based reinforcement learning.

What the paper used

Trained on over 10 million hours of raw audio using Qwen3-4B backbone on FSDP infrastructure.

Prerequisites

transformer architectures and reinforcement learning

Scope

new model architecture

The Gap

Section 4.2 and Section 6.1.3 describe the Slow AR backbone (Qwen3-4B) pre-training, which supports a maximum context length of 16,384 tokens. Consequently, in long-audio benchmarks, inputs exceeding 8,192 tokens are forcibly truncated at sentence boundaries.

“Since our model is pretrained to support a maximum of 8,192 context length (Section 4.2), samples exceeding this context limit are truncated at sentence boundaries.”
From the paper, Section 6.1.3, page 11. Matched against the PDF text.

Limitation

The strict context window limitation of the underlying Transformer backbone prevents the model from generating seamless, continuous, hours-long audio (such as audiobooks or podcasts) without heuristic chunking, which often loses long-term semantic and emotional consistency.

Proposed Solution

Integrate a memory-efficient sequence modeling mechanism, such as a streaming KV-cache, Transformer-XL style recurrence, or state-space models (e.g., Mamba), into the Slow AR Qwen3-4B backbone. Fine-tune this modified architecture using chunk-based reinforcement learning with a reward function that explicitly penalizes emotional and timbre drift across chunk boundaries.

Why This Matters

Unlocking unbounded context generation would eliminate the need for manual chunking and stitching, allowing for fully automated, high-fidelity, and contextually coherent long-form media production.

How to Get Started

How to test it: We hypothesize that integrating a streaming KV-cache or Mamba backbone into Qwen3-4B will enable continuous generation beyond 8,192 tokens without degradation, evaluated using Word Error Rate and WavLM speaker similarity standard deviation on multi-hour audiobooks, stopping if timbre drift exceeds a 0.12 standard deviation threshold.

  1. Incorporate a streaming KV-cache or a Transformer-XL style recurrence mechanism into the Qwen3-4B Slow AR backbone code.
  2. Design a reward function that explicitly measures and penalizes emotional and timbre drift across audio chunk boundaries.
  3. Set up a chunk-based reinforcement learning fine-tuning loop on a large compute cluster.
  4. Evaluate the extended model on long-form audiobook synthesis benchmarks exceeding 16,384 tokens.

Code linked to this paper on Hugging Face: github.com/fishaudio/fish-speech

Suggested by AI. Adapt it to your setup.

Has This Been Done?

No close match in 10 results

The listed literature contains surveys and tutorials on state-space models and generative audio, but none apply dual-autoregressive architectures or chunk-based reinforcement learning with emotion/timbre drift penalties for unbounded audiobook synthesis.

We searched OpenAlex and arXiv for “Integrate a memory-efficient sequence modeling mechanism and chunk-based reinforcement learning into an autoregressive audio backbone to support unbounded context synthesis.” and “Infinite-Context Dual-Autoregressive Architecture for Unbounded Audiobook Synthesis” and an AI compared the idea with the top 10 results. A small search can miss relevant work.

Source Paper

Fish Audio S2 Technical Report

Shijia Liao, Yuxuan Wang, Songting Liu +11 more · 2026

This paper introduces Fish Audio S2, an open-source text-to-speech (TTS) system that achieves multi-speaker, multi-turn generation and fine-grained natural language control over vocal features (e.g., emotion, prosody). The core contribution is an end-to-end framework that seamlessly integrates large-scale data curation, a highly optimized inference engine, and a novel reinforcement learning alignment process that reuses data filtering models as reward signals.

Related Ideas

Is this a good idea?

Share onXLinkedIn