Infinite-Context Dual-Autoregressive Architecture for Unbounded Audiobook Synthesis
Integrate a memory-efficient sequence modeling mechanism and chunk-based reinforcement learning into an autoregressive audio backbone to support unbounded context synthesis.
Time
Compute level
Resources
What the paper used
Prerequisites
Scope
The Gap
Section 4.2 and Section 6.1.3 describe the Slow AR backbone (Qwen3-4B) pre-training, which supports a maximum context length of 16,384 tokens. Consequently, in long-audio benchmarks, inputs exceeding 8,192 tokens are forcibly truncated at sentence boundaries.
“Since our model is pretrained to support a maximum of 8,192 context length (Section 4.2), samples exceeding this context limit are truncated at sentence boundaries.”
Limitation
The strict context window limitation of the underlying Transformer backbone prevents the model from generating seamless, continuous, hours-long audio (such as audiobooks or podcasts) without heuristic chunking, which often loses long-term semantic and emotional consistency.
Proposed Solution
Integrate a memory-efficient sequence modeling mechanism, such as a streaming KV-cache, Transformer-XL style recurrence, or state-space models (e.g., Mamba), into the Slow AR Qwen3-4B backbone. Fine-tune this modified architecture using chunk-based reinforcement learning with a reward function that explicitly penalizes emotional and timbre drift across chunk boundaries.
Why This Matters
Unlocking unbounded context generation would eliminate the need for manual chunking and stitching, allowing for fully automated, high-fidelity, and contextually coherent long-form media production.
How to Get Started
How to test it: We hypothesize that integrating a streaming KV-cache or Mamba backbone into Qwen3-4B will enable continuous generation beyond 8,192 tokens without degradation, evaluated using Word Error Rate and WavLM speaker similarity standard deviation on multi-hour audiobooks, stopping if timbre drift exceeds a 0.12 standard deviation threshold.
- Incorporate a streaming KV-cache or a Transformer-XL style recurrence mechanism into the Qwen3-4B Slow AR backbone code.
- Design a reward function that explicitly measures and penalizes emotional and timbre drift across audio chunk boundaries.
- Set up a chunk-based reinforcement learning fine-tuning loop on a large compute cluster.
- Evaluate the extended model on long-form audiobook synthesis benchmarks exceeding 16,384 tokens.
Code linked to this paper on Hugging Face: github.com/fishaudio/fish-speech
Suggested by AI. Adapt it to your setup.
Has This Been Done?
The listed literature contains surveys and tutorials on state-space models and generative audio, but none apply dual-autoregressive architectures or chunk-based reinforcement learning with emotion/timbre drift penalties for unbounded audiobook synthesis.
- A Tutorial on Generative Models for Music Audio · 2026 · 0 citations
- State-space modeling in long sequence processing: a survey on recurrence in the transformer era · 2025 · 39 citations
We searched OpenAlex and arXiv for “Integrate a memory-efficient sequence modeling mechanism and chunk-based reinforcement learning into an autoregressive audio backbone to support unbounded context synthesis.” and “Infinite-Context Dual-Autoregressive Architecture for Unbounded Audiobook Synthesis” and an AI compared the idea with the top 10 results. A small search can miss relevant work.
Source Paper
Fish Audio S2 Technical ReportShijia Liao, Yuxuan Wang, Songting Liu +11 more · 2026
This paper introduces Fish Audio S2, an open-source text-to-speech (TTS) system that achieves multi-speaker, multi-turn generation and fine-grained natural language control over vocal features (e.g., emotion, prosody). The core contribution is an end-to-end framework that seamlessly integrates large-scale data curation, a highly optimized inference engine, and a novel reinforcement learning alignment process that reuses data filtering models as reward signals.
Related Ideas
- Sliding-Window Attention with Global Context for Long-Form SynthesisLTX-2: Efficient Joint Audio-Visual Foundation Model🟠Medium
- State-Space Memory Integration for Unbounded Audio ContextOmniForcing: Unleashing Real-time Joint Audio-Visual Generation⛔Very hard
- Native Long-Context Modeling via Ring AttentionSAM Audio: Segment Anything in Audio🔴Hard
- Integration of Sequence Parallelism for Long-Context TrainingLlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models🔴Hard
- Infinite-Context Linear State Generalization for Feedforward ReconstructionLoGeR: Long-Context Geometric Reconstruction with Hybrid Memory🟠Medium