Homogeneous Mamba-Based Audio Tokenization
Replace Causal Transformer blocks in audio tokenizers with Mamba selective state space models to achieve linear computational complexity and capture global dependencies.
Time
Compute level
Resources
What the paper used
Prerequisites
Scope
The Gap
Section 3.1 advocates for a 'homogeneous' Transformer architecture. However, the use of sliding-window attention (10s window) incurs quadratic computational costs within the window and fails to capture truly global dependencies beyond that window, limiting efficiency on long-form audio.
Limitation
Standard Transformers, even with sliding windows, are computationally heavy for high-resolution audio sequences and lack a global receptive field for very long recordings.
Proposed Solution
Replace the Causal Transformer blocks in the CAT architecture with Selective State Space Model (SSM) blocks, specifically the Mamba architecture. This would maintain the 'homogeneous' design philosophy while reducing complexity from quadratic to linear and enabling a global receptive field without windowing artifacts.
Why This Matters
This would significantly reduce inference latency and memory usage, enabling the processing of much longer audio contexts (e.g., hour-long podcasts) which is critical for future foundation models.
How to Get Started
How to test it: We hypothesize that replacing Transformer blocks with Mamba SSM blocks reduces computational complexity while maintaining reconstruction quality. We will compare against the CAT Transformer baseline on STOI and PESQ metrics. If reconstruction fidelity drops significantly below the baseline, we will stop.
- Set up the baseline CAT architecture and reproduce sliding-window attention performance metrics on long-form audio.
- Integrate Mamba selective state space model blocks into the audio tokenization pipeline.
- Train the Mamba-based audio tokenizer on a large-scale audio corpus using a large compute cluster.
- Benchmark inference speed, memory footprint, and reconstruction quality against the original Transformer baseline.
Code linked to this paper on Hugging Face: github.com/OpenMOSS/MOSS-Audio-Tokenizer
Suggested by AI. Adapt it to your setup.
Has This Been Done?
While 'Audio Mamba: Bidirectional State Space Model for Audio Representation Learning' and survey papers like 'Mamba-360' explore Mamba for audio and general sequence modeling, none of the listed works specifically substitute Causal Transformer blocks with Mamba within the CAT audio tokenization architecture. The listed papers leave the specific integration into the CAT architecture unexplored.
- Audio Mamba: Bidirectional State Space Model for Audio Representation Learning · 2024 · 87 citations
- Mamba-360: Survey of State Space Models as Transformer Alternative for Long Sequence Modelling: Methods, Applications, and Challenges · 2024 · 77 citations
We searched OpenAlex for “Replace Causal Transformer blocks in audio tokenizers with Mamba selective state space models to achieve linear computational complexity and capture global dependencies.” and “Homogeneous Mamba-Based Audio Tokenization” and an AI compared the idea with the top 6 results. A small search can miss relevant work.
Source Paper
MOSS-Audio-Tokenizer: Scaling Audio Tokenizers for Future Audio Foundation ModelsYitian Gong, Kuangwei Chen, Zhaoye Fei +9 more · 2026
This paper introduces MOSS-Audio-Tokenizer, a unified discrete audio tokenizer designed for audio foundation models. It utilizes a homogeneous, purely Transformer-based architecture (CAT) without CNNs, trained end-to-end on 3 million hours of audio to achieve state-of-the-art reconstruction and generation capabilities.
Related Ideas
- State-Space Memory Integration for Unbounded Audio ContextOmniForcing: Unleashing Real-time Joint Audio-Visual Generation⛔Very hard
- Streaming-Compatible Architecture via Block-Causal AttentionFlavors of Moonshine: Tiny Specialized ASR Models for Edge Devices🔴Hard
- Linear-Complexity Backbone via State Space ModelsA decoder-only foundation model for time-series forecasting⛔Very hard
- Sliding-Window Attention with Global Context for Long-Form SynthesisLTX-2: Efficient Joint Audio-Visual Foundation Model🟠Medium
- Native Long-Context Modeling via Ring AttentionSAM Audio: Segment Anything in Audio🔴Hard