Skip to content
AI-generated research idea🔴Hard🔬Significant ResearchNo close match found

Homogeneous Mamba-Based Audio Tokenization

Replace Causal Transformer blocks in audio tokenizers with Mamba selective state space models to achieve linear computational complexity and capture global dependencies.

Time

2-3 weeks

Compute level

Large cluster

Resources

A multi-GPU cluster with at least 8x A100 (80GB) or H100 GPUs for training on multi-hour audio datasets.

What the paper used

1.6 billion parameters, pre-trained on 3 million hours of diverse audio data

Prerequisites

transformer architectures

Scope

new model architecture

The Gap

Section 3.1 advocates for a 'homogeneous' Transformer architecture. However, the use of sliding-window attention (10s window) incurs quadratic computational costs within the window and fails to capture truly global dependencies beyond that window, limiting efficiency on long-form audio.

Limitation

Standard Transformers, even with sliding windows, are computationally heavy for high-resolution audio sequences and lack a global receptive field for very long recordings.

Proposed Solution

Replace the Causal Transformer blocks in the CAT architecture with Selective State Space Model (SSM) blocks, specifically the Mamba architecture. This would maintain the 'homogeneous' design philosophy while reducing complexity from quadratic to linear and enabling a global receptive field without windowing artifacts.

Why This Matters

This would significantly reduce inference latency and memory usage, enabling the processing of much longer audio contexts (e.g., hour-long podcasts) which is critical for future foundation models.

How to Get Started

How to test it: We hypothesize that replacing Transformer blocks with Mamba SSM blocks reduces computational complexity while maintaining reconstruction quality. We will compare against the CAT Transformer baseline on STOI and PESQ metrics. If reconstruction fidelity drops significantly below the baseline, we will stop.

  1. Set up the baseline CAT architecture and reproduce sliding-window attention performance metrics on long-form audio.
  2. Integrate Mamba selective state space model blocks into the audio tokenization pipeline.
  3. Train the Mamba-based audio tokenizer on a large-scale audio corpus using a large compute cluster.
  4. Benchmark inference speed, memory footprint, and reconstruction quality against the original Transformer baseline.

Code linked to this paper on Hugging Face: github.com/OpenMOSS/MOSS-Audio-Tokenizer

Suggested by AI. Adapt it to your setup.

Has This Been Done?

No close match in 6 results

While 'Audio Mamba: Bidirectional State Space Model for Audio Representation Learning' and survey papers like 'Mamba-360' explore Mamba for audio and general sequence modeling, none of the listed works specifically substitute Causal Transformer blocks with Mamba within the CAT audio tokenization architecture. The listed papers leave the specific integration into the CAT architecture unexplored.

We searched OpenAlex for “Replace Causal Transformer blocks in audio tokenizers with Mamba selective state space models to achieve linear computational complexity and capture global dependencies.” and “Homogeneous Mamba-Based Audio Tokenization” and an AI compared the idea with the top 6 results. A small search can miss relevant work.

Source Paper

MOSS-Audio-Tokenizer: Scaling Audio Tokenizers for Future Audio Foundation Models

Yitian Gong, Kuangwei Chen, Zhaoye Fei +9 more · 2026

This paper introduces MOSS-Audio-Tokenizer, a unified discrete audio tokenizer designed for audio foundation models. It utilizes a homogeneous, purely Transformer-based architecture (CAT) without CNNs, trained end-to-end on 3 million hours of audio to achieve state-of-the-art reconstruction and generation capabilities.

Related Ideas

Is this a good idea?

Share onXLinkedIn