Streamable SAN-M Architecture for Real-Time Interaction
Enable real-time streaming speech recognition by modifying memory-equipped self-attention encoders with causal chunk-based masking and Transducer losses.
Time
Compute level
Resources
Prerequisites
Scope
The Gap
The SenseVoice-Small model (Section 2.2) uses a memory-equipped self-attention network (SAN-M) designed for non-autoregressive, end-to-end processing. However, the authors state in the limitations that the model is not designed for streaming transcription.
Limitation
The current architecture requires the entire audio segment to be processed before returning a result, making it unsuitable for live captions or real-time human-agent translation.
Proposed Solution
Modify the SAN-M encoder to support chunk-based processing with causal attention masks. Implement a streaming Transducer (RNN-T) loss alongside the existing CTC loss to enable the model to emit tokens incrementally as audio frames are received.
Why This Matters
Enabling streaming capabilities is essential for real-world deployment in low-latency environments such as live telecommunications, accessibility tools, and interactive voice assistants.
How to Get Started
- Set up a baseline training pipeline using public ASR datasets like LibriSpeech and Common Voice.
- Modify the memory-equipped self-attention network encoder to support chunk-based processing with causal attention masks.
- Implement an RNN-T loss function alongside the existing CTC loss in the training framework.
- Evaluate real-time streaming latency, token emission speed, and word error rate on benchmark validation sets.
Code linked to this paper on Hugging Face: github.com/funaudiollm/cosyvoice
Suggested by AI. Adapt it to your setup.
Has This Been Done?
The listed works cover surveys on attention-based ASR and label-synchronous neural transducers for online recognition and translation, but none propose modifying the SAN-M architecture with chunk-based processing and streaming RNN-T loss.
- Thank you for attention: A survey on attention-based artificial neural networks for automatic speech recognition · 2024 · 47 citations
- Label-Synchronous Neural Transducer for Adaptable Online E2E Speech Recognition · 2024 · 14 citations
- Label-Synchronous Neural Transducer for E2E Simultaneous Speech Translation · 2024 · 9 citations
We searched OpenAlex for “Enable real-time streaming speech recognition by modifying memory-equipped self-attention encoders with causal chunk-based masking and Transducer losses.” and “Streamable SAN-M Architecture for Real-Time Interaction” and an AI compared the idea with the top 10 results. A small search can miss relevant work.
Source Paper
FunAudioLLM: Voice Understanding and Generation Foundation Models for Natural Interaction Between Humans and LLMsTongyi SpeechTeam · 2024
FunAudioLLM is a foundation model family comprising SenseVoice for multilingual speech understanding and CosyVoice for natural voice generation. These models aim to bridge the gap between human voice interaction and large language models (LLMs) by providing high-speed ASR, emotion recognition, and zero-shot voice cloning capabilities.
Related Ideas
- Streaming-Compatible Architecture via Block-Causal AttentionFlavors of Moonshine: Tiny Specialized ASR Models for Edge Devices🔴Hard
- State-Space Memory Integration for Unbounded Audio ContextOmniForcing: Unleashing Real-time Joint Audio-Visual Generation⛔Very hard
- Causal Diffusion Decoding for 25Hz Semantic TokensQwen3-TTS Technical Report🔴Hard
- Sliding-Window Attention with Global Context for Long-Form SynthesisLTX-2: Efficient Joint Audio-Visual Foundation Model🟠Medium
- Homogeneous Mamba-Based Audio TokenizationMOSS-Audio-Tokenizer: Scaling Audio Tokenizers for Future Audio Foundation Models🔴Hard