Joint Inter-DP and Intra-Sequence Layout Optimization for Multi-Document Sparse Attention
A method that unifies inter-data-parallel sample dispatch and intra-sequence query-key block folding to balance computational load during long-context sparse attention pretraining.
Time
Compute level
Resources
What the paper used
Prerequisites
Scope
The Gap
Section 3.4.2 details Q-block pairing and layout planning under DeepSeek Sparse Attention (DSA) across data-parallel replicas and context-parallel ranks for multi-document sequences. Document-isolated causal masking creates non-monotonic workloads across tokens.
“The current DSA data scoring still estimates cost under fixed folding and does not automatically call dynamic paired owners, so when the two are combined the cost objectives should be checked for consistency”
Limitation
Inter-DP sample scheduling and intra-replica dynamic pairing operate as decoupled heuristics; the inter-DP scoring estimates cost under fixed folding assumptions without accounting for dynamic paired owners, leading to sub-optimal load balancing across data-parallel ranks.
Proposed Solution
Formulate a bi-level optimization scheduler that couples inter-DP micro-batch assignment with intra-sequence CP/TP folding. Integrate an empirical DSA kernel cost model—reflecting actual LightningIndexer scan times and SparseFlashAttention memory overheads—into the global dispatch objective. Use this joint cost model to solve a unified assignment problem that simultaneously allocates micro-batches across DP ranks via greedy bin-packing and determines complementary head-tail Q-block folding within CP groups before tensor execution.
Why This Matters
Addressing this disconnect is expected to eliminate straggler ranks in large-scale long-context training, improving cluster-wide NPU/GPU utilization and reducing iteration step times.
How to Get Started
How to test it: Train an MoE model with DSA on multi-document packed sequences of 256K context on 32-64 devices; the joint scheduler should reduce maximum rank idle time during attention by over 20% compared to the decoupled SNAKE baseline, stopping if scheduler runtime overhead exceeds 1% of the step time.
- Profile LightningIndexer and SparseFlashAttention kernel runtimes across varied document length distributions to parameterize an accurate cost function.
- Implement a prototype offline scheduler that jointly optimizes DP micro-batch distribution and CP block pairing.
- Integrate the joint scheduler into the data loading pipeline of MindSpore HyperParallel or Megatron-LM.
- Run multi-rank pretraining steps on synthetic multi-document workloads to verify reduction in synchronization wait times.
Suggested by AI. Adapt it to your setup.
Has This Been Done?
Papers like 'SparseBalance: Load-Balanced Long Context Training with Dynamic Sparse Attention' and 'SparseServe: Unlocking Parallelism for Dynamic Sparse Attention in Long-Context LLM Serving' discuss parallelization and load balancing for dynamic sparse attention mechanisms. However, they leave open the specific bi-level joint optimization scheduler coupling inter-DP micro-batch assignment with intra-sequence CP/TP folding using an empirical DSA kernel cost model.
- SparseServe: Unlocking Parallelism for Dynamic Sparse Attention in Long-Context LLM Serving · 2026 · 14 citations
- SparseBalance: Load-Balanced Long Context Training with Dynamic Sparse Attention · 2026 · 1 citations
- S-HPLB: Efficient LLM Attention Serving via Sparsity-Aware Head Parallelism Load Balance · 2026 · 1 citations
We searched OpenAlex and arXiv for “dynamic load balancing context parallelism sparse attention” and “Joint Inter-DP and Intra-Sequence Layout Optimization for Multi-Document Sparse Attention” and an AI compared the idea with the top 10 results. A small search can miss relevant work.
Source Paper
TeleChat to Xing4: Advancing Frontier Agentic Models on Domestic Chips and FrameworksXingchen AGI Lab, Baotian Hu, Bingkai Yang +136 more · 2026
This technical report presents Xing 4, a family of large language models scaling up to hundreds of billions of parameters trained natively on Ascend 910C NPUs using MindSpore HyperParallel. The report establishes design principles combining sparse attention, wider-and-shallower MoE topologies, and speculative decoding, supported by systems optimizations including hybrid sharded data parallelism, owner-based Muon execution, and CPU core isolation. It also introduces TeleGym, a unified execution and reinforcement learning platform for long-horizon agent and reasoning workloads.
Related Ideas
- Integration of Sequence Parallelism for Long-Context TrainingLlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models🔴Hard
- Multi-Shape Generalization and End-to-End Workload Evaluation for NPU Kernel OptimizationAscendGYM: A Cost-Aware Kernel Optimization Benchmark for Real-World Workloads on Ascend NPUs🟠Medium
- Hybrid Kernel Scheduling for Prefill OptimizationBitnet.cpp: Efficient Edge Inference for Ternary LLMs🟡Fairly easy
- End-to-End Trainable Sparse Attention Indexers for RLGLM-5: from Vibe Coding to Agentic Engineering🔴Hard
- Dual-Axis KV Sharing: Integrating Joint-KV with Cross-Layer and Prefix-Shared AttentionMulti-Branch Transformer🟠Medium