Skip to content
AI-generated research idea🟠Medium💡Moderate InnovationRelated work found

Joint Inter-DP and Intra-Sequence Layout Optimization for Multi-Document Sparse Attention

A method that unifies inter-data-parallel sample dispatch and intra-sequence query-key block folding to balance computational load during long-context sparse attention pretraining.

Time

several weeks

Compute level

Multiple GPUs

Resources

Distributed cluster with 32 to 64 accelerators, multi-document long-sequence pretraining dataset

What the paper used

Evaluated on a 1,024-NPU Ascend profiling cluster and scaled to 8,192 Ascend 910C cards with sequence lengths of 262,144 tokens.

Prerequisites

distributed training systems, sparse attention kernels, parallel scheduling

Scope

modify existing code

The Gap

Section 3.4.2 details Q-block pairing and layout planning under DeepSeek Sparse Attention (DSA) across data-parallel replicas and context-parallel ranks for multi-document sequences. Document-isolated causal masking creates non-monotonic workloads across tokens.

“The current DSA data scoring still estimates cost under fixed folding and does not automatically call dynamic paired owners, so when the two are combined the cost objectives should be checked for consistency”
From the paper, Section 3.4.2, page 21. Matched against the PDF text.

Limitation

Inter-DP sample scheduling and intra-replica dynamic pairing operate as decoupled heuristics; the inter-DP scoring estimates cost under fixed folding assumptions without accounting for dynamic paired owners, leading to sub-optimal load balancing across data-parallel ranks.

Proposed Solution

Formulate a bi-level optimization scheduler that couples inter-DP micro-batch assignment with intra-sequence CP/TP folding. Integrate an empirical DSA kernel cost model—reflecting actual LightningIndexer scan times and SparseFlashAttention memory overheads—into the global dispatch objective. Use this joint cost model to solve a unified assignment problem that simultaneously allocates micro-batches across DP ranks via greedy bin-packing and determines complementary head-tail Q-block folding within CP groups before tensor execution.

Why This Matters

Addressing this disconnect is expected to eliminate straggler ranks in large-scale long-context training, improving cluster-wide NPU/GPU utilization and reducing iteration step times.

How to Get Started

How to test it: Train an MoE model with DSA on multi-document packed sequences of 256K context on 32-64 devices; the joint scheduler should reduce maximum rank idle time during attention by over 20% compared to the decoupled SNAKE baseline, stopping if scheduler runtime overhead exceeds 1% of the step time.

  1. Profile LightningIndexer and SparseFlashAttention kernel runtimes across varied document length distributions to parameterize an accurate cost function.
  2. Implement a prototype offline scheduler that jointly optimizes DP micro-batch distribution and CP block pairing.
  3. Integrate the joint scheduler into the data loading pipeline of MindSpore HyperParallel or Megatron-LM.
  4. Run multi-rank pretraining steps on synthetic multi-document workloads to verify reduction in synchronization wait times.

Suggested by AI. Adapt it to your setup.

Has This Been Done?

Related work found

Papers like 'SparseBalance: Load-Balanced Long Context Training with Dynamic Sparse Attention' and 'SparseServe: Unlocking Parallelism for Dynamic Sparse Attention in Long-Context LLM Serving' discuss parallelization and load balancing for dynamic sparse attention mechanisms. However, they leave open the specific bi-level joint optimization scheduler coupling inter-DP micro-batch assignment with intra-sequence CP/TP folding using an empirical DSA kernel cost model.

We searched OpenAlex and arXiv for “dynamic load balancing context parallelism sparse attention” and “Joint Inter-DP and Intra-Sequence Layout Optimization for Multi-Document Sparse Attention” and an AI compared the idea with the top 10 results. A small search can miss relevant work.

Source Paper

TeleChat to Xing4: Advancing Frontier Agentic Models on Domestic Chips and Frameworks

Xingchen AGI Lab, Baotian Hu, Bingkai Yang +136 more · 2026

This technical report presents Xing 4, a family of large language models scaling up to hundreds of billions of parameters trained natively on Ascend 910C NPUs using MindSpore HyperParallel. The report establishes design principles combining sparse attention, wider-and-shallower MoE topologies, and speculative decoding, supported by systems optimizations including hybrid sharded data parallelism, owner-based Muon execution, and CPU core isolation. It also introduces TeleGym, a unified execution and reinforcement learning platform for long-horizon agent and reasoning workloads.

Related Ideas

Is this a good idea?

Share onXLinkedIn