Skip to content
AI-generated research idea🔴Hard🔬Significant ResearchRelated work found

Continuous Batching and Expert Co-scheduling for Multi-User SSD Streaming

Design an expert-aware continuous batching scheduler that groups requests predicting overlapping sets of experts within the same decode cycle.

Time

2-3 weeks to build the expert-aware continuous batching scheduler, extend staged slot buffers to batched expert unions, and run paired multi-tenant throughput and memory benchmarks.

Compute level

Single GPU or less

Resources

One Mac mini M4 Pro 24 GB and one MacBook M2 16 GB running the released 19.5 GB 35B and 4.5 GB 8B int4 checkpoints under simulated concurrent multi-user request traces.

What the paper used

Single-device serving experiments run on a Mac mini M4 Pro 24 GB (2.9 GiB peak active memory, 20.4 tok/s for 35B) and a MacBook M2 16 GB holding an 18.4 GiB release directory; training GPU compute is not stated.

Prerequisites

LLM serving frameworks (vLLM, TensorRT-LLM), continuous batching algorithms, systems scheduling.

Scope

New model architecture and runtime scheduler components.

The Gap

In Section 6, the authors state that EDGE0 serves one request at a time via FIFO serialization because batching fundamentally alters the expert working set in ways not supported by their per-request slot management and prerouting heads.

“Concurrency belongs to a serving layer, not the engine, and batching changes the expert working set in ways our per-request profiles do not model.”
From the paper, Section 6, page 11. Matched against the PDF text.

Limitation

The framework cannot support multi-tenant or concurrent serving; concurrently active sequences cause expert union explosion across layers, causing SSD bandwidth saturation and catastrophic cache thrashing.

Proposed Solution

Design an expert-aware continuous batching scheduler. Extend the prerouter heads to output batch-wide expert priority scores, and implement an expert affinity-based request dispatcher that co-schedules sequences predicting overlapping sets of experts within the same decode cycle, bounding the total active expert footprint to fixed buffer limits.

Why This Matters

Enables EDGE0 to scale from single-user desktop utilities to multi-user on-premise local servers, multiplying aggregate token serving throughput across concurrent users.

How to Get Started

How to test it: Hypothesis: An expert-affinity continuous batching scheduler driven by prerouter scores bounds the per-step union of active experts and increases aggregate decode throughput over FIFO serving within a 24 GB memory budget. Baseline: FIFO-serialized Edge0-35B and naive uncoordinated batching on Mac mini M4 Pro 24 GB; Metric: Aggregate tok/s across batch sizes 2-8, MiB read from SSD per step, and peak active memory; Stop if batching causes SSD thrashing or active memory growth that yields less than 1.2x FIFO aggregate throughput or causes OOM/swapping on 24 GB.

  1. Modify the prerouter heads to output batch-wide expert priority scores rather than single-request selections.
  2. Implement an expert affinity-based request dispatcher queue that groups sequences requiring shared expert blocks.
  3. Integrate the scheduler into the local SSD streaming evaluation framework to handle concurrent user requests.
  4. Benchmark throughput, expert cache hit rates, and latency on a workstation equipped with high-speed PCIe NVMe SSDs.

Code linked to this paper on Hugging Face: github.com/Edge0-AI/edge0

Suggested by AI. Adapt it to your setup.

Has This Been Done?

Related work found

Papers such as LayerScope, HarMoEny, and HCRMap discuss expert-aware scheduling, cross-layer scheduling, and expert residency mapping for Mixture-of-Experts inference. However, they do not specifically implement the proposed continuous batching scheduler with prerouter output scores for multi-user SSD streaming.

We searched OpenAlex and arXiv for “Design an expert-aware continuous batching scheduler that groups requests predicting overlapping sets of experts within the same decode cycle.” and “Continuous Batching and Expert Co-scheduling for Multi-User SSD Streaming” and an AI compared the idea with the top 10 results. A small search can miss relevant work.

Source Paper

The Other Half of the Memory Wall: Serving 35B MoEs from SSD with Trained Routing Prediction

Yu Lin, Yiming Wang, Runyuan Cai +2 more · 2026

The paper introduces EDGE0, a streaming inference framework designed to serve large mixture-of-experts (MoE) models (such as 35B and 8B variants) on consumer devices constrained by limited RAM. It overcomes SSD loading latency by using a cross-token prerouter that predicts routing one token ahead, executes the prediction directly as the actual routing to completely overlap SSD expert streaming with compute, and restores accuracy loss from int4 quantization and routing replacement via an unmerged distillation LoRA.

Related Ideas

Is this a good idea?

Share onXLinkedIn