Hierarchical Edge-Cloud Architecture for Sub-Second Reactive Replanning
Pair slow cloud frontier models with lightweight local policies for sub-second reactive parameter adaptation during dynamic robotic contact tasks.
Time
Compute level
Resources
What the paper used
Prerequisites
Scope
The Gap
Section 3.1 and Section J.1 note that the frontier model is queried via a remote API that incurs multi-second latency between turns. Primitives must run quasi-statically until self-termination, preventing the system from dynamically altering trajectory or force targets during execution.
“Querying a remote frontier model adds seconds of latency between primitives, which suits quasi-static contact tasks but precludes fast reactive replanning during dynamic motion.”
Limitation
API query latencies preclude dynamic replanning and reactive adjustments to non-stationary contact conditions during primitive execution.
Proposed Solution
Deploy a hierarchical edge-cloud architecture pairing the remote frontier model with an on-device lightweight multimodal model (e.g., 3B-7B parameters) running at 10-20 Hz. The remote model establishes high-level task plans and parameter search spaces, while the local model ingests streaming 20 Hz force-torque readings and wrist images to modulate admittance parameters (such as lead clamp L, target velocity v, or normal force Fd) mid-primitive.
Why This Matters
Overcoming the API latency bottleneck allows frozen-model robotic systems to handle dynamic and time-sensitive interactions like catching slipping objects or wiping moving surfaces.
How to Get Started
How to test it: Test on a dynamic slipping or unjamming task in MuJoCo comparing standard ForceHarness against the edge-cloud hierarchy, measuring recovery success rate and episode execution time. Stop if edge inference adds more than 100 ms loop latency or degrades admittance controller stability.
- Benchmark the latency profile of local 3B-7B parameter VLMs fine-tuned or prompted to predict continuous scalar control deltas from short force windows.
- Modify the ForceHarness inner loop in Figure A1 to accept asynchronous parameter updates from an edge process.
- Evaluate the hybrid controller on HiddenPhys Peg and Latch tasks subjected to time-varying external perturbations.
Suggested by AI. Adapt it to your setup.
Has This Been Done?
Papers such as IROS: A Dual-Process Architecture for Real-Time VLM-Based Indoor Navigation and Slow Brain, Fast Planner investigate hierarchical dual-process or edge-cloud configurations for latency-resilient robotics. They do not specifically target the setup of combining a remote frontier model with an on-device lightweight multimodal model ingesting 20 Hz force-torque readings for mid-primitive admittance modulation.
- IROS: A Dual-Process Architecture for Real-Time VLM-Based Indoor Navigation · 2026 · 8 citations
- Asynchronous Robot Policies · 2026 · 0 citations
- Slow Brain, Fast Planner: Latency-Resilient VLM-Augmented Urban Navigation · 2026 · 3 citations
We searched OpenAlex and arXiv for “hierarchical VLM robot control edge cloud reactive force” and “Hierarchical Edge-Cloud Architecture for Sub-Second Reactive Replanning” and an AI compared the idea with the top 10 results. A small search can miss relevant work.
Source Paper
ForceHarness: Turning Force into Action and Physics for Frozen Frontier ModelsKe Sun, Kevin Yuchen Ma, Yike Zhang +2 more · 2026
ForceHarness enables API-only, frozen frontier multimodal models to perform contact-rich robotic manipulation without demonstrations or fine-tuning. It introduces a three-layer architecture comprising force sensing and calibration, self-terminating closed-loop force primitives, and a physics translation layer that converts force traces into interpretable physical quantities. Across real-robot hardware and simulation benchmarks including HiddenPhys and ForceBench, the system dramatically improves zero-shot manipulation success and matches expert force controllers.
Related Ideas
- Feature-Parity Verification and Latency-Accuracy Co-Evaluation for Served Frontier VLAsScaleVLA: Scaling Vision-Language-Action Models to 552 Billion Parameters🟡Fairly easy
- Dual-Rate Hierarchical Control with Closed-Loop Residual Correction for Contact-Rich Bimanual ManipulationScaleVLA: Scaling Vision-Language-Action Models to 552 Billion Parameters🔴Hard
- Adaptive Token Sparsity and Early Exiting for Real-Time VLM InferencePenguin-VL: Exploring the Efficiency Limits of VLM with LLM-based Vision Encoders🔴Hard
- Token-Normalized and Convergence-Controlled Evaluation of VLM Scaling Laws in RoboticsScaleVLA: Scaling Vision-Language-Action Models to 552 Billion Parameters🟡Fairly easy
- Parameter-Efficient Action Adaptation and Multi-Layer Feature Readout for Frontier VLMsScaleVLA: Scaling Vision-Language-Action Models to 552 Billion Parameters🔴Hard