Skip to content
AI-generated research idea🟡Fairly easy🔧Minor TweakNo close match found

Token-Normalized and Convergence-Controlled Evaluation of VLM Scaling Laws in Robotics

Isolate visual token count, convergence training budgets, and subset-only normalization to measure the true contribution of backbone parameter scale in robotic manipulation policies.

Time

several weeks

Compute level

Large cluster

Resources

Multi-GPU to cluster H200 resources (4 H200 GPUs for 552B feature extraction across token budgets, plus multi-GPU nodes for 3-seed converged action-head training on LIBERO and RoboCasa).

What the paper used

Trained on 64 NVIDIA H200 GPUs (8 nodes x 8 GPUs, batch size 640, 5k-96k steps); served for timing on 4 H200 GPUs for 552B ScaleVLA and 1 H200 GPU for 2B-32B backbones across LIBERO, RoboCasa, DexJoCo, and 303 real-robot episodes (0.52M frames).

Prerequisites

Experimental methodology, VLA training pipelines, tokenization/patch resolution handling in VLMs, and statistical evaluation.

Scope

Modify preprocessing, tokenization, normalization, and training budget scripts for existing benchmarks.

The Gap

In the latency and sample-efficiency studies (Sections 4.3, 4.4, and Appendices A, C, and D.1), ScaleVLA receives 568 input tokens per RoboCasa observation compared to 212 tokens for all 2B–32B baselines, all policies use one training seed, demonstration sweeps retain full-dataset 1st/99th percentile action normalization statistics, and fixed-budget comparisons at 5,000 steps under-train smaller 2B models relative to 40,000 steps.

“ScaleVLA also receives more input tokens per observation. We did not isolate how much this contributes to latency.”
From the paper, Section 4.3, page 7. Matched against the PDF text.

Limitation

Higher input token resolution (568 vs. 212 tokens), single-seed training, full-dataset normalization leakage in low-data sweeps, and unconverged fixed-step budgets for smaller models confound how much of ScaleVLA's accuracy and latency differences stem purely from parameter scaling.

Proposed Solution

Design a strictly controlled scaling benchmark that equalizes visual patch/token budgets across 2B, 8B, 32B, and 552B backbones via matched image resolution or token pooling, computes state-action normalization strictly from each N-demonstration subset, trains every backbone to validation convergence across at least three random seeds, and compares both fixed-step and converged performance.

Why This Matters

Resolving these confounds will establish clean, reproducible scaling laws separating representational scale from visual token budget and optimization effects, guiding future VLA compute allocation.

How to Get Started

How to test it: Test the hypothesis that ScaleVLA retains a statistically significant (>5%) success advantage over Cosmos-Reason2-32B and Qwen3-VL-32B on LIBERO-PRO and RoboCasa Gen-100 when input token count is matched at ~212 tokens, models are trained to convergence across 3 seeds, and normalization uses subset-only percentiles; stop if token-matched 552B drops to parity with 32B.

  1. Inspect the vision preprocessing pipelines of DeepSeek-V4.1-Flash-1017, Qwen3-VL, and Cosmos-Reason2 to align image resolution and patch counts to produce matched ~212 and ~568 token budgets.
  2. Modify the LIBERO demonstration-sweep preprocessing script so 1st and 99th percentile state/action statistics are computed strictly inside each N in {5, 10, 25, 50} subset.
  3. Train Cosmos-Reason2-2B, Cosmos-Reason2-32B, and 552B action heads on LIBERO Goal/Long until validation loss plateaus (up to 40k steps) across 3 random seeds.
  4. Measure both task success rates and vLLM/H200 inference latencies under token-matched conditions to decompose parameter-count vs. token-count effects.

Suggested by AI. Adapt it to your setup.

Has This Been Done?

No close match in 10 results

None of the provided literature establishes a controlled token-budget scaling benchmark paired with strict sample-subset normalization and convergence-controlled training across diverse VLA backbone sizes. While papers like How Many Demonstrations Does a VLA Need? examine sample efficiency, they do not address the proposed multi-seed scaling law protocol.

We searched OpenAlex and arXiv for “VLA scaling laws visual token count sample efficiency convergence ablation” and “Token-Normalized and Convergence-Controlled Evaluation of VLM Scaling Laws in Robotics” and an AI compared the idea with the top 10 results. A small search can miss relevant work.

Source Paper

ScaleVLA: Scaling Vision-Language-Action Models to 552 Billion Parameters

Ruixi Yu, Dairu Liu, Xuchuan Chen +9 more · 2026

This paper investigates whether scaling the vision-language model (VLM) backbone of a vision-language-action (VLA) policy to frontier scale (552B parameters) improves robotic manipulation. By connecting a frozen 552B sparse mixture-of-experts VLM (DeepSeek-V4.1-Flash-1017) to a trainable flow-matching diffusion transformer action head, the authors demonstrate substantial gains in sample efficiency, out-of-distribution generalization, and contact-rich real-world insertion while maintaining subsecond per-chunk latency.

Related Ideas

Is this a good idea?

Share onXLinkedIn