Bad data for robots? Training Vision-Language-Action (VLA) models—the neural networks that convert camera imagery and text instructions into physical robot actions—typically requires thousands of high-quality human demonstrations. Collecting pristine demonstration data at scale, however, is prohibitively expensive and time-consuming.
Most crowdsourced datasets are inherently noisy, filled with missed grabs, hesitant trajectories, and failed attempts.
A recent study presents a systematic evaluation of how to handle this suboptimal data during pretraining to maximize downstream task performance. To conduct these controlled experiments, the researchers introduced AXIS-Bench, a benchmark dataset containing 5,000 naturally flawed human simulation trajectories across 24 manipulation tasks.
Filtering vs. Weighting vs. Conditioning
When pretraining VLA models on noisy datasets, researchers generally rely on three main strategies:
- Filtering: Eliminating trajectories or segments that fall below a specific quality threshold.
- Weighting: Downweighting lower-quality trajectories in the loss function during training.
- Conditioning: Retaining all trajectories while attaching explicit quality labels, allowing the network to distinguish between high- and low-performing actions.
The study compared these approaches across multiple label granularities—evaluating quality at the full-episode level versus localized trajectory chunks—and across different reward definitions.
Key Empirical Findings
- Conditioning Outperforms Deletion: Plain behavioral cloning—simply imitating all provided data—underperforms relative to quality-conditioned pretraining. Rather than discarding flawed demonstrations, tagging full episodes with quality metadata yields higher mean task success. The model retains broad spatial context from low-quality attempts without adopting their failure modes.
- Granularity Matters: Episode-level labels prove most effective for conditioning strategies, whereas filtering yields better results when applied at the localized chunk level.
- Constant-Tag Fine-Tuning: During downstream fine-tuning, prepending a fixed, high-quality tag to the model input significantly increases success rates, even if quality tags are absent during inference.
- Real-World Transfer: Pretraining with quality conditioning in simulation translated directly to physical hardware, increasing real-world task success rates from 9.6% to 31.7%.
By leveraging structured quality tags rather than relying on perfect datasets, the framework established in AXIS-Bench offers a scalable path toward training generalist robotic models on uncurated, large-scale demonstration data.
Transparency Note: This article was prepared with the assistance of artificial intelligence technologies. Every piece is thoroughly reviewed and verified by our editorial team, who retain full responsibility for the final content.
Picture: Gemini AI
Original abstract:
how should demonstration quality be used during vision-language-action (VLA) pretraining to improve performance after target-task fine-tuning? We compare weighting, filtering, and conditioning across reward definitions and label granularities using 5,000 naturally suboptimal human demonstrations and 24 target tasks. Within each VLA, we match demonstrations and training budgets. Additional comparisons examine pretraining data size and the roles of quality tags during fine-tuning and inference. Across three VLAs, we find that approach-aware episode conditioning improves mean success over plain behavioral cloning; conditioning and filtering favor episode and chunk labels, respectively, on average, with model-dependent reversals; and constant-tag fine-tuning improves mean success after conditioned pretraining with either tagged or untagged inference. We introduce AXIS-Bench to support these controlled comparisons. Built on crowdsourced simulation demonstrations, it provides recorded object and contact states for recomputing quality labels on shared trajectories. Replayable trajectories support workspace-aligned rendering and user-defined real-world target tasks. In our real-world experiment, simulation pretraining raises success from 9.6% to 31.7%.
Project page:
https://axis-bench.github.io/.
On arXiv