VLA

See also: Terms and artificial intelligence terms

A Vision-Language-Action model (VLA) is a type of foundation model that enables robot control through the integration of visual perception, natural language processing, and action generation capabilities. VLAs represent a significant advancement in embodied AI, allowing robots to understand and execute complex tasks based on visual inputs and natural language commands while directly outputting motor control actions. These models are transforming robotics by reducing the need for complex, task-specific programming, enabling robots to learn from visual and language data much like humans do.

Overview

VLA models are constructed by extending vision-language models (VLMs) with the capability to generate robot action sequences. Unlike traditional robotic control systems that require separate modules for perception, planning, and control, VLAs provide an end-to-end solution that directly maps visual observations and language instructions to robot actions.

The key innovation of VLA models is their ability to leverage pre-trained knowledge from large-scale internet data while adapting to robotic control tasks. This approach addresses one of the fundamental challenges in robotics: the need for extensive task-specific training data for each robot, environment, and application. Actions are represented in the same token space as text so that gradient updates align linguistic and kinematic concepts, allowing the model to inherit world knowledge from internet-scale corpora while grounding it in robot trajectories.

How It Works

A VLA model typically consists of three main components that work together in a unified, end-to-end system:

Vision Encoder

This component processes visual input from cameras and other sensors, converting raw pixel data into a meaningful representation of the environment. It identifies objects, understands their spatial relationships, and extracts other relevant visual features.

Language Model

The language model interprets natural language commands given to the robot in the form of text or speech. It understands the user's intent and extracts the goal of the task, as well as any constraints or preferences.

Action Policy

This component takes the outputs from the vision encoder and the language model and generates a sequence of actions for the robot to execute. It plans the robot's movements, controls its actuators, and ensures that the task is performed safely and efficiently.

This integrated approach is a key advantage of VLAs over traditional robotic systems, where perception, planning, and control are often treated as separate and independent modules.

History

The concept of Vision-Language-Action models emerged from the convergence of advances in large language models (LLMs), computer vision, and robotic learning. The term "VLA" was first coined in the Google RT-2 paper, which described the model as transforming "Pixels to Actions".

Early Development

The foundation for VLA models was laid by earlier work in robotic learning, including:

• RT-1 (Robotics Transformer 1): Introduced in late 2022, RT-1 was the first "large robot model" trained with robot demonstration data collected from 13 robots over 17 months. It demonstrated the potential of transformer architecture for robotic control but did not yet integrate vision-language pre-training.

• PaLM-E: An embodied multimodal language model that enhanced robots' ability to understand their surroundings through vision models.

RT-2: The First True VLA

In July 2023, Google DeepMind announced Robotic Transformer 2 (RT-2), described as "a first-of-its-kind vision-language-action (VLA) model". RT-2 represented a breakthrough in enabling robots to:

• Apply knowledge and reasoning from large web datasets to real-world robotic tasks
• Understand and execute commands not present in robot training data
• Perform rudimentary reasoning in response to user commands
• Use chain-of-thought reasoning for multi-stage semantic reasoning

Architecture

VLA models typically consist of three main components:

Vision Module

The vision module processes raw image inputs into perceptual tokens. Common architectures include:

• Vision Transformers (ViT) such as SigLIP and DINOv2
• Fused visual encoders that combine multiple pre-trained vision models
• Resolution standardization (typically 224×224 pixels for models like PaliGemma)

Language Module

The language module integrates visual information with natural language instructions and performs cognitive reasoning. This typically involves:

• Pre-trained large language models such as Llama 2, PaLM, or Qwen
• Token-based processing of both visual and linguistic inputs
• Cross-attention mechanisms between vision and language modalities

Action Module

The action module generates robot control commands. Two main approaches have emerged:

Discrete Action Tokenization

Early VLA models like RT-2 represent robot actions as discrete tokens within the language model's vocabulary. Actions are encoded as text strings (for example "1 128 91 241 5 101 127 217") that map to specific motor commands.

Continuous Action Generation

More recent models employ continuous action generation through:

• Flow Matching: Used in models like π0, enabling direct prediction of continuous action sequences at high frequencies (up to 50Hz)
• Diffusion Models: Applied in models like CogACT for generating smooth, multi-modal action distributions
• FAST Tokenization: Frequency-space Action Sequence Tokenization using Discrete Cosine Transform for efficient autoregressive generation

Training

Data Requirements

VLA models are typically trained on two types of data:

1. Internet-scale vision-language data: Provides semantic understanding and reasoning capabilities
2. Robot demonstration data: Teaches specific motor control and manipulation skills

The Open X-Embodiment dataset, released in October 2023, represents the largest collection of robot learning data, containing:

• Over 1 million real robot trajectories
• 22 different robot embodiments
• 527 skills across 160,266 tasks
• Data from 21 institutions worldwide

Training Methodology

VLA training typically involves:

1. Pre-training: Starting from a pre-trained VLM to inherit semantic knowledge
2. Co-fine-tuning: Training on both vision-language tasks and robot control data simultaneously
3. Action representation learning: Teaching the model to output appropriate action tokens or continuous values
4. Multi-task learning: Training on diverse tasks to improve generalization

Notable Models

RT-2 (Robotic Transformer 2)

Developed by Google DeepMind, RT-2 was the first model to be explicitly called a VLA. It comes in two variants:

• A 12B parameter version based on PaLM-E
• A 55B parameter version based on PaLI-X

Key capabilities include:

• Understanding of spatial relationships and object properties
• Chain-of-thought reasoning for complex tasks
• Generalization to novel objects and instructions

In over 6,000 physical trials, RT-2 doubled success on unseen tasks (62%) versus RT-1 (32%).

OpenVLA

Released in 2024, OpenVLA is a 7B-parameter open-source VLA that democratized access to VLA technology. Key features include:

• Built on Llama 2 7B with fused SigLIP and DinoV2 vision encoders
• Trained on 970k robot manipulation trajectories
• Support for LoRA fine-tuning for efficient adaptation
• 16.5% better performance than RT-2-X (55B) despite having 7x fewer parameters
• Can be fine-tuned on a single GPU

π0 (Pi-Zero)

Developed by Physical Intelligence, π0 introduced several innovations:

• Full upper-body humanoid control including individual finger movements
• Multi-robot collaboration capabilities
• Flow matching for high-frequency (50Hz) continuous control
• Training on data from 7 robotic platforms and 68 unique tasks

π0.5

An extension of π0 that demonstrates "open-world generalization":

• Can operate in entirely new environments not seen during training
• Performs long-horizon tasks like cleaning kitchens or bedrooms
• Uses hierarchical reasoning with high-level language planning and low-level motor control
• Co-trained on heterogeneous data sources for improved transfer learning

Gemini Robotics

Google DeepMind's Gemini Robotics on-device VLA executes dexterous tasks without cloud connectivity, critical for privacy-sensitive sites and edge deployment.

Other Notable Models

Model	Developer	Key Features	Parameters
Helix	Figure AI	First VLA for full humanoid upper-body control, multi-robot collaboration, deployed in BMW factories, runs entirely on embedded GPUs	7B (System 2), 80M (System 1)
VLAS	Various researchers	First end-to-end model integrating speech modality for robot manipulation	Not specified
CogACT	Various researchers	Uses Diffusion Transformer for action decoding, strong performance on SIMPLER benchmark	~7.3B total
SmolVLA	Hugging Face	Lightweight model optimized for CPU inference and consumer hardware, asynchronous inference stack	450M
MiniVLA	Stanford AI Lab	7x smaller than OpenVLA with improved inference speed	~1B

Applications

VLA models have demonstrated capabilities across a wide range of robotic manipulation tasks:

Basic Manipulation

• Pick and place operations
• Opening and closing drawers, doors, and containers
• Grasping objects of various shapes and materials
• Tool use and manipulation

Complex Tasks

• Folding laundry from a hamper
• Assembling cardboard boxes
• Table bussing and kitchen cleaning
• Multi-step meal preparation
• Collaborative tasks between multiple robots

Industry Applications

• Manufacturing: VLA-powered robots automate assembly, packaging, and quality control tasks. Figure AI's Helix has been deployed in BMW factories for industrial automation.
• Logistics and Warehousing: Zero-shot grasping of unseen SKUs, sorting packages, and loading/unloading operations. Fine-tuning OpenVLA with few-shot LoRA achieves reliable warehouse picking.
• Healthcare: Assisting with delivering supplies, patient care support, and simple medical procedures, freeing healthcare professionals for more complex tasks.
• Home and Personal Assistance: Cooking, cleaning, laundry tasks, and providing companionship and assistance to the elderly and people with disabilities.
• Disaster Response: Search and rescue operations, damage assessment, and emergency supply delivery in hazardous environments.

Emerging Applications

Evidence suggests VLAs are becoming key in fields like:

• Autonomous vehicles for complex navigation and interaction
• Precision agriculture for crop monitoring and harvesting
• Augmented reality navigation systems
• Medical robotics for surgical assistance

Emergent Capabilities

VLA models have shown emergent behaviors not explicitly programmed:

• Understanding spatial relationships (for example "pick up the object to the left of the red block")
• Semantic reasoning (for example identifying which object could serve as an improvised hammer)
• Adapting to novel objects and environments
• Following abstract instructions (for example "clean the kitchen")

Technical Challenges

Action Space Representation

One of the key challenges in VLA development is representing the continuous, high-dimensional action space of robots in a way that can be effectively learned and generated by neural networks. Solutions include:

• Discretization into action tokens
• Flow matching for continuous generation
• Hierarchical action representations
• Action chunking for temporal coherence

Real-Time Performance

Achieving real-time control (typically 10-50Hz) while running large neural networks remains challenging. Approaches to address this include:

• Model quantization and optimization
• Efficient inference architectures
• Hardware acceleration
• Asynchronous inference pipelines

Generalization

While VLAs show improved generalization compared to traditional approaches, challenges remain:

• Sim-to-real transfer
• Cross-embodiment transfer between different robot types
• Long-horizon task planning
• Robustness to environmental variations

Safety and Reliability

Ensuring the safety and reliability of VLA-powered robots is critical, especially in human-centric environments. Robots must operate safely and predictably, even in the presence of unexpected events and disturbances. Current access to many VLA models remains gated to mitigate potential misuse.

Data Efficiency

Training VLA models requires large amounts of data, which can be expensive and time-consuming to collect. Improving the data efficiency of VLA models through better architectures and training techniques is an active area of research.

Ethical Considerations

The development and deployment of VLA-powered robots raise several ethical concerns that need careful consideration:

• Job displacement in manufacturing and service sectors
• Privacy concerns with robots operating in personal spaces
• Bias in decision-making inherited from training data
• Ensuring equitable access to beneficial robotic technologies

Future Directions

Research Areas

Active areas of VLA research include:

• Multi-modal integration: Incorporating tactile, force, and proprioceptive feedback beyond vision and language
• Self-improvement: Enabling robots to learn from their own experience through reinforcement learning
• Long-horizon planning: Improving capabilities for complex, multi-step tasks requiring extended reasoning
• Safety and robustness: Ensuring reliable operation in human environments with formal verification methods
• Efficiency: Reducing computational requirements for deployment on edge devices
• Agentic AI adaptation: Enabling VLAs to adapt more independently to new tasks without extensive fine-tuning

Industry Adoption

VLA models are beginning to see widespread adoption in:

• Manufacturing and assembly lines
• Warehouse automation and logistics
• Service robotics in hospitality and retail
• Healthcare applications and elderly care
• Home assistance and personal robotics
• Agricultural automation

See Also

• Vision-language model
• Embodied artificial intelligence
• Robot learning
• Transformer (machine learning model)
• Foundation model
• Open X-Embodiment
• Multimodal learning
• SayCan
• PaLM-E
• Large action model

External Links

• Open X-Embodiment Dataset Official Website
• OpenVLA Project Page
• Physical Intelligence Company Website
• OpenVLA GitHub Repository
• Physical Intelligence OpenPi Repository
• OpenVLA Model Collection on Hugging Face