A robot looks at a mug sitting on a table.
Someone tells it, “Pick up the mug and put it in the sink.”
Recognizing the mug is one challenge. Understanding the instruction is another. Connecting both pieces of information to the correct physical behavior is the problem vision-language-action models are designed to solve.
Vision-language-action, or VLA, models connect visual perception, language understanding and physical actions. For robotics and embodied AI systems, that makes high-quality real-world training data especially important.
But collecting VLA data involves much more than recording people completing tasks.
Useful datasets need to preserve the relationships between what the system sees, what it is instructed to do, the actions that follow and the resulting changes in the physical environment.
So what does effective real-world VLA training data look like, and how should teams approach collecting it?
What Is a Vision-Language-Action Model?
A vision-language-action model combines visual understanding and language comprehension with the ability to predict or generate physical actions.
At a high level, a VLA system may need to:
- See and interpret its surroundings
- Understand a task or instruction
- Decide what should happen next
- Act through a robot or another embodied system
- Adjust as the environment changes
VLA models have become an important area of robotics research because they offer a path toward systems capable of handling broader sets of tasks instead of relying entirely on narrowly programmed behaviors.
Google DeepMind’s RT-2 helped establish the modern VLA concept by adapting vision-language models to predict robot actions. The system combined large-scale vision-language knowledge with robotic demonstration data, allowing visual and linguistic understanding to contribute directly to robot control.
More recent systems have continued building on this approach across different robot types and manipulation scenarios. Google DeepMind’s Gemini Robotics On-Device 2, announced in July 2026, is another example of a VLA system intended for general-purpose manipulation across multiple tasks, environments and robotic embodiments.
The core data challenge remains consistent: models need examples that connect perception and language with actions in the physical world.
Why VLA Models Need Real-World Training Data
Large language models can learn from enormous quantities of text.
Vision models can learn from massive collections of images and video.
Robotics introduces another dimension: understanding how actions physically change an environment.
Consider the instruction:
“Put the red block inside the box.”
The system needs to determine:
- Which object is the red block
- Which object is the box
- Where both objects are located
- How to move toward the block
- How to grasp it
- How far it needs to move
- How to transport it without dropping it
- Where the block should be placed
- When the task has successfully been completed
And the scene does not remain static.
Once the block has been picked up, the environment looks different from the original observation. The model must continue interpreting the changing scene and deciding what action should happen next.
Real-world datasets can capture these continuous relationships between perception, actions and changing physical conditions.
What Types of Data Do VLA Models Need?
Exact data requirements vary based on the robot, task and model architecture. However, VLA training programs often rely on several connected types of information.
Visual Observations
The system needs to understand the environment before, during and after an action takes place.
Visual data may include:
- Objects
- Hands
- Tools
- Furniture
- People
- Obstacles
- Work surfaces
- Environmental layouts
- Changes in object position
- Changes in object state
Video is particularly valuable because physical interactions happen over time.
For example, footage showing someone opening a drawer contains much more information about the action itself than a single image showing the drawer already open.
Language Instructions
Language establishes the desired task or goal.
Instructions may be simple and specific:
“Pick up the blue cup.”
Or they may require more interpretation:
“Clear everything off the table.”
“Put the groceries away.”
“Bring me something I can drink from.”
Multiple phrases can refer to the same underlying physical behavior.
That linguistic variation can be valuable when training systems that eventually need to interpret natural instructions from real users.
Physical Actions
Actions connect perception and instruction to what happens next in the environment.
Depending on the application, action data may include:
- Reaching
- Grasping
- Releasing
- Moving
- Rotating
- Pushing
- Pulling
- Opening
- Closing
- Walking
- Turning
- Tool use
- Multi-step manipulation
For robot-generated datasets, these actions may also be represented through information such as joint states, end-effector positions, gripper states or other system-specific control signals.
For human demonstrations, video and additional sensor streams can document how people perform comparable physical behaviors.
Task Outcomes
It can be equally important to record whether an action succeeded.
Did the object make it to the correct location?
Did the cabinet door actually open?
Was something dropped?
Was the complete task finished?
Outcome information gives additional structure to a demonstration dataset and can help identify failures and edge cases that may also be valuable for training.
How to Collect Real-World VLA Training Data
Real-world VLA data collection should begin with the behavior the AI system needs to learn.
Camera selection should come afterward.
1. Define the Tasks the System Needs to Perform
Begin by identifying the activities the model is ultimately expected to handle.
For a household robot, examples might include:
- Picking up objects
- Opening cabinets
- Loading a dishwasher
- Folding clothes
- Clearing a table
- Moving objects between rooms
- Using basic household tools
Industrial applications might require demonstrations involving assembly, machine components, containers, equipment or workplace materials.
Once the larger behaviors are defined, they can be broken into smaller actions and transitions.
2. Determine Which Variables Need to Change
The next step is identifying meaningful variation.
If the objective is teaching a system to pick up cups, recording 5,000 demonstrations involving the same cup on the same table may produce a large dataset without providing enough diversity.
Variables may include:
- Object size
- Shape
- Material
- Color
- Orientation
- Position
- Background
- Lighting
- Participant
- Room layout
- Camera perspective
- Starting position
The goal is to expose the model to the types of differences it may encounter when operating outside the collection environment.
3. Choose the Right Capture Perspective
Different camera perspectives reveal different aspects of an activity.
First-Person or Egocentric Capture
Wearable cameras show tasks from approximately the perspective of the person performing them.
This can be useful for:
- Hands-in-view interactions
- Object manipulation
- Tool use
- Navigation
- Sequential activities
First-person footage creates a particularly close relationship between the visual observation and the action that follows.
Third-Person Capture
External cameras provide broader environmental and body context.
They can capture:
- Full-body movement
- Relationships between people and objects
- Movement through a room
- Interactions that become hidden from a first-person camera
Multi-Angle Capture
Some VLA datasets can benefit from recording the same demonstration from several synchronized viewpoints.
A manipulation task might simultaneously include:
- First-person footage
- An overhead view
- A frontal camera
- One or more side views
Capturing multiple perspectives can provide a more complete representation of the activity while reducing the chance that important interactions are hidden from every camera.
4. Connect Physical Tasks With Language
Because language is fundamental to VLA systems, data collection should also account for how instructions are represented.
A participant might receive an instruction such as:
“Place the spoon inside the drawer.”
The same task could then be described in several other ways:
“Put the spoon away.”
“Move the utensil into the drawer.”
“Take the spoon from the counter and put it in the open drawer.”
The physical objective remains largely unchanged even though the wording differs.
Capturing those relationships can help systems learn that multiple language instructions may correspond to the same behavior.
5. Capture Different Valid Ways to Complete a Task
People rarely complete physical tasks in exactly the same manner.
One participant may open a cabinet with the right hand while another uses the left.
Someone may move an object out of the way before opening a drawer.
Another person may open the drawer first.
One participant might approach a counter straight on while someone else approaches from an angle.
These variations do not necessarily represent errors.
For systems intended to function in uncontrolled environments, they can be useful demonstrations of the fact that there may be several valid approaches to achieving the same goal.
A strong collection program can intentionally include those differences rather than forcing every demonstration into an identical sequence.
6. Include Longer, Multi-Step Activities
Individual actions matter, but many valuable robotic tasks consist of several connected steps.
Consider the instruction:
“Put the cereal away.”
A person may:
- Locate the cereal box
- Reach toward it
- Grasp the box
- Lift it from the surface
- Turn toward the cabinet
- Open the cabinet
- Move the cereal inside
- Place it on a shelf
- Release it
- Close the cabinet
Each step depends on the state created by the previous action.
Multi-step demonstrations preserve these longer relationships between intent, visual perception and physical action.
7. Collect Data in Different Real-World Environments
Physical-world AI needs environmental diversity.
One residential kitchen may look completely different from another.
Warehouses, offices, factories and other commercial environments introduce their own variations.
Collection programs may intentionally change:
- Furniture
- Room dimensions
- Object placement
- Lighting
- Flooring
- Background clutter
- Equipment
- Navigation paths
The objective is to give the model opportunities to learn the task itself rather than memorizing the appearance of one particular location.
8. Build Documentation Into the Collection Process
Large datasets quickly become difficult to use if teams cannot determine exactly what each file contains.
Depending on the project, metadata may document:
- Task
- Participant
- Environment
- Object set
- Camera setup
- Language instruction
- Attempt number
- Completion status
- Known issues
- Capture time
- Technical specifications
The specific schema will vary based on the downstream training pipeline.
What matters is defining it before large-scale collection begins rather than trying to reconstruct that information afterward.
VLA Training Data Should Include More Than Perfect Examples
It can be tempting to keep only demonstrations where everything goes exactly as planned.
Real environments rarely behave that way.
A person might reach toward the wrong item.
An object could slip.
Someone may need to readjust their grip.
A drawer could stick.
A target object might be partially obscured.
Depending on the training objective, these imperfect examples may be useful rather than disposable
