Language AI got its food for free.
The internet gave language AI a free corpus. Physical AI and Robots in particular have to earn its data, one interaction at a time.
Every hour of useful data a robot learns from has to be deliberately created, and the highest quality data is captured in the real world. People have to perform tasks. Cameras and sensors have to capture what happens. That data has to be structured, checked, and often translated from a human body into a robot with very different physical constraints.
Somebody has to cook the robot food.
And it turns out that building the right menu for Physical AI is more complicated than simply collecting more video.
A robot may need to understand what a task looks like from a person’s point of view, what the rest of the body is doing outside that field of view, how quickly joints are moving, how human motion translates to a different embodiment, and whether an otherwise correct action is appropriate in a particular situation.
Those are different information problems. They require different kinds of data.
So what, exactly, do robots need to eat?
A physical task contains more information than a video can show
Consider an ordinary task such as putting groceries away.
A person recognizes an object, reaches for it, adjusts their grip, turns their body, opens a cabinet or refrigerator, moves the object into position, releases it, and moves on to the next item.
That simple sequence contains multiple forms of information at once.
There is what the person sees. There is what their hands are doing. There is the movement of the rest of the body. There are changes in speed, orientation, and joint position. There is the relationship between the person and the surrounding environment. And there is context that determines whether an action makes sense in the first place.
No single data source captures all of those signals equally well.
This is why a Physical AI data strategy should start with the information gap rather than the data type.
Does the model need a better view of the task? Better understanding of whole-body motion? More precise kinematics? A way to translate human movement into a particular robot body? Demonstrations recorded directly on that robot? Better evaluation of whether an otherwise successful action is appropriate in context?
Those are different problems, and they call for different forms of data.
First-person data teaches the task from inside the body
Egocentric data records a task from the actor’s own point of view.
Most internet videos were created for human viewers, not for robot training. It is typically filmed from the outside. But a robot learning to manipulate the physical world often needs to see a task from the perspective of the body performing it.
That footage has to be made.
A person wears camera glasses, a head-mounted camera, or another first-person capture device while washing dishes, folding laundry, putting groceries away, or performing whatever task the system needs to learn.
Now the model can observe where the hands go, which objects enter the field of view, where attention is directed, and the sequence in which the task unfolds.
And the environment matters just as much as the task.
Different kitchens. Different lighting. Different objects. Different hands. Different camera positions. Left-handed and right-handed people.
A narrowly collected dataset can teach a correspondingly narrow version of the world.
Put more simply: a robot trained only on right-handed demonstrations risks learning a very right-handed version of the task.
Diversity is not something to sprinkle onto a Physical AI dataset later. It has to be designed into the collection itself.
Third-person data shows what the first-person view cannot
You cannot see your own legs while you are walking.
That simple observation explains why first-person video is not enough.
Egocentric data shows what the actor agent sees. Exocentric data records the same action from outside the body, revealing posture, balance, leg movement, torso orientation, and other aspects of whole-body motion that may never enter the first-person camera’s field of view.
The more useful question is therefore not whether a robotics team should collect first-person or third-person data.
It is what happens when they have both.
When egocentric and exocentric streams capture the same task on the same clock, the model can begin connecting two sides of the physical experience:
Here is what I saw. Here is what my body did.
For embodied learning, that relationship between perception and action can be more valuable than simply accumulating more footage from one perspective.
Source: Helix AI https://www.figure.ai/news/helix
Annotation turns video into structured motion data
Recording the task is still only the beginning.
Raw video can be enriched with keypoints, segmentation masks, bounding boxes, labels, ontologies, and other annotations that make important parts of the scene explicit for training and evaluation.
Pose estimation makes the value especially easy to see.
Run a pose model across video and the person is no longer just a collection of pixels. The system can track skeletal joint positions frame by frame and build a structured representation of how the body moved, including estimates of joints that may temporarily disappear behind another part of the body.
A video tells you that movement happened.
A pose representation starts telling you how the body moved.
That creates a bridge from ordinary visual demonstrations into motion data that can be analyzed, compared, quality-checked, and used later in the robot-learning pipeline.
Pixels do not tell you everything about movement
Video is still an indirect measurement of motion.
This is where inertial measurement units, or IMUs, become useful.
An IMU is the kind of sensor technology that can tell when the device has moved or rotated. It measures acceleration and angular velocity, and some systems add a magnetometer to provide heading information.
Place multiple trackers on the body and you can record how different parts of the body actually move as a person performs a task.
That provides a different kind of signal than video.
Video tells you what a movement looked like.
Inertial data tells you more about how that movement unfolded: how quickly the body accelerated, how it rotated, and how different segments coordinated through the action.
That distinction matters for robotics because executing a movement is not simply a visual problem. A robot eventually has to reproduce motion through its own joints and actuators, at appropriate speeds and within its own physical limits.
You do not get all of that from pixels alone.
Wearable motion data, therefore, does not replace video. It fills in information that cameras are less suited to measuring directly.
Motion capture establishes the reference
Sometimes approximate motion is enough.
Sometimes it is not.
For higher-fidelity movement data, optical motion capture (MoCap) provides another level of precision.
This is essentially Hollywood technology applied to Physical AI. A performer wears a suit covered in reflective markers while an array of infrared cameras tracks those markers through three-dimensional space.
The result is a highly precise representation of the body’s movement over time.
That matters because video pose estimation and wearable sensing both involve their own sources of estimation and error. High-fidelity optical motion capture can provide a ground-truth reference against which those other representations can be compared and evaluated.
The practical question for robotics teams is not whether every training program needs motion capture.
It is where approximation is acceptable and where small errors begin to propagate through the learning pipeline.
When the answer requires the highest-fidelity representation of the original human motion, ground truth matters.
Retargeting is where one body becomes another
Capturing human movement accurately does not mean a robot can reproduce it.
A humanoid may look vaguely human, but it has different proportions, different joints, different ranges of motion, different actuator limits, and different balance constraints.
That creates one of the central translation problems in robot learning: retargeting.
Retargeting means taking motion captured from one body and mapping it onto another.
Record a person completing a task, then transfer that movement onto a robot whose body may be dramatically different from the person who demonstrated it.
It sounds straightforward until the geometry meets physics.
A pose can look correct mathematically and still be impossible for the robot to execute. Maybe the robot’s knee does not bend that far. Maybe the trajectory would push a joint outside its operating range. Maybe the movement would destabilize the robot. Maybe the required dynamics do not translate cleanly to the target body.
The goal is not to tell the robot, “Copy this human exactly.”
It is to preserve the useful information in the demonstration while expressing it through a body with different capabilities and constraints.
And that is where a demonstration pipeline can quietly lose fidelity.
A team therefore has to ask two separate questions: How accurately did we capture the original human behavior? And how much of that behavior survived the transfer to the target robot?
Teleoperation removes the retargeting gap
There is another way to approach the problem: collect the demonstration directly through the robot.
In teleoperation, a human remotely drives a real robot through a task using controllers, VR, a puppet rig, or another interface.
While the operator performs the task, the system can log what the robot is doing: control commands, joint states, camera frames, and other signals generated during execution.
That changes the nature of the data.
With a separately recorded human demonstration, the motion eventually has to cross the retargeting gap from one body into another.
With teleoperation, the demonstration is born in the robot’s body.
The resulting data already reflects that robot’s coordinates and physical configuration.
That does not make large-scale human demonstrations, wearable motion, or motion capture unnecessary. Each solves a different information problem.
Instead, teleoperation gives robotics teams another lever when they need demonstrations that are native to a specific embodiment.
Source: https://github.com/unitreerobotics/xr_teleoperate
The same action can be safe in one context and dangerous in another
Being physically capable of performing a task is not the same as understanding when to perform it.
Picture a robot opening a jar of pickles.
Harmless.
Now picture it making the exact same twisting motion on a bottle of pills, with a toddler nearby.
Same basic task. Same motion of the lid.
Completely different stakes.
That is the challenge of trust and safety in embodied AI.
Evaluation cannot stop at asking, “Did the robot successfully open the container?”
It also has to ask what the object is, who is nearby, what the surrounding circumstances are, what could happen next, and whether the action is appropriate in that context.
The physical world creates an enormous surface area for scenarios that cannot be captured by simple task-success metrics.
Human behavior is unpredictable. Environments change. A behavior that was safe in one scenario may not be safe in another. And as systems continue operating after deployment, assurance increasingly becomes a continuous problem rather than a one-time launch check.
Before robots operate broadly in homes and workplaces, they will need to be tested against scenarios designed to expose exactly these kinds of contextual failures.
The point is not to order everything on the menu
Egocentric video, Exocentric video,. Annotation,. Wearable motion,. Optical motion capture, Teleoperation – tThere is no reason every Physical AI program should use every modality.
The right data strategy starts with the information the system is missing.
If the model does not understand the task from the agent’s perspective, collect first-person demonstrations.
If the missing information is whole-body movement, add synchronized external views.
If pixels are not capturing the required motion dynamics, add wearable sensors.
If estimated movement needs a higher-fidelity reference, use optical motion capture.
If valuable information is being lost between the human demonstration and the target robot, examine the retargeting pipeline.
If demonstrations need to reflect a particular robot’s native embodiment, teleoperation may make more sense.
And if the robot can execute the physical task but cannot reliably determine when that action is appropriate, the problem has moved into contextual evaluation and safety.
These data types are complementary, not interchangeable.
More video does not automatically solve a missing kinematics problem.
More precise human motion does not eliminate an embodiment mismatch.
And more successful task demonstrations do not answer every safety question.
The right question is not, “How much data can we collect?” It is, “What does the robot still need to learn?”
Building the Physical AI data pipeline
For robotics teams moving from research toward production, answering that question creates an operational challenge of its own.
Human contributors, capture environments, synchronized cameras, annotation, quality control, pose estimation, wearable motion, high-fidelity reference data, retargeting, robot-native demonstrations, and safety evaluation all have to connect into a usable training pipeline.
That is where Innodata is applying its experience in large-scale data collection, enrichment, and evaluation to Physical AI.
The objective is not simply to make more robot data.
It is to identify the training or evaluation signal a system is missing, then create the data required to close that gap.
Because robots do not get their training data for free.
Somebody still has to make the food.