EgoGaming Data › Robotics and embodied AI
For Robotics and Embodied AI
Human Head and Hand Motion for Robotics, Recorded in VR
EgoGaming records people doing tasks in VR and delivers their measured head pose and the 6DoF grip pose of each hand controller for every frame, synchronized with what they saw. For robotics teams it is human motion data: two-handed trajectories with a matching first-person view, in RLDS episodes.
What Is Useful for Robot Learning
- Measured trajectories: 6DoF grip poses for both hands and the head, from the headset’s own tracking at the game’s frame rate, with flags where tracking was lost.
- Per-step actions: the RLDS
actionis each device’s change in position and rotation to the next step. - A view to condition on: the first-person image at every step, so a policy sees what the person saw.
- A familiar format: RLDS, the episode format used by Open X-Embodiment, loads with TensorFlow Datasets.
Recorded to Your Tasks
For robotics we recommend custom recording: you describe the tasks (reaching, picking up, placing, using a tool), the number of hours and the player mix, and we record them in a VR game we are cleared to record or in your own VR environment. Custom recordings are exclusive to you for 24 months, and you pay only for hours that pass review. How licensing works.
What This Data Is Not
- Not real-world video. The picture is a rendered game, so it does not replace camera footage of real objects.
- Not finger-level hand data, mostly. The hands are controller grip poses. Articulated hand joints exist only where the game used hand tracking, which is rare today.
- No forces or contact. VR controllers do not measure grip force or touch.
- Not robot data. It is human motion; retargeting it to your robot is your step.
Judge the Trajectories Yourself
The free sample has 100 action clips in RLDS and WebDataset. Pick “Robotics” in the form and describe your tasks.