The EgoGaming VR gameplay dataset is first-person video from commercial PC VR games, recorded from paid adult players who consented in writing. Every video frame comes with the measured 6DoF pose of the headset and both controllers, taken at the same instant on one capture clock, plus velocities, inputs and tracking flags.

Our recorder sits between the game and the headset, so it captures what the game itself sees and sends: the rendered view and the tracking the game asked for. Nothing is estimated from the video afterwards. Watch a recording with its poses.

At a Glance

Source	Adults playing commercial PC VR games they own, more than 15 games so far
Video	The left-eye view the player saw, at the game’s own frame rate (about 70–90 fps)
Motion	6DoF head pose and left and right controller grip poses for every video frame
Also per frame	Linear and angular velocity per device; tracking-quality flags
Inputs and events	Button, trigger and stick input where the capture route records it; tracking loss, recenters and state changes
Hand joints	Only in the sessions where the headset tracked hands
Headsets	PC VR, mostly Meta Quest 3S linked to a PC, plus Quest 2 and Valve Index
Review	Automatic checks and a human review of every session
Formats	RLDS (TFDS), WebDataset, or Parquet, CSV or JSON tables with MP4 video
Documentation	Data card, license terms, session list, column guide and checksums in every delivery
Access	Commercial or academic license; free sample of 100 action clips

Video

The video is the game’s own rendered left-eye view, exactly what the player saw, not a screen recording or a mirror window. It keeps the game’s frame rate, so a game running at 72 or 90 Hz gives 72 or 90 frames a second. Deliveries are H.264 MP4, 720 pixels high in the WebDataset format and 320×180 JPEG steps in RLDS. Video has no audio by default, because game audio can carry other players’ voices.

Head and Controller Poses

Each video frame has the position and orientation of the headset and both controllers at that instant. Positions are metres in the game’s stage space, with y up and the floor at y = 0. Orientations are unit quaternions (x, y, z, w). Controller poses are the grip pose.

All streams share one clock, t_capture_ns, and frame_index joins rows of the same frame. In the WebDataset format, video frame i of a clip is row i of its pose, velocity and flag arrays. A recenter moves the origin; each one is logged as an event.

Tracking Quality Flags

Every pose row carries pos_valid, ori_valid, pos_tracked and ori_tracked for each device. Valid but not tracked means the runtime was extrapolating from the IMU with no camera fix: those samples look plausible and are wrong, so exclude them from velocity and displacement work. A controller with no row at all was not being reported, usually because it was put down. The flags are always delivered.

Inputs, Events and Hand Joints

Inputs: controller button, trigger and stick input, where the capture route records it.
Events: tracking loss, recenters and state changes, with their times.
Hand joints: 26 articulated joints per hand, only in sessions where the headset tracked hands. Most games use controllers, so there is little of this today; elsewhere the hands are the controller poses.

How It Is Captured

Players install the EgoGaming recorder on a Windows PC. For most games it runs as an OpenXR API layer; for older SteamVR titles that do not use OpenXR, it uses an OpenVR capture route. Either way it records what the game asks the headset for, in the game’s own stage space. Headsets connect to the PC over Link, Air Link, Virtual Desktop or SteamVR. Standalone headset games are not covered.

How It Is Reviewed

Every session goes through automatic checks (complete files, tracking quality, dropped frames) and is watched by a person. Reviewers mark idle time, menus and headset-off moments, and reject sessions that are idle, in the wrong game or by the wrong person. Sessions under a minute are never sold. Every review decision is made by a person.

Formats

Format	Best for	What you get
RLDS (TFDS)	World-model and robotics pipelines	One episode per clip (up to 60 s), steps at 15 Hz: 320×180 image, 21-float state (head, left, right as px, py, pz, qx, qy, qz, qw), velocity, flags, hand joints, an 18-float per-device `action`, and episode metadata
WebDataset	Large training runs that stream from storage	TAR shards, one sample per clip at the full recorded rate: MP4, frame, pose, velocity and flag arrays, plus poses, hands and inputs Parquet and an events file
Plain files	Analysis and research	Parquet, CSV or JSON tables per session with MP4 video and a frame index
LeRobot, HDF5	-	Planned. Tell us if you need one

The exact layout (clip length, picture size) is agreed for each order. See the field lists and a loading example.

Who the Players Are

Players are adults, 18 or older, mostly in the United States. They sign up at egogaming.ai, pass a review, sign a player contract and agree in writing, on the website and again in the recorder, that their recordings can be licensed. They are paid for every hour that passes our checks. We do not collect age, gender or other demographics today; for a custom order we can recruit to a target player mix.

Known Limits

The video is a rendered game view, not the real world. Multiplayer games can show other players’ avatars and usernames.
Only the head and the two controllers are measured. Arms and bodies drawn in our demo are estimated between those points.
Most hours so far come from a few games. We share the current title list with buyers on request.
PC VR only; standalone headset games are not covered.
VR movement can identify people, so every file is personal data under the license, even though it is pseudonymized.

See It in Your Own Pipeline

The free sample has 100 action clips in RLDS and WebDataset, with the data card.

Request a Free Sample How Licensing Works

The EgoGaming VR Gameplay Dataset: What’s in Every Recording