Collecting robot training data with Rokoko Smartgloves

Robotics engineer Parsa Mesgarha from YAAK.ai built a teleoperation rig using Rokoko Smartgloves II and the Rokoko Coil Pro to control a robot arm by hand - collecting 221 demonstrations and training a policy that performed the task autonomously.

The problem: robots learn from demonstrations, and demonstrations are hard to collect

Modern robots increasingly learn manipulation skills the way people do - by watching examples. But someone has to create those examples. Teleoperation, where a human operator moves a robot in real time, is one of the most reliable ways to generate high-quality demonstration data for training robot policies. The catch: most teleoperation setups are clunky, expensive, or imprecise. The operator's hand and wrist motion has to be captured accurately, mapped onto a robot and recorded cleanly enough to train on. Parsa's write-up is a candid, end-to-end look at solving exactly that, and Rokoko's gloves sit at the center of the set-up.

Read about Parsa’s set-up and the whole experience here.

The teleoperation setup

Parsa's teleoperation stack combined robotics hardware with Rokoko's hand-tracking:

The Smartgloves II track what the fingers and wrist are doing, the Coil Pro answers the harder question - where in space is the hand right now - using an electromagnetic field for absolute positioning. Together they give the system both the pose and the location it needs to control the robot.

Why he chose Rokoko over Manus gloves

Before landing on Rokoko, Parsa evaluated the Manus Metagloves Pro and passed on them. According to his write-up, the reasons were calibration limitations, an obtrusive design, the cost and SDK restrictions. He chose Rokoko Smartgloves II in part because of established academic precedent - the DexCap research project from Stanford that used a similar approach.

How the system works

Parsa's pipeline turned raw glove data into robot motion in a handful of clear steps:

1. Read the glove data via Rokoko's RGMP protocol.
2. Calibrate the coordinate systems between glove-space and robot-space using six directional movements, so "forward" for the hand means "forward" for the robot.
3. Map fingers to motors - five human fingers onto the robot hand's six motors.
4. Add a "clutch" - a mechanism that lets the operator reposition their arm comfortably without the robot lurching, similar to lifting and repositioning a computer mouse.
5. Solve inverse kinematics using damped least squares (DLS) to convert the wrist pose into robot joint angles.
6. Smooth the output with filters to reduce jitter before it reaches the robot.

Before touching physical hardware, he validated the whole mapping in a lightweight simulator (built in Rerun), then recorded everything - glove, robot, and camera data - in the MCAP format for later training.

The honest part: what was hard

• The electromagnetic tracker could flip 180°. The Coil occasionally settled on a mirror-image "opposite-facing" solution. His fix was refreshingly low-tech: angle the Coil ~30° so gravity breaks the symmetry and disambiguates the reading.
• Accuracy has a working range. Tracking degraded beyond ~1.1 m and distorted within ~50 cm of the tracker, so operator positioning matters.
• IK stability. He started with one IK method (SSIK) but switched to damped least squares after occasional computation pauses started bleeding into the recorded behavior.
• Hardware wear. A thumb motor on one robot hand failed, limiting some work to a single-arm demo.

The results

• 221 unique robot demonstrations collected in total.
• 28 minutes of robot motion across 99 demonstrations (about an hour including resets).
• Operator skill improved fast - control reliability rose from 73.5% to 99.3%, and median IK error dropped from 93 mm to 1.3 mm by the end of the session.
• A policy trained on the collected data performed the task autonomously.
• Non-expert teammates could contribute usable demonstrations after a short familiarization - a sign the system is learnable, not just operable by its creator.

That learning curve - near-perfect control within a single session - is the headline. It says the human-to-robot mapping is intuitive enough to scale beyond one expert operator, which is exactly what teams collecting training data at volume need.

What this means if you're building robots

If your team is collecting demonstration data for imitation learning or robot foundation models, here’s the takeaway: motion capture gloves are a viable, precise, and affordable teleoperation front-end and the operator ramp-up is short. Rokoko Smartgloves II handle the finger and wrist capture, the Coil Pro adds the absolute spatial tracking that teleoperation demands.

Rokoko makes motion capture hardware and software for animators, game developers, VFX artists, and robotics. For robotics teams specifically, the same tools used across film and games are increasingly being used to teach robots how to move. If you are building robots that learn from demonstration then reach out to our team about testing out Rokoko Smartgloves II and Coil Pro for teleoperation and training-data collection.