Quick Start

Simulation Demo

cd example

# Vision Pro real-time teleoperation (recommended), replace <vision-pro-ip> with actual IP (found in Tracking Streamer)
python teleop_sim.py --input visionpro --ip <vision-pro-ip> --hand left

# Playback pre-recorded data (when Vision Pro is unavailable)
python teleop_sim.py --play data/avp1.pkl --hand left

# Use an MP4 video as input
python teleop_sim.py --video <your-video.mp4> --hand right
python teleop_sim.py --video <your-video.mp4> --hand right --show-video

# Use Intel RealSense as live input
python teleop_sim.py --realsense --hand right
python teleop_sim.py --realsense --hand right --show-video

Expected results:

With Vision Pro real-time teleoperation, the simulation interface will display a simulated hand following the user's hand movements. With pre-recorded data playback, the result will look similar to the following demo:

When using MP4 or RealSense input, you can add --show-video to inspect the original image together with the MediaPipe landmark overlay for debugging.

Real Hardware Control

Run Examples

cd example

# Vision Pro real-time input (recommended, requires Apple Vision Pro configuration), replace <vision-pro-ip> with actual IP (found in Tracking Streamer)
python teleop_real.py --input visionpro --ip <vision-pro-ip> --hand right

# Playback pre-recorded data (when Vision Pro is unavailable)
python teleop_real.py --play data/avp1.pkl --hand right

# Use an MP4 video as input
python teleop_real.py --video <your-video.mp4> --hand right
python teleop_real.py --video <your-video.mp4> --hand right --show-video

# Use Intel RealSense as live input
python teleop_real.py --realsense --hand right
python teleop_real.py --realsense --hand right --show-video

Expected results:

With Vision Pro real-time teleoperation, the Wuji Hand will follow the user's hand movements. With pre-recorded data playback, the Wuji Hand motion trajectory will be similar to the expected results video in the Simulation Demo. MP4 video input is useful for offline validation and demos, while RealSense is useful for live input testing when Vision Pro is unavailable.

Record Input Data

When using Vision Pro, you can simultaneously record input data for later playback and debugging:

cd example
python teleop_real.py --input visionpro --record

Hand Model: Wuji Hand (Default) and Wuji Hand 2

The commands above drive the Wuji Hand (USB) by default — no extra flag needed. To drive Wuji Hand 2 (networked), pass --hand-model wuji_hand_2 with a Wuji Hand 2 config. The Wuji Hand 2 config points the optimizer at the Wuji Hand 2 model via optimizer.urdf_path / mjcf_path / link_naming, so no code change is needed:

cd example

# Wuji Glove input -> Wuji Hand 2 (right hand)
python teleop_real.py --input wuji_glove --hand right --glove-sn <YOUR_SN> \
    --config config/adaptive_analytical_wuji_glove_wuji_hand_2_right.yaml

# Multiple Wuji Hand 2 devices online — select one by address:
python teleop_real.py --input wuji_glove --hand right --glove-sn <YOUR_SN> \
    --wuji-hand-2-ip <hand-ip>:50001 \
    --config config/adaptive_analytical_wuji_glove_wuji_hand_2_right.yaml

Without --hand-model, the model is inferred automatically: Wuji Hand 2 is selected when optimizer.urdf_path / optimizer.mjcf_path or the config file path contains hand2, otherwise Wuji Hand. The hand_2 in the sample config file names doesn't match — recognition relies on the model paths inside the config.

When --wuji-hand-2-ip is omitted, the script scans the network for Wuji Hand 2 devices. With one online it auto-connects. With several online it matches --hand left/right against each device's reported handedness.

Wuji Glove Input

Wuji Glove is the recommended live input for teleoperation. It publishes 21 MediaPipe-format hand keypoints through wuji_sdk and drives the Wuji Glove example configs.

Prepare the Glove in Wuji Studio

Connect and calibrate the glove in Wuji Studio before teleoperation:

1. Download Wuji Studio from the Wuji Studio releases page.
2. Install the desktop app on Ubuntu: sudo apt install ./wuji-studio_<version>_amd64.deb
3. Connect the Wuji Glove, then launch Wuji Studio.
4. In the Connect view, search for the glove and connect it.
5. Open Visualization to confirm the glove's live motion and contact status.
6. Click the user avatar at the bottom of the Activity bar, then choose Create profile to add a profile. Studio switches to it automatically. The preset Default profile doesn't store calibration results, so calibrating under it has no effect.
7. Open Calibration and follow the on-screen steps. Wuji Studio saves the calibration file when the procedure completes.

A profile in Wuji Studio is an SDK user, and the calibrated hand model is stored per profile. Teleoperation runs under whichever profile is currently selected, so keep that profile active after calibration for the results to apply. For profile management, see Wuji Studio Basic Settings.

Run With the Glove

cd example

# Simulation
python teleop_sim.py --input wuji_glove --hand right --glove-sn <YOUR_SN>

# Real hardware
python teleop_real.py --input wuji_glove --hand right --glove-sn <YOUR_SN>

The Wuji Glove path adds per-hand configs (adaptive_analytical_wuji_glove_left.yaml / adaptive_analytical_wuji_glove_right.yaml). To drive a Wuji Hand 2 with the glove, pass the matching Wuji Hand 2 config — see the Hand Model section above.

Calibrate Neutral-Pose Offsets

calibrate_offset.py records the neutral-pose wrist and thumb offsets, so the retargeted hand matches your relaxed hand pose:

cd example
python calibrate_offset.py --hand right \
    --config config/adaptive_analytical_wuji_glove_right.yaml

Command Reference

teleop_sim.py (Simulation)

teleop_real.py (Real Hardware)