Quick Start

Before running examples, make sure:

You have completed Installation. If you encounter missing model files, check if submodules are initialized: git submodule update --init --recursive
When using Vision Pro, install Tracking Streamer on your Apple Vision Pro (available from App Store or GitHub)

Simulation Demo

cd example

# Vision Pro real-time teleoperation (recommended), replace <vision-pro-ip> with actual IP (found in Tracking Streamer)
python teleop_sim.py --input visionpro --ip <vision-pro-ip> --hand left

# Playback pre-recorded data (when Vision Pro is unavailable)
python teleop_sim.py --play data/avp1.pkl --hand left

# Use an MP4 video as input
python teleop_sim.py --video <your-video.mp4> --hand right
python teleop_sim.py --video <your-video.mp4> --hand right --show-video

# Use Intel RealSense as live input
python teleop_sim.py --realsense --hand right
python teleop_sim.py --realsense --hand right --show-video

Expected results: With Vision Pro real-time teleoperation, the simulation interface will display a simulated hand following the user’s hand movements. With pre-recorded data playback, the result will look similar to the following demo:

When using MP4 or RealSense input, you can add --show-video to inspect the original image together with the MediaPipe landmark overlay for debugging.

Real Hardware Control

Run Examples

cd example

# Vision Pro real-time input (recommended, requires Apple Vision Pro configuration), replace <vision-pro-ip> with actual IP (found in Tracking Streamer)
python teleop_real.py --input visionpro --ip <vision-pro-ip> --hand right

# Playback pre-recorded data (when Vision Pro is unavailable)
python teleop_real.py --play data/avp1.pkl --hand right

# Use an MP4 video as input
python teleop_real.py --video <your-video.mp4> --hand right
python teleop_real.py --video <your-video.mp4> --hand right --show-video

# Use Intel RealSense as live input
python teleop_real.py --realsense --hand right
python teleop_real.py --realsense --hand right --show-video

Linux USB Permissions: Wuji Hand communicates via USB serial. Linux restricts serial device access for regular users by default. We recommend adding your user to the dialout group: sudo usermod -aG dialout $USER (requires logout/login). sudo chmod a+rw /dev/ttyUSB0 is only for temporary troubleshooting and resets after reboot.

Expected results: With Vision Pro real-time teleoperation, the Wuji Hand will follow the user’s hand movements. With pre-recorded data playback, the Wuji Hand motion trajectory will be similar to the expected results video in the Simulation Demo. MP4 video input is useful for offline validation and demos, while RealSense is useful for live input testing when Vision Pro is unavailable.

Record Input Data

When using Vision Pro, you can simultaneously record input data for later playback and debugging:

cd example
python teleop_real.py --input visionpro --record

When you use --video or --realsense, the example scripts automatically switch to config/adaptive_analytical_video.yaml. With --input wuji_glove, they switch to the matching adaptive_analytical_wuji_glove_*.yaml. You can still override this behavior with --config.

Hand Model: Wuji Hand (Default) and Wuji Hand 2

The commands above drive the Wuji Hand (USB) by default — no extra flag needed. To drive Wuji Hand 2 (networked), pass --hand-model wuji_hand_2 with a Wuji Hand 2 config. The Wuji Hand 2 config points the optimizer at the Wuji Hand 2 model via optimizer.urdf_path / mjcf_path / link_naming, so no code change is needed:

cd example

# Wuji Glove input -> Wuji Hand 2 (right hand)
python teleop_real.py --input wuji_glove --hand right --glove-sn <YOUR_SN> \
    --config config/adaptive_analytical_wuji_glove_wuji_hand_2_right.yaml

# Multiple Wuji Hand 2 devices online — select one by address:
python teleop_real.py --input wuji_glove --hand right --glove-sn <YOUR_SN> \
    --wuji-hand-2-ip <hand-ip>:50001 \
    --config config/adaptive_analytical_wuji_glove_wuji_hand_2_right.yaml

Without --hand-model, the model is inferred automatically: Wuji Hand 2 is selected when optimizer.urdf_path / optimizer.mjcf_path or the config file path contains hand2, otherwise Wuji Hand. The hand_2 in the sample config file names doesn’t match — recognition relies on the model paths inside the config. When --wuji-hand-2-ip is omitted, the script scans the network for Wuji Hand 2 devices. With one online it auto-connects. With several online it matches --hand left/right against each device’s reported handedness.

Wuji Hand 2 and Wuji Hand firmware must match the installed wuji_sdk. Mismatched firmware fails fast at connect with a clear error.

Wuji Glove Input

Wuji Glove is the recommended live input for teleoperation. It publishes 21 MediaPipe-format hand keypoints through wuji_sdk and drives the Wuji Glove example configs.

Prepare the Glove in Wuji Studio

Connect and calibrate the glove in Wuji Studio before teleoperation:

Download Wuji Studio from the Wuji Studio releases page.
Install the desktop app on Ubuntu: sudo apt install ./wuji-studio_<version>_amd64.deb
Connect the Wuji Glove, then launch Wuji Studio.
In the Connect view, search for the glove and connect it.
Open Visualization to confirm the glove’s live motion and contact status.
Click the user avatar at the bottom of the Activity bar, then choose Create profile to add a profile. Studio switches to it automatically. The preset Default profile doesn’t store calibration results, so calibrating under it has no effect.
Open Calibration and follow the on-screen steps. Wuji Studio saves the calibration file when the procedure completes.

A profile in Wuji Studio is an SDK user, and the calibrated hand model is stored per profile. Teleoperation runs under whichever profile is currently selected, so keep that profile active after calibration for the results to apply. For profile management, see Wuji Studio Basic Settings.

Run With the Glove

cd example

# Simulation
python teleop_sim.py --input wuji_glove --hand right --glove-sn <YOUR_SN>

# Real hardware
python teleop_real.py --input wuji_glove --hand right --glove-sn <YOUR_SN>

The Wuji Glove path adds per-hand configs (adaptive_analytical_wuji_glove_left.yaml / adaptive_analytical_wuji_glove_right.yaml). To drive a Wuji Hand 2 with the glove, pass the matching Wuji Hand 2 config — see the Hand Model section above.

Calibrate Neutral-Pose Offsets

calibrate_offset.py records the neutral-pose wrist and thumb offsets, so the retargeted hand matches your relaxed hand pose:

cd example
python calibrate_offset.py --hand right \
    --config config/adaptive_analytical_wuji_glove_right.yaml

Command Reference

teleop_sim.py (Simulation)

Option	Default	Description
`--config`	`config/adaptive_analytical_avp.yaml`	YAML configuration file
`--hand`	`left`	Hand side (`left`/`right`)
`--input`	-	Input type (`visionpro`/`mediapipe_replay`/`video`/`realsense`/`wuji_glove`)
`--play FILE`	-	Play recorded data (shortcut for `--input mediapipe_replay`)
`--video FILE`	-	Use an MP4 file as input (shortcut for `--input video`)
`--realsense`	-	Use Intel RealSense as input (shortcut for `--input realsense`)
`--show-video`	-	Show the image with MediaPipe overlay, only used in `video`/`realsense` mode
`--device-name`	`glove`	wuji_sdk device name for the Wuji Glove
`--glove-sn`	-	Wuji Glove serial number (required when multiple Wuji devices are online)
`--ip`	`192.168.50.127`	Vision Pro IP address
`--speed`	`1.0`	Playback speed
`--no-loop`	-	Disable loop playback
`--record`	-	Record input data
`--output FILE`	-	Recording output path

teleop_real.py (Real Hardware)

Option	Default	Description
`--config`	`config/adaptive_analytical_avp.yaml`	YAML configuration file
`--hand`	`right`	Hand side (`left`/`right`)
`--input`	-	Input type (`visionpro`/`mediapipe_replay`/`video`/`realsense`/`wuji_glove`)
`--play FILE`	-	Play recorded data (shortcut for `--input mediapipe_replay`)
`--video FILE`	-	Use an MP4 file as input (shortcut for `--input video`)
`--realsense`	-	Use Intel RealSense as input (shortcut for `--input realsense`)
`--show-video`	-	Show the image with MediaPipe overlay, only used in `video`/`realsense` mode
`--device-name`	`glove`	wuji_sdk device name for the Wuji Glove
`--glove-sn`	-	Wuji Glove serial number (required when multiple Wuji devices are online)
`--ip`	`192.168.50.127`	Vision Pro IP address
`--speed`	`1.0`	Playback speed
`--no-loop`	-	Disable loop playback
`--record`	-	Record input data
`--output FILE`	-	Recording output path
`--hand-model`	inferred from config	Hand model: `wuji_hand` (Wuji Hand, USB) or `wuji_hand_2` (Wuji Hand 2, networked)
`--hand-serial`	-	Wuji Hand serial number
`--wuji-hand-2-ip`	auto-scan	Wuji Hand 2 network address, e.g. `192.168.1.111:50001`
`--kp`	`3.0`	Wuji Hand 2 MIT controller kp
`--kd`	`0.1`	Wuji Hand 2 MIT controller kd
`--current-limit`	`1.5`	Wuji Hand 2 per-joint system current limit, amps

Overview

Hardware

Software

More

Simulation Demo

Real Hardware Control

Run Examples

Record Input Data

Hand Model: Wuji Hand (Default) and Wuji Hand 2

Wuji Glove Input

Prepare the Glove in Wuji Studio

Run With the Glove

Calibrate Neutral-Pose Offsets

Command Reference

teleop_sim.py (Simulation)

teleop_real.py (Real Hardware)

​Simulation Demo

​Real Hardware Control

​Run Examples

​Record Input Data

​Hand Model: Wuji Hand (Default) and Wuji Hand 2

​Wuji Glove Input

​Prepare the Glove in Wuji Studio

​Run With the Glove

​Calibrate Neutral-Pose Offsets

​Command Reference

​teleop_sim.py (Simulation)

​teleop_real.py (Real Hardware)

Simulation Demo

Real Hardware Control

Run Examples

Record Input Data

Hand Model: Wuji Hand (Default) and Wuji Hand 2

Wuji Glove Input

Prepare the Glove in Wuji Studio

Run With the Glove

Calibrate Neutral-Pose Offsets

Command Reference

teleop_sim.py (Simulation)

teleop_real.py (Real Hardware)