Installation

System Requirements

Software Versions

Software	Version
Python	3.10
Wuji Hand SDK	See Release Notes for latest updates
Firmware	See Release Notes for latest updates

Input Devices

Device	Description
Apple Vision Pro	Real-time hand tracking (recommended)
Pre-recorded Data	Playback MediaPipe format data for validation and debugging
MP4 Video	Extract keypoints from offline hand videos for demos, debugging, and validation
Intel RealSense	Extract keypoints from a live RGB camera stream

The system uses a unified MediaPipe keypoint format as input. Current public examples support Vision Pro, replay data, MP4 video, and Intel RealSense. Beyond that, you can integrate Pico, Meta Quest, or other image-based hand-tracking solutions for custom development.

Installation Guide

Clone Repository

git clone --recurse-submodules https://github.com/wuji-technology/wuji-retargeting.git
cd wuji-retargeting
git lfs pull

Use the --recurse-submodules flag to pull the Wuji Hand URDF model submodule.

Install Dependencies

pip install -r requirements.txt
pip install -e .

Ubuntu 22.04 note: The distro's stock pip 22.0.2 has a build-isolation bug that can install this package as UNKNOWN 0.0.0 with none of its dependencies. Upgrade pip first: python3 -m pip install -U pip.

If you encounter any issues during installation, please refer to Troubleshooting.

Optional Input Mode Dependencies

Vision Pro / Pre-recorded data: The base installation above is sufficient.
MP4 video: Install the video extra dependencies:

pip install -e ".[video]"

Intel RealSense: In addition to the base installation, install the RealSense extra dependencies:

pip install -e ".[realsense]"

MP4 video mode requires you to install the video extras and provide your own input video file. The RealSense extras already include all video dependencies, so there's no need to install both. RealSense mode requires a connected Intel RealSense device that is accessible from your environment.

Run in Docker

This repository ships no official Dockerfile. To run it in a container, build your own image — the installation steps above work unchanged inside the container.

Live and real-hardware paths need an extra mount. The Wuji Hand URDF model ships with the submodule and is already inside the image, but the Wuji SDK (wuji_sdk for Wuji Glove, wujihandpy for the real hand) reads per-device assets from the host's $HOME/.wuji — for example the device URDF ~/.wuji/sdk/models/<serial>_hand.urdf and parameters ~/.wuji/sdk/params/<serial>.toml, provisioned by Wuji Studio. These files do not exist in a fresh container, so mount them in:

docker run --rm -it \
    -v ~/.wuji:/root/.wuji \
    your-retargeting-image

Mount the host's ~/.wuji to the container user's home (/root/.wuji when running as root, adjust for other users). Real-hardware and Wuji Glove paths additionally need device connectivity (USB passthrough or host networking). Simulation and replay paths (e.g. teleop_sim.py --play data/avp1.pkl) do not require ~/.wuji.

Configure Apple Vision Pro

To use Vision Pro for real-time hand tracking, complete the following configuration. If you don't have a Vision Pro, you can use pre-recorded data to verify system functionality.

Install VisionOS App

Install Tracking Streamer from the App Store on your Apple Vision Pro.

Install Python Library

pip install --upgrade avp_stream

Keep the avp_stream library up to date. The latest App Store version requires avp_stream >= 2.50.0. The VisionOS app will display a warning if the Python library version is outdated.

Optional: iOS Companion App

Install Tracking Manager from the App Store on your iPhone/iPad for managing recordings, settings, and camera calibration.

No additional network configuration is required. It works directly after installation.

On this page