Retargeting Sessions: Keypoints to Joint Angles

Overview

Feed in a set of MediaPipe-format hand keypoints, get back a joint command you can send straight to a dexterous hand. This step connects to no device at all. This step performs computation only. The keypoints can come from anywhere. A camera, a Wuji Glove, a recorded replay, VR headset hand tracking — as long as the array has the same shape and units, everything downstream stays identical. Run this example first to confirm retargeting works and to see the input and output shapes. When you then attach real hardware, there’s far less ground to search. Matching examples: examples/python/retargeting/0.retarget_session.py and examples/c/retargeting/0_retarget_session.c. The C interface maps one to one onto Python. For the interface list, input format, and platform limits, see Retargeting. This page covers only how to run it and what the measurements show.

Prerequisites

Hardware

None. This example uses synthetic keypoints, so it neither scans for nor connects to a device.

Software

Item	Requirement
Operating system	Ubuntu 22.04 LTS or later
Architecture	Retargeting ships prebuilt wheels for Linux x86_64 and Linux aarch64 only
Wuji SDK	v2026.7.21 or later. That release moved the retargeting interface to the package top level. On earlier versions you import from the `wuji_sdk.retargeting` submodule, and this example’s imports don’t apply
Python	3.10 or later, plus `numpy` to hold the keypoint and joint-angle arrays
C toolchain	CMake 3.16 or later, plus the extracted SDK package, which supplies `include/` and `lib/libwuji_sdk_c.so`

Python
C

pip install wuji-sdk numpy

cmake -S examples/c/retargeting -B build/retargeting \
  -DWUJI_SDK_INCLUDE_DIR="${SDK}/include" \
  -DWUJI_SDK_LIB="${SDK}/lib/libwuji_sdk_c.so"
cmake --build build/retargeting --target 0_retarget_session

Before You Start

Nothing. This example is the smallest starting point on the retargeting path. Get it running, then move on to real hardware.

Data Flow

A single frame travels the path below. The session object keeps state across frames, so create it once and reuse it in your loop. Keypoints are in meters, joint commands in radians. The output is always float32 with a fixed shape that doesn’t vary with the number of online joints. float64 input is accepted and converted internally.

Runtime Handling

Branch point	Value	Effect
Hand model	`HandModel.WujiHand2` or `HandModel.WujiHand`	Selects the internal tuning profile. You never name a config file path
Handedness	`Handedness.Right` or `Handedness.Left`	The example defaults to the right hand. Keypoints and target hand must be the same side, and mixing them gives mirrored results
Tracking gap	Whether you call `reset()`	Skip the reset and the previous segment’s warm-start and smoothing state carry into the new one, so the first few frames jump
Degenerate keypoints	All zeros, or containing NaN or Inf	`step()` raises `WujiException: retargeting: invalid keypoints: contains NaN or Inf`. Confirm the keypoint source is valid before solving, and never pass an uninitialized array as a placeholder
Wrong shape	Element count other than 63	Python raises `ValueError: cannot reshape array of size N into shape (63,)`. Python works with the same length-63 layout as C internally
C-side error	create or step returns something other than `WUJI_STATUS_OK`	Call `wuji_last_error()` for the reason. A session that was created still needs `free`, or it leaks

The session is stateful. Reuse one session across a continuous tracking segment, reset between segments, and never create one per frame.

Integration Steps

Three fixed steps in a fixed order. C adds an explicit free.

Python
C

from wuji_sdk import HandModel, Handedness, RetargetSession

# 1. Create the session once per tracking segment
session = RetargetSession.for_hand(
    HandModel.WujiHand2, side=Handedness.Right
)

# 2. Solve each frame. keypoints is (21, 3) float32 in meters
qpos = session.step(keypoints)   # (20,) float32, radians

# 3. Clear state after a tracking gap, then start the next segment
session.reset()

#include "wuji_sdk.h"

/* 1. Create the session once per tracking segment */
struct WujiRetargetSession *session = NULL;
WujiStatus st = wuji_retarget_session_create(
    WUJI_HAND_MODEL_WUJI_HAND2, WUJI_HANDEDNESS_RIGHT, &session);
if (st != WUJI_STATUS_OK) {
    fprintf(stderr, "create failed (%d): %s\n", (int)st, wuji_last_error());
    return EXIT_FAILURE;
}

/* 2. Solve each frame. keypoints is a flat array of 63, row-major xyz, in meters */
float qpos[20];   /* radians, firmware joint order */
st = wuji_retarget_session_step(session, keypoints, qpos);

/* 3. Clear state after a tracking gap */
wuji_retarget_session_reset(session);

/* 4. Free it when you're done */
wuji_retarget_session_free(session);

The only real difference between the two sides is the keypoint memory layout. Python takes a (21, 3) two-dimensional array, C takes a flat array of 63 laying out x, y, and z for each point in landmark order.

Running It

Python
C

python examples/python/retargeting/0.retarget_session.py

./build/retargeting/0_retarget_session

Both sides print the same structure: the joint command’s shape and dtype, then the 20 joint angles grouped by finger.

qpos shape : (20,), dtype float32
joint command (radians, finger-major thumb/index/middle/ring/pinky):
  thumb : ['+1.291', '+0.206', '+0.345', '+0.039']
  index : ['+0.219', '+0.654', '+0.202', '+0.087']
  middle: ['+0.290', '+0.477', '+0.004', '+0.001']
  ring  : ['+0.166', '+0.274', '-0.002', '-0.001']
  pinky : ['+0.061', '+0.130', '+0.029', '+0.039']

C skips the shape line and starts at the joint command:

joint command (radians, finger-major thumb/index/middle/ring/pinky):
  thumb : [+1.291, +0.206, +0.345, +0.039]
  index : [+0.219, +0.654, +0.202, +0.087]
  middle: [+0.290, +0.477, +0.004, +0.001]
  ring  : [+0.166, +0.274, -0.002, -0.001]
  pinky : [+0.061, +0.130, +0.029, +0.039]

These are actual results, measured on Wuji SDK v2026.8.31, Python 3.12, the C SDK for x86_64-linux-gnu, and Linux x86_64, with the example’s built-in synthetic open right-hand pose as input. For the same input, both interfaces produce bit-identical joint angles across all 20 values, because they share one solver implementation. Change the keypoints or the hand model and the numbers change, but the shape and grouping don’t.

Adapting It

Swap In a Real Keypoint Source

Replace the example’s make_open_hand_keypoints() with your source. As long as it produces (21, 3) keypoints in meters in MediaPipe landmark order, nothing downstream changes. The Wuji Glove’s hand_skeleton() stream is already in this format, and Retargeting shows how to connect it.

Change Hand Model and Handedness

Targeting a Wuji Hand instead, change HandModel.WujiHand2 to HandModel.WujiHand. For a left-hand assembly, change Handedness.Right to Handedness.Left and confirm the keypoints come from a left hand too.

Move to a Real-Time Loop

This example solves one frame. In a real-time setting, call step() on a fixed period in a loop and handle the case where no new frame arrived. For the full loop, see Retargeting.

Input Validity

Confirm the keypoints aren’t degenerate before solving. An all-zero array, or one holding NaN or Inf, makes step() raise invalid keypoints. When tracking is lost, repeating the last valid frame is safer than sending zeros.

Handling Tracking Gaps

The hand leaves the frame, the device disconnects, a replay cuts to a new segment — call reset() once after any of these. Skip it and the first few frames after recovery ease out of the old state, which looks like lag.

Retargeting: interface description and parameter meanings
Common Data Types: the structures behind keypoints and joint commands

​Overview

​Prerequisites

​Hardware

​Software

​Before You Start

​Data Flow

​Runtime Handling

​Integration Steps

​Running It

​Adapting It

​Swap In a Real Keypoint Source

​Change Hand Model and Handedness

​Move to a Real-Time Loop

​Input Validity

​Handling Tracking Gaps

​Related