Sim-to-real Deployment

The deploy bridge runs the exported ONNX policy on a real Wuji Hand. A vision module reads the camera directly through the MVS SDK, tracks an ArUco-tagged cube anchored to a wrist AprilTag world frame, and publishes the pose over ZMQ. The play_real process subscribes to that pose, runs ONNX inference, and closes the loop by sending commands to the hand driver. This page assumes the rig is already built and calibrated. If not, start with Hardware Setup.

You don’t need to train to deploy. Download the pretrained policy.onnx and policy_config.json from Releases and pass the policy.onnx path as --ckpt. The released policy is what produces the demo.

Pipeline

+-----------------+       +------------------+       +-------------------+
| Camera + tags   | -MVS->| cube observer    | -ZMQ->| play_real         |
| (aruco+apriltag)|       | (cube_world_     |       | (policy + control |
|                 |       |  observer.py)    |       |  + viewer)        |
+-----------------+       +------------------+       +-------------------+
                                                              |
                                                              v
                                                       +--------------+
                                                       | WujiHandDriver
                                                       | (real hardware)
                                                       +--------------+

RealHandEnv subclasses mjlab’s ManagerBasedRlEnv, so the same observation and action managers as training apply verbatim — no duplicated pipelines. Inside the env’s step(), actions go to the hand driver, and observations come from a mix of joint state (via the driver) and cube pose (via ZMQ from the observer). The ONNX policy loads through a loader that reads the sidecar JSON for control-mode parameters (action_scale, ema_alpha, warmup_time_s, control_mode, history_len, ctrl_dt) captured at export time. This keeps deploy inference identical to the sim policy that produced the ONNX.

Run the Closed Loop

After the hardware setup is complete, run each command in its own terminal, in order. home is a one-shot reset. vision and play-real both stay running, so they can’t share a terminal.

# Terminal 1 — one-shot reset, then exits
pixi run -e deploy home
# Terminal 2 — cube observer, stays running (OpenCV preview)
pixi run -e deploy vision
# Terminal 3 — closed-loop control + mirror viewer, stays running
pixi run -e deploy play-real --ckpt <path-to.onnx>

Pose-Estimation Tuning

Once the hardware is fixed, observer.yaml gives four knobs to trade noise against lag.

Parameter	Default	Effect
`rotation_filter.process_noise`	0.5	Higher is more agile and noisier
`rotation_filter.measurement_noise`	0.1	Lower trusts PnP more
`position_filter.alpha`	0.8	Low-pass in [0, 1], where 1.0 is no filter
`pnp.reproj_threshold`	6.0 px	Fits above this are dropped, and the cube goes lost

Two presets ship in observer.yaml:

Agile (fast response, more noise): process_noise: 0.5, measurement_noise: 0.1, alpha: 0.8
Smooth (stable, slower response): process_noise: 0.01, measurement_noise: 2.0, alpha: 0.2

The shipped default is the agile preset, which is the configuration the trained policy was deployed against. Tuning by symptom:

Cube jitter when held still — switch to the smooth preset.
Pose lag during fast reorientation — switch to the agile preset.
Cube drops to lost repeatedly — raise pnp.reproj_threshold to ~8.0 px and re-check the camera intrinsics calibration. If axes look swapped, fix face_rotations in cube_tags.json or re-stick the offending tag.

End-to-End Smoke Test

Walk these five checkpoints in order. If any fails, jump back to the indicated setup section before continuing.

Step 1 — Home the Hand

pixi run -e deploy home

Expected: a 3-second smooth ramp, 20 joints landing within ±2° of home, and a “home reached” message. Finger stutter or a hard stop means you should first stop control and cut actuator power so the joints are de-energized, then unplug and re-plug the USB cable and retry.

Step 2 — Start the Cube Observer

pixi run -e deploy vision

Expected: the OpenCV preview appears, a yellow “World Sampling: N/100” bar fills as the wrist tag stays in view, the label flips to green “WORLD FIXED” once 100 samples are averaged, and the cube axes overlay holds steady on a static cube.

Step 3 — Verify the ZMQ Pose Stream

In another terminal (keep vision running), confirm cube poses publish on port 5555:

pixi run -e deploy python - <<'EOF'
import json, zmq
sock = zmq.Context().socket(zmq.SUB)
sock.connect("tcp://localhost:5555")
sock.subscribe(b"")
sock.setsockopt(zmq.RCVTIMEO, 5000)  # fail after 5s instead of blocking forever
try:
    for _ in range(3):
        msg = json.loads(sock.recv_string())
        p = msg["cube1"]["position"]
        print(f"frame={msg['frame']:5d}  pos=({p['x']:+.3f},{p['y']:+.3f},{p['z']:+.3f})")
except zmq.Again:
    print("no pose received in 5s — is `vision` running and publishing on port 5555?")
EOF

You should see three fresh frame numbers and stable positions.

Step 4 — Visual Cube-Pose Check

With vision still running:

pixi run -e deploy python deploy/reorient/tools/calib_check.py

This opens a MuJoCo passive viewer of the digital twin. The hand mirrors live encoder readings, and the cube renders at the observer’s pose estimate. Move the physical cube and watch the rendered cube follow. This catches what the ZMQ check can’t:

Axis mismatches — rotate the physical cube around one face axis and confirm the rendered cube rotates around the same axis. A mirrored or 90°-off rotation means the face_rotations in cube_tags.json are wrong, or a tag was placed in the wrong orientation.
Position offset — center the cube on the palm. The rendered cube should sit on the palm geom. A > 2 cm offset usually means the hand mounting or camera intrinsics are off.

Step 5 — Run the Closed-Loop Policy

pixi run -e deploy play-real --ckpt <path-to.onnx>

Expected: the ONNX policy loads and prints its sidecar JSON, the hand homes, a passive MuJoCo mirror viewer opens (real joints, observed cube, and a translucent goal cube above), and the hand reorients the cube toward the goal with trial outcomes printed inline.

Troubleshooting

Symptom	Likely cause	Fix
Camera fails to open	MVS SDK not installed or `MVS_PYTHON_PATH` unset	Re-do the Hikvision MVS SDK setup, rerun the import smoke test
Wrist AprilTag never detected	Lighting, wrong tag family, ID, or size	Confirm AprilTag36h11, ID 0, 48 mm, and raise lighting
World Sampling bar never fills	Wrist tag small or blurry	Reposition so the tag is ≥ 80 px wide, refocus
Cube observer drops the cube often	Reprojection-error gate firing	Re-do the camera intrinsics calibration, verify the `cube_tags.json` face mapping with Step 4
Policy diverges on the first step	Tag orientation mismatch	Fix `face_rotations` in `cube_tags.json` or re-place the offending tag
Hand judders during rollout	`ctrl_dt` mismatch between policy and hardware	Check the ONNX sidecar `ctrl_dt`, lower the hardware low-pass cutoff

For a deeper look at the task design behind the policy, see Architecture.

Overview

Hardware

Software

More

Pipeline

Run the Closed Loop

Pose-Estimation Tuning

End-to-End Smoke Test

Step 1 — Home the Hand

Step 2 — Start the Cube Observer

Step 3 — Verify the ZMQ Pose Stream

Step 4 — Visual Cube-Pose Check

Step 5 — Run the Closed-Loop Policy

Troubleshooting

​Pipeline

​Run the Closed Loop

​Pose-Estimation Tuning

​End-to-End Smoke Test

​Step 1 — Home the Hand

​Step 2 — Start the Cube Observer

​Step 3 — Verify the ZMQ Pose Stream

​Step 4 — Visual Cube-Pose Check

​Step 5 — Run the Closed-Loop Policy

​Troubleshooting

Pipeline

Run the Closed Loop

Pose-Estimation Tuning

End-to-End Smoke Test

Step 1 — Home the Hand

Step 2 — Start the Cube Observer

Step 3 — Verify the ZMQ Pose Stream

Step 4 — Visual Cube-Pose Check

Step 5 — Run the Closed-Loop Policy

Troubleshooting