Sim-to-real Deployment

The deploy bridge runs the exported ONNX policy on a real Wuji Hand. A vision module reads the camera directly through the MVS SDK, tracks an ArUco-tagged cube anchored to a wrist AprilTag world frame, and publishes the pose over ZMQ. The play_real process subscribes to that pose, runs ONNX inference, and closes the loop by sending commands to the hand driver.

This page assumes the rig is already built and calibrated. If not, start with Hardware Setup.

Pipeline

+-----------------+       +------------------+       +-------------------+
| Camera + tags   | -MVS->| cube observer    | -ZMQ->| play_real         |
| (aruco+apriltag)|       | (cube_world_     |       | (policy + control |
|                 |       |  observer.py)    |       |  + viewer)        |
+-----------------+       +------------------+       +-------------------+
                                                              |
                                                              v
                                                       +--------------+
                                                       | WujiHandDriver
                                                       | (real hardware)
                                                       +--------------+

RealHandEnv subclasses mjlab's ManagerBasedRlEnv, so the same observation and action managers as training apply verbatim — no duplicated pipelines. Inside the env's step(), actions go to the hand driver, and observations come from a mix of joint state (via the driver) and cube pose (via ZMQ from the observer).

The ONNX policy loads through a loader that reads the sidecar JSON for control-mode parameters (action_scale, ema_alpha, ctrl_dt, history_len) captured at export time. This keeps deploy inference identical to the sim policy that produced the ONNX.

Run the Closed Loop

After the hardware setup is complete, run each command in its own terminal, in order. home is a one-shot reset. vision and play-real both stay running, so they can't share a terminal.

# Terminal 1 — one-shot reset, then exits
pixi run -e deploy home
# Terminal 2 — cube observer, stays running (OpenCV preview)
pixi run -e deploy vision
# Terminal 3 — closed-loop control + mirror viewer, stays running
pixi run -e deploy play-real --ckpt <path-to.onnx>

Pose-Estimation Tuning

Once the hardware is fixed, observer.yaml gives four knobs to trade noise against lag.

Two presets ship in observer.yaml:

• Agile (fast response, more noise): process_noise: 0.5, measurement_noise: 0.1, alpha: 0.8
• Smooth (stable, slower response): process_noise: 0.01, measurement_noise: 2.0, alpha: 0.2

The shipped default is the agile preset, which is the configuration the trained policy was deployed against.

Tuning by symptom:

• Cube jitter when held still — switch to the smooth preset.
• Pose lag during fast reorientation — switch to the agile preset.
• Cube drops to lost repeatedly — raise pnp.reproj_threshold to ~8.0 px and re-check the camera intrinsics calibration. If axes look swapped, fix cube_tags.json face rotations or re-stick the offending tag.

End-to-End Smoke Test

Walk these five checkpoints in order. If any fails, jump back to the indicated setup section before continuing.

Step 1 — Home the Hand

pixi run -e deploy home

Expected: a 3-second smooth ramp, 20 joints landing within ±2° of home, and a "home reached" message. Finger stutter or a hard stop means you should first stop control and cut actuator power so the joints are de-energized, then unplug and re-plug the USB cable and retry.

Step 2 — Start the Cube Observer

pixi run -e deploy vision

Expected: the OpenCV preview appears, a yellow "World Sampling: N/100" bar fills as the wrist tag stays in view, the label flips to green "WORLD FIXED" once 100 samples are averaged, and the cube axes overlay holds steady on a static cube.

Step 3 — Verify the ZMQ Pose Stream

In a second terminal with vision running, confirm cube poses publish on port 5555:

pixi run -e deploy python - <<'EOF'
import json, zmq
sock = zmq.Context().socket(zmq.SUB)
sock.connect("tcp://localhost:5555")
sock.subscribe(b"")
sock.setsockopt(zmq.RCVTIMEO, 5000)  # fail after 5s instead of blocking forever
try:
    for _ in range(3):
        msg = json.loads(sock.recv_string())
        p = msg["cube1"]["position"]
        print(f"frame={msg['frame']:5d}  pos=({p['x']:+.3f},{p['y']:+.3f},{p['z']:+.3f})")
except zmq.Again:
    print("no pose received in 5s — is `vision` running and publishing on port 5555?")
EOF

You should see three fresh frame numbers and stable positions.

Step 4 — Visual Cube-Pose Check

With vision still running:

pixi run -e deploy python deploy/reorient/tools/calib_check.py

This opens a MuJoCo passive viewer of the digital twin. The hand mirrors live encoder readings, and the cube renders at the observer's pose estimate. Move the physical cube and watch the rendered cube follow. This catches what the ZMQ check can't:

• Axis mismatches — rotate the physical cube around one face axis and confirm the rendered cube rotates around the same axis. A mirrored or 90°-off rotation means cube_tags.json face rotations are wrong, or a tag was placed in the wrong orientation.
• Position offset — center the cube on the palm. The rendered cube should sit on the palm geom. A > 2 cm offset usually means the hand mounting or camera intrinsics are off.

Step 5 — Run the Closed-Loop Policy

pixi run -e deploy play-real --ckpt <path-to.onnx>

Expected: the ONNX policy loads and prints its sidecar JSON, the hand homes, a passive MuJoCo mirror viewer opens (real joints, observed cube, and a translucent goal cube above), and the hand reorients the cube toward the goal with trial outcomes printed inline.

Troubleshooting

For a deeper look at the task design behind the policy, see Architecture.