> ## Documentation Index
> Fetch the complete documentation index at: https://docs.wuji.tech/llms.txt
> Use this file to discover all available pages before exploring further.

# Sim-to-real Deployment

> Run the trained reorient policy on the physical Wuji Hand — software pipeline, pose-estimation tuning, closed-loop control, and the end-to-end smoke test.

The deploy bridge runs the exported ONNX policy on a real Wuji Hand. A vision module reads the camera directly through the MVS SDK, tracks an ArUco-tagged cube anchored to a wrist AprilTag world frame, and publishes the pose over ZMQ. The `play_real` process subscribes to that pose, runs ONNX inference, and closes the loop by sending commands to the hand driver.

This page assumes the rig is already built and calibrated. If not, start with [Hardware Setup](/en/docs/wuji-mjlab/hardware).

<Info>
  You don't need to train to deploy. Download the pretrained `policy.onnx` and `policy_config.json` from [Releases](https://github.com/wuji-technology/wuji-mjlab/releases) and pass the `policy.onnx` path as `--ckpt`. The released policy is what produces the demo.
</Info>

## Pipeline

```text theme={null}
+-----------------+       +------------------+       +-------------------+
| Camera + tags   | -MVS->| cube observer    | -ZMQ->| play_real         |
| (aruco+apriltag)|       | (cube_world_     |       | (policy + control |
|                 |       |  observer.py)    |       |  + viewer)        |
+-----------------+       +------------------+       +-------------------+
                                                              |
                                                              v
                                                       +--------------+
                                                       | WujiHandDriver
                                                       | (real hardware)
                                                       +--------------+
```

`RealHandEnv` subclasses mjlab's `ManagerBasedRlEnv`, so the same observation and action managers as training apply verbatim — no duplicated pipelines. Inside the env's `step()`, actions go to the hand driver, and observations come from a mix of joint state (via the driver) and cube pose (via ZMQ from the observer).

The ONNX policy loads through a loader that reads the sidecar JSON for control-mode parameters (`action_scale`, `ema_alpha`, `warmup_time_s`, `control_mode`, `history_len`, `ctrl_dt`) captured at export time. This keeps deploy inference identical to the sim policy that produced the ONNX.

## Run the Closed Loop

After the hardware setup is complete, run each command in its own terminal, in order. `home` is a one-shot reset. `vision` and `play-real` both stay running, so they can't share a terminal.

```bash theme={null}
# Terminal 1 — one-shot reset, then exits
pixi run -e deploy home
# Terminal 2 — cube observer, stays running (OpenCV preview)
pixi run -e deploy vision
# Terminal 3 — closed-loop control + mirror viewer, stays running
pixi run -e deploy play-real --ckpt <path-to.onnx>
```

## Pose-Estimation Tuning

Once the hardware is fixed, [`observer.yaml`](https://github.com/wuji-technology/wuji-mjlab/blob/main/deploy/reorient/config/observer.yaml) gives four knobs to trade noise against lag.

| Parameter | Default | Effect |
| - | - | - |
| `rotation_filter.process_noise` | 0.5 | Higher is more agile and noisier |
| `rotation_filter.measurement_noise` | 0.1 | Lower trusts PnP more |
| `position_filter.alpha` | 0.8 | Low-pass in \[0, 1], where 1.0 is no filter |
| `pnp.reproj_threshold` | 6.0 px | Fits above this are dropped, and the cube goes lost |

Two presets ship in `observer.yaml`:

* **Agile** (fast response, more noise): `process_noise: 0.5`, `measurement_noise: 0.1`, `alpha: 0.8`
* **Smooth** (stable, slower response): `process_noise: 0.01`, `measurement_noise: 2.0`, `alpha: 0.2`

The shipped default is the agile preset, which is the configuration the trained policy was deployed against.

Tuning by symptom:

* Cube jitter when held still — switch to the smooth preset.
* Pose lag during fast reorientation — switch to the agile preset.
* Cube drops to lost repeatedly — raise `pnp.reproj_threshold` to \~8.0 px and re-check the camera intrinsics calibration. If axes look swapped, fix `face_rotations` in `cube_tags.json` or re-stick the offending tag.

## End-to-End Smoke Test

Walk these five checkpoints in order. If any fails, jump back to the indicated setup section before continuing.

### Step 1 — Home the Hand

```bash theme={null}
pixi run -e deploy home
```

Expected: a 3-second smooth ramp, 20 joints landing within ±2° of home, and a "home reached" message. Finger stutter or a hard stop means you should first stop control and cut actuator power so the joints are de-energized, then unplug and re-plug the USB cable and retry.

### Step 2 — Start the Cube Observer

```bash theme={null}
pixi run -e deploy vision
```

Expected: the OpenCV preview appears, a yellow "World Sampling: N/100" bar fills as the wrist tag stays in view, the label flips to green "WORLD FIXED" once 100 samples are averaged, and the cube axes overlay holds steady on a static cube.

### Step 3 — Verify the ZMQ Pose Stream

In another terminal (keep `vision` running), confirm cube poses publish on port 5555:

```bash theme={null}
pixi run -e deploy python - <<'EOF'
import json, zmq
sock = zmq.Context().socket(zmq.SUB)
sock.connect("tcp://localhost:5555")
sock.subscribe(b"")
sock.setsockopt(zmq.RCVTIMEO, 5000)  # fail after 5s instead of blocking forever
try:
    for _ in range(3):
        msg = json.loads(sock.recv_string())
        p = msg["cube1"]["position"]
        print(f"frame={msg['frame']:5d}  pos=({p['x']:+.3f},{p['y']:+.3f},{p['z']:+.3f})")
except zmq.Again:
    print("no pose received in 5s — is `vision` running and publishing on port 5555?")
EOF
```

You should see three fresh frame numbers and stable positions.

### Step 4 — Visual Cube-Pose Check

With `vision` still running:

```bash theme={null}
pixi run -e deploy python deploy/reorient/tools/calib_check.py
```

This opens a MuJoCo passive viewer of the digital twin. The hand mirrors live encoder readings, and the cube renders at the observer's pose estimate. Move the physical cube and watch the rendered cube follow. This catches what the ZMQ check can't:

* **Axis mismatches** — rotate the physical cube around one face axis and confirm the rendered cube rotates around the same axis. A mirrored or 90°-off rotation means the `face_rotations` in `cube_tags.json` are wrong, or a tag was placed in the wrong orientation.
* **Position offset** — center the cube on the palm. The rendered cube should sit on the palm geom. A > 2 cm offset usually means the hand mounting or camera intrinsics are off.

### Step 5 — Run the Closed-Loop Policy

```bash theme={null}
pixi run -e deploy play-real --ckpt <path-to.onnx>
```

Expected: the ONNX policy loads and prints its sidecar JSON, the hand homes, a passive MuJoCo mirror viewer opens (real joints, observed cube, and a translucent goal cube above), and the hand reorients the cube toward the goal with trial outcomes printed inline.

### Troubleshooting

| Symptom | Likely cause | Fix |
| - | - | - |
| Camera fails to open | MVS SDK not installed or `MVS_PYTHON_PATH` unset | Re-do the Hikvision MVS SDK setup, rerun the import smoke test |
| Wrist AprilTag never detected | Lighting, wrong tag family, ID, or size | Confirm AprilTag36h11, ID 0, 48 mm, and raise lighting |
| World Sampling bar never fills | Wrist tag small or blurry | Reposition so the tag is ≥ 80 px wide, refocus |
| Cube observer drops the cube often | Reprojection-error gate firing | Re-do the camera intrinsics calibration, verify the `cube_tags.json` face mapping with Step 4 |
| Policy diverges on the first step | Tag orientation mismatch | Fix `face_rotations` in `cube_tags.json` or re-place the offending tag |
| Hand judders during rollout | `ctrl_dt` mismatch between policy and hardware | Check the ONNX sidecar `ctrl_dt`, lower the hardware low-pass cutoff |

For a deeper look at the task design behind the policy, see [Architecture](/en/docs/wuji-mjlab/architecture).


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.