feat(mcap): add SOMA body and hand recording, replay, and live preview

docs/source/references/mcap_record_replay.rst

@@ -239,6 +239,146 @@ The ``record_controller`` / ``replay_controller`` and ``record_full_body`` /
 — without it, ``uv`` may resolve against the system Python and fail to match
 the lockfile.
 
+SOMA body workflows
+^^^^^^^^^^^^^^^^^^^
+
+The body commands accept ``--body-schema full-body-pose`` (default) or
+``--body-schema soma``. SOMA mode supports both ``SomaBodyJointRotations`` and
+``SomaBodyJointPoses``. They render the native 77-joint body topology, including
+available finger joints, without converting to ``FullBodyPose``. See
+:doc:`/device/soma` for the transport contract.
+
+FullBodyPose mode includes controllers by default. SOMA mode uses only the
+body stream, including available finger joints. Pass ``--controllers`` to
+include controller recording, or ``--no-controllers`` to disable it for
+FullBodyPose. Standalone hands use the separate hand commands; these examples
+do not extract or fuse them with the body.
+
+Install the example and demo publisher into a configured development checkout:
+
+.. code-block:: bash
+
+   uv pip install -e ./examples/mcap_record_replay
+   uv pip install -e ./examples/soma_body_publisher
+
+Start the live viewer and connect a CloudXR client. Live preview does not read
+or write an MCAP file:
+
+.. code-block:: bash
+
+   uv run --no-sync python -m isaaccapture_examples.mcap_record_replay.live_full_body \
+     --accept-eula --body-schema soma --soma-body-representation joint-rotations
+
+In a second terminal, start the existing publisher after the runtime is running:
+
+.. code-block:: bash
+
+   source ~/.cloudxr/run/cloudxr.env
+   uv run --no-sync python -m isaaccapture_examples.soma_body_publisher \
+     --body-representation joint-rotations --loop
+
+Record in another terminal, then stop the live viewer and publisher before
+replaying on the same Viser port:
+
+.. code-block:: bash
+
+   uv run --no-sync python -m isaaccapture_examples.mcap_record_replay.record_full_body \
+     10 recordings/soma_body_joint-rotations_demo.mcap \
+     --accept-eula --body-schema soma --soma-body-representation joint-rotations
+
+   uv run --no-sync python -m isaaccapture_examples.mcap_record_replay.replay_full_body \
+     recordings/soma_body_joint-rotations_demo.mcap \
+     --body-schema soma --soma-body-representation joint-rotations
+
+For evaluated poses, use ``--soma-body-representation joint-poses`` on the
+viewer, recorder, and replay commands, and ``--body-representation joint-poses``
+on the publisher. Use a separate output file. Replay works without OpenXR.
+As with the other viewers, ``--host`` and ``--port`` control browser access.
+
+The recorder stores received SOMA payloads unchanged, without FK. Viewing joint
+rotations uses ``SomaBodySource`` and its existing FK helper against a prepared
+neutral model. Viewing joint poses maps directly without loading SOMA-X assets
+or running FK. Unavailable joints and connecting bones are hidden.
+
+``--soma-body-collection-id`` selects the publisher collection and MCAP channel
+base. Its default, ``soma_body_demo``, matches the publisher's ``--collection-id``.
+Replay reads
+``<collection>/soma_body_joint_rotations_tracked`` or
+``<collection>/soma_body_joint_poses_tracked``. If no file is supplied, it selects
+the newest ``soma_body_<representation>_*.mcap`` under ``./recordings/``.
+
+Native replay still advances one tracked sample per update. The viewer uses the
+selected channel's average capture rate for pacing; ``--rate`` overrides it
+(``0`` is unpaced). It does not resample or synchronize streams. FullBodyPose
+and OpenXR hand replay retain their 60 Hz, duration-based playback.
+
+SOMA hand workflows
+^^^^^^^^^^^^^^^^^^^
+
+The hand commands accept ``--hand-schema openxr-hand-pose`` (default) or
+``--hand-schema soma``. SOMA mode supports ``SomaHandJointRotations`` and
+``SomaHandJointPoses``. Each left/right collection retains its anatomical
+``hand_side`` and uses the native 25-joint Wrist/finger layout. These commands
+do not fuse standalone hands with a body stream.
+
+Install the MCAP example and the existing hand publisher:
+
+.. code-block:: bash
+
+   uv pip install -e ./examples/mcap_record_replay
+   uv pip install -e ./examples/soma_hand_publisher
+
+Start the viewer and connect a CloudXR client:
+
+.. code-block:: bash
+
+   uv run --no-sync python -m isaaccapture_examples.mcap_record_replay.live_hand \
+     --accept-eula --hand-schema soma --soma-hand-representation joint-rotations
+
+After the runtime is running, start the publisher in a second terminal:
+
+.. code-block:: bash
+
+   source ~/.cloudxr/run/cloudxr.env
+   uv run --no-sync python -m isaaccapture_examples.soma_hand_publisher \
+     --hand-representation joint-rotations --loop
+
+Record in another terminal, then stop the viewer and publisher before replaying
+on the same Viser port:
+
+.. code-block:: bash
+
+   uv run --no-sync python -m isaaccapture_examples.mcap_record_replay.record_hand \
+     10 recordings/soma_hand_joint-rotations_demo.mcap \
+     --accept-eula --hand-schema soma --soma-hand-representation joint-rotations
+
+   uv run --no-sync python -m isaaccapture_examples.mcap_record_replay.replay_hand \
+     recordings/soma_hand_joint-rotations_demo.mcap \
+     --hand-schema soma --soma-hand-representation joint-rotations
+
+For evaluated poses, use ``--soma-hand-representation joint-poses`` on the
+viewer, recorder, and replay commands, and ``--hand-representation joint-poses``
+on the publisher. Use a separate output file. Live preview does not read or
+write MCAP; replay does not require OpenXR. ``--host`` and ``--port`` control
+browser access.
+
+Recording preserves received payloads without FK. Viewing rotations uses
+``SomaHandSource`` FK against prepared neutral models. Joint poses map directly
+without loading models. The viewer colors left green and right blue and hides
+unavailable joints and bones.
+
+``--soma-left-hand-collection-id`` and ``--soma-right-hand-collection-id`` default
+to ``soma_hand_left_demo`` and ``soma_hand_right_demo``, matching the publisher's
+``--left-collection-id`` and ``--right-collection-id``. Replay reads each
+collection's ``soma_hand_joint_rotations_tracked`` or
+``soma_hand_joint_poses_tracked`` channel. With no file argument, it selects the
+newest ``soma_hand_<representation>_*.mcap`` under ``./recordings/``.
+
+The trackers retain independent, one-sample-per-update replay. Pacing defaults
+to the busier selected hand's average capture rate; ``--rate`` overrides it.
+Missing or exhausted hands are hidden, and playback repeats after the longer
+channel finishes. No timestamp synchronization or resampling is added.
+
 API Reference
 -------------
 

examples/mcap_record_replay/README.md

@@ -29,14 +29,148 @@ keep a viewer local. Replay repeats until Ctrl+C; no loop flag is needed.
 
 | Channel | Live | Record | Replay |
 | --- | --- | --- | --- |
-| Hands | `live_hand` | `record_hand` | `replay_hand` |
+| Hands (OpenXR or SOMA) | `live_hand` | `record_hand` | `replay_hand` |
 | Controllers | `live_controller` | `record_controller` | `replay_controller` |
-| Full body | `live_full_body` | `record_full_body` | `replay_full_body` |
+| Full body (FullBodyPose or SOMA) | `live_full_body` | `record_full_body` | `replay_full_body` |
 | Raw hand joint SE3 poses | `live_joint_se3_pose` | `record_joint_se3_pose` | `replay_joint_se3_pose` |
 | VIVE SE3 trackers | — | `record_se3_vive` | `replay_se3_vive` |
 
 `record_*` takes an optional duration in seconds and an optional output path.
 Recording needs a live OpenXR runtime; replay needs only the file.
 
+The body commands default to `--body-schema full-body-pose`, including
+controllers. The hand commands default to `--hand-schema openxr-hand-pose`.
+Select SOMA with `--body-schema soma` or `--hand-schema soma`. These options
+and the `--soma-*` options below match the DeviceIO live viewer.
+
+SOMA body mode uses only the body stream by default, including any available
+finger joints. Pass `--controllers` to include controller recording, or
+`--no-controllers` to disable it for FullBodyPose. Standalone hand streams use
+the separate hand commands; they are not extracted from or fused with the body.
+
+## SOMA body
+
+Install the consumer and the existing demo publisher into your configured
+development environment:
+
+```bash
+uv pip install -e ./examples/mcap_record_replay
+uv pip install -e ./examples/soma_body_publisher
+```
+
+Start the live-only viewer, then connect a CloudXR client:
+
+```bash
+uv run --no-sync python -m isaaccapture_examples.mcap_record_replay.live_full_body \
+  --accept-eula --body-schema soma --soma-body-representation joint-rotations
+```
+
+In another terminal, start the publisher after the runtime is running:
+
+```bash
+source ~/.cloudxr/run/cloudxr.env
+uv run --no-sync python -m isaaccapture_examples.soma_body_publisher \
+  --body-representation joint-rotations --loop
+```
+
+Record the received SOMA data in a third terminal:
+
+```bash
+uv run --no-sync python -m isaaccapture_examples.mcap_record_replay.record_full_body \
+  10 recordings/soma_body_joint-rotations_demo.mcap \
+  --accept-eula --body-schema soma --soma-body-representation joint-rotations
+```
+
+Stop the live viewer and publisher before replaying on the same Viser port.
+Replay does not need a runtime or client:
+
+```bash
+uv run --no-sync python -m isaaccapture_examples.mcap_record_replay.replay_full_body \
+  recordings/soma_body_joint-rotations_demo.mcap \
+  --body-schema soma --soma-body-representation joint-rotations
+```
+
+For evaluated joint poses, use `--soma-body-representation joint-poses` on
+the viewer, recorder, and replay commands, and `--body-representation joint-poses`
+on the publisher. Use a separate output file, such as
+`recordings/soma_body_joint-poses_demo.mcap`. Both paths render
+SOMA's native 77-joint topology, including available finger joints. Unavailable
+joints and their connecting bones are hidden.
+
+Recording stores the received schema unchanged. Joint rotations use the
+existing `SomaBodySource` FK helper with a prepared neutral model when viewing;
+joint poses map directly without loading a model or running FK.
+
+`--soma-body-collection-id` selects the input collection and MCAP channel base.
+It defaults to `soma_body_demo`, matching the publisher's `--collection-id`.
+Recordings contain
+`<collection>/soma_body_joint_rotations` and its `_tracked` channel, or the
+corresponding `soma_body_joint_poses` pair. Replay consumes the tracked channel.
+Without a file argument, replay selects the newest recording matching its
+representation under `./recordings/`.
+
+Replay advances one tracked sample per update. Viewer pacing defaults to the
+selected channel's average capture rate; `--rate` overrides it (`0` is unpaced).
+This does not resample or synchronize streams. FullBodyPose and OpenXR hand
+replay retain their existing 60 Hz pacing and duration-based loop.
+
+## SOMA hands
+
+Install the existing hand demo publisher alongside the MCAP example:
+
+```bash
+uv pip install -e ./examples/mcap_record_replay
+uv pip install -e ./examples/soma_hand_publisher
+```
+
+Start the live-only viewer and connect a CloudXR client:
+
+```bash
+uv run --no-sync python -m isaaccapture_examples.mcap_record_replay.live_hand \
+  --accept-eula --hand-schema soma --soma-hand-representation joint-rotations
+```
+
+After the runtime is running, start the publisher in a second terminal:
+
+```bash
+source ~/.cloudxr/run/cloudxr.env
+uv run --no-sync python -m isaaccapture_examples.soma_hand_publisher \
+  --hand-representation joint-rotations --loop
+```
+
+Record in a third terminal. Stop the viewer and publisher before replaying on
+the same Viser port:
+
+```bash
+uv run --no-sync python -m isaaccapture_examples.mcap_record_replay.record_hand \
+  10 recordings/soma_hand_joint-rotations_demo.mcap \
+  --accept-eula --hand-schema soma --soma-hand-representation joint-rotations
+
+uv run --no-sync python -m isaaccapture_examples.mcap_record_replay.replay_hand \
+  recordings/soma_hand_joint-rotations_demo.mcap \
+  --hand-schema soma --soma-hand-representation joint-rotations
+```
+
+For evaluated poses, use `--soma-hand-representation joint-poses` on the
+viewer, recorder, and replay commands, and `--hand-representation joint-poses`
+on the publisher. Use a separate output file. The viewer renders each hand's
+native 25-joint Wrist/finger skeleton, with left green and right blue. Joint rotations
+use `SomaHandSource` FK against prepared neutral models; joint poses map directly
+without loading models. Recording does not run FK in either mode.
+
+`--soma-left-hand-collection-id` and `--soma-right-hand-collection-id` default to
+`soma_hand_left_demo` and `soma_hand_right_demo`, matching the publisher's
+`--left-collection-id` and `--right-collection-id`. Each collection is recorded
+separately with its anatomical `hand_side` intact. Replay reads its
+`soma_hand_joint_rotations_tracked` or `soma_hand_joint_poses_tracked` channel.
+Without a file argument, it selects the newest
+`soma_hand_<representation>_*.mcap` under `./recordings/`.
+
+Left and right trackers advance independently once per replay update, following
+the existing native behavior. Viewer pacing defaults to the busier selected
+hand's average capture rate; `--rate` overrides it. Missing or exhausted hands
+are hidden, and each pass ends when the longer channel finishes. No timestamp
+synchronization, resampling, or body/hand fusion is performed.
+
 A C++ recorder lives in `cpp/`. Docs:
 `docs/source/references/mcap_record_replay.rst`.

examples/mcap_record_replay/pyproject.toml

@@ -18,6 +18,7 @@ dependencies = [
     "isaaccapture[cloudxr]",
     "mcap>=1.2.0",
     "numpy>=1.23.0",
+    "py-soma-x==0.3.1",
     "viser>=0.2.0",
 ]
 
@@ -32,5 +33,5 @@ sources = ["python"]
 [tool.uv.sources]
 # Editable, so edits under src/python/ need no rebuild. Dropped from the
 # installed copy, which has no repo above it.
-# TODO(#880): nothing runs this example in CI; neither path is covered.
+# TODO(#880): the example install paths are not covered in CI.
 isaaccapture = { path = "../..", editable = true }